The Roblox Web Application has been built as a monolith. As in the diagram below, it requires all engineers developing on one project to release together. The project is built using a .Net MVC framework, including Asp .Net Web Form, so applying any change requires MSBuild — even for JS/CSS/Image changes.
Build and Deployment
On the consumer side, loading web assets has two scenarios — the browser’s first time requesting the resource, and pulling from CDN after the first request. Our flowchart below shows how uploading assets to CDN is included in the first-time-loading effort.
Then, after the first-time-loading scenario, the browser will start pulling a resource from the CDN as such:
With over 10+ years of continuous development and an aging code base, our development and release pipelines lack the efficiency we expect from the CDN. This has happened for a few reasons:
- Even if there are only web asset changes (such as Javascript, CSS, HTML, images, etc.), the entire website needs to be built and deployed.
- If any issues arise from changes, all committed changes from different teams have to wait for the problem to be resolved before they can be released.
Additionally, web assets are bundled and uploaded to the CDN by a class method, making modern Frontend technology – such as ES6+(ECMAScript 6+) or Typescript – hard to adapt for long-term development. Thus, the current build and deployment pipeline creates obstacles for everyone.
Since 2017, Roblox has tested a variety of approaches to isolate build and deployment processes for web assets.
- The first successful approach has been to take our learnings from the creation of smaller API sites and apply it to create app sites. App sites contain web assets per component and deploy as their own project. An app site is not an individual web site: rather, it provides endpoints to bundle assets and returns the redirected CDN URL.
- The second phase is to introduce a Webpack bundle system as a replacement for our assemblies class method, in conjunction with a new static content service. This allows us to upload bundles into the CDN, thereby removing web server deployment and only updating the CDN URL for each component.
Phase I – Web App (Site)
Since we’ve already developed a modern JS framework for some of the feature components, it’s natural to start by isolating the client-side rendering project from the main Roblox website project. We treat the main project as a skeleton, so the main project only needs to know which web app site to request. Then, each project hosts a group of similar feature components and their own assets.
Each component inside the project will have its own controller to provide an endpoint to bundle Javascript, CSS, HTML, and the endpoint response redirects to the CDN URL. For example, https://chatsite.roblox.com/chat/{version}/get-javascript-bundle as an endpoint returns https://js.rbxcdn.com/{hashNumber}.js. The redirect response will be the 301 Http status, and the response will be cached into the browser after the first request.
In order to notify the browser of a change, we add a version number into the app site URL for cache busting purposes. The version number is a GUID and is added/updated from the admin site. For each change made from the app site, developers need to deploy the app site and then go to the admin site to update the GUID to flush the cache.
Thus we inherit the same MSBuild system for bundling, but each project will do its own MSBuild, so the owner of each project has flexibility to release their changes (as below).
Build and Deployment
The benefit of this approach is that when engineers deploy new code, they only need to do two steps. First, they release an app site which has the necessary changes. Then they update the hash value for each app site URL to notify browsers there are updates and pull updated resources. For the first time, as below, the browser will request resource by updated URL and download resource from the relative URL before CDN URL is ready.
After the first time loading, once the CDN URL is ready, the browser will start pulling resources from CDN directly. Like so:
Now with this approach, we’re able to provide a translation resource for each component without having to perform a website release. The translation resource does not need bundling, and it has its own endpoint to get the translation resource from LocaleResourceFactory as part of the JS file.
The project inherits the same class method to bundle assets and upload to the CDN. This mechanism requires image loads from the CSS rule to stay on the relative path. Since the CSS bundle remains a static farm, the image stays in the same server physically. So for Phase I, images continue uploading from the main website as before.
Pros
- Developers are able to develop and release their own web assets for any FE change, such as Javascript, HTML, or CSS.
- Developers can update the translation string by only releasing the consumer app site instead of a web release.
- As a side effect, only the client-side rendering page can be deployed from the app site build pipeline which encourages teams to move faster on client-side rendering refactor.
Cons
- Due to legacy bundle system re-use, images and modern JS can still not be easily developed and deployed from the app site.
- Because the bundles are generated when the browser sends a request to the app site, there is the possibility that if the new CDN URL is not generated successfully or is in broken status, then the browser could be cached before the next update.
Phase II – Web App (Folder)
After Phase I, an obvious question arose: Why do we still have to redirect the CDN URL from the app site? Why can’t we embed the CDN URL into the main website code?
So we begin phase II trying to introduce a static content service to upload web assets to S3 during build time, and then return the CDN URL into DB while inheriting the Phase I component structure (and keeping the Roblox website as a skeleton). Each component resource bundle will then map to a static component Enum string name. The component name will be embedded into the website skeleton page.
The website will request static content service for the latest enabled content by component name and will get back a CDN URL which the CDN will pull from S3 if it does not have it cached. Then, with this static content service benefit, we finish bundling assets during build time. The CDN URL is then ready before we enable the new version and validation. Since we embedded the CDN URL into the Roblox website project directly, there is no need to deploy the web server to hit an extra app site to get the CDN URL. This saves developers from web server deployment.
When engineers deploy changes for the first time, only one step is required to update each static component bundle CDN URL from the admin site. The URL will be pulled from CDN directly since it is not yet cached.
After the first time, CDN URL will be available when browsers requests from static content service.
The second part of phase II is replacing the current bundling system with a Webpack tool. Webpack has been used to apply an SCSS compiler, merge/minify assets, upload images, compile ES6, source-map, etc. This webpack tool decouples the web frontend project from MSBuild, so each web app is a folder to maintain the resource. And thanks to the URL loader, we are able to load an image to the CDN and embed the CDN URL into the CSS bundle file. As you can see from below, the file upload and CDN URL generation happens during the project build time, and the CDN URL has been embedded into the Roblox website, so we don’t need any MS deployment to release new CDN URLs.
So the current build and deployment pipeline of WebApp works like this:
Pros
- Developers do not need to deploy a web server to release a change. They only need to go to the admin site and update the latest CDN URL with the related component name. This is a great improvement for our release pipeline.
- Images get uploaded and deployed with each component and served from the CDN.
- Webpack tool introduction upgrades the entire web frontend development environment, freeing engineers to go to ES6+, Typescript, etc.
Cons
- As a trade-off, we lost translation resource support from this phase, so any new resource we introduce needs to go through web release.
- As in Phase I, any code duplication or dependencies between bundles are hard to identify.
What’s Next
While we’ve made great strides, we would like to continue to study Webpack and other possible tools to address the issue of dependencies between bundles. Also, our translation resource fetch needs to be able to fetch dynamically to enable isolated deployment for any new resource namespace introduction. We’ve come a long way, but we’re always looking for new opportunities to make our build pipeline faster and easier to develop.
Neither Roblox Corporation nor this blog endorses or supports any company or service. Also, no guarantees or promises are made regarding the accuracy, reliability or completeness of the information contained in this blog.
This blog post was originally published on the Roblox Tech Blog.