Our Games Team has been spending significant time making ROBLOX Battle as robust and feature-rich as it can be. We’ve already covered some of the more interesting updates, including the core gameplay mechanics that have been introduced recently and level design. Today, we had the chance to chat with Sorcus, the Games Team Lead, about using metrics to inform and gauge infrastructural changes made to ROBLOX Battle. Oh, and ROBLOX Battle is now open source. Read on for more.

Making the decision to completely overhaul ROBLOX Battle was challenging–as the Lead of the Games Team, I’m tasked with thinking about the big picture. Whereas Dan worked on gameplay specifically, and Luke was tasked with designing levels, my job was overseeing every single update to ROBLOX Battle. Early on during talks about revamping our flagship title, one thing became abundantly clear to me: if we’re going to do this, we want metrics.

You Bet Your Metrics

ROBLOX is unique as a gaming platform because developers can launch their games and test them directly with players who are eager to provide constructive feedback. I decided early on that we have to revise our approach to making Battle. There are so many avenues for builders, developers and gamers to express their opinions–whether that’s in the comments of the blog, in one of our many forums, or on the site. We are constantly listening. But a sort of truth lies within hardcore data that isn’t muddled by bickering or banter. I decided, before we even started implementing new features, that ROBLOX Battle would be data-driven. 

We started with basic metrics that Stravant (an intern who worked on Battle’s data persistence) and Newtrat (an intern who developed the advanced metrics system) had already been collecting. With their custom tools, I could enter any instance of Battle and track telling statistics, such as average play time for each person and average bounce rate (people who leave the game in less than a minute). A solid game has longer average play times and very small bounce rates. Before Battle’s redesign, its average play time was around six minutes, while its average bounce rate was around 25%. My initial goal was to increase average playtime as much as possible.

Metrics became our focus, and we started tracking everything. A new rule arose among the Games Team: no new game mechanic, level, or feature could be implemented without being able to track all of the metrics. We started answering all sorts of questions. What percentage of users play this new map? How often? What’s the return rate? Most importantly: is this map good enough to be in this game?

I’ll give you an example: at one point, players pretty much unanimously decided that the rocket launcher was too powerful. We’d heard this dissent before on numerous occasions, though now we had metrics to see if this was a legitimate complaint. We wrote a fairly complex script (just over 3,000 lines long) to track weapon metrics, then plotted a graph using the data we collected. This method of tracking data became a super useful tool, and it’s something we’re going to allow builders to utilize when we release the Developers page (whoops, I think I’ve just said too much).

These graphs told us all sorts of useful things–in this particular case, we were able to see that the rocket launcher was indeed too useful in comparison to other Battle weapons. Rather than nerfing (a gaming term for weakening) the weapon, we decided to make every other weapon more powerful. This reduced the effectiveness of the rocket launcher while encouraging players to experiment with other types of deadly weapons. Metrics allowed us to be sure that we made the right decision.

I’m not afraid to say that more than once I’ve thought, “I don’t like this feature, we should consider removing it.” I’ve learned time and time again that my personal opinion is not always in line with those that belong to the players. Because in more than one instant, I’ve had that exact thought, looked at the metrics and realized, “well, users like it.” We want to make the best decisions possible when designing games–and the most concrete place to start is with solid, indisputable data.

Metric-Based Solutions

Building Battle around a data-driven structure helped us address several issues and hiccups along the way. One of the most important charts we were constantly reviewing was server times–we realized at one point that no server lived longer than eight hours. This number was way too low, so we knew there had to be some sort of bug in the system. Upon investigating, we found that some levels weren’t loading properly, so we fixed them. Once we did that, we went back to the numbers to track the server shelf life after fixing the bug–an average of 21 hours. Now we’re talking.

What’s great about ROBLOX from a game-development perspective is that you are never truly done with a game. We’re a dynamic platform–you get your game out there, and constantly iterate on it. How do we base decisions on the next changes we’re going to make? We have weekly design meetings where talk about all aspects of the game. These meetings are like an open-court–everyone gets the chance to voice their opinion and share what changes they think would be most effective.

Alas, not every feature we talk about can be implemented immediately. We’ve learned to consider the cost of making changes. If we want to implement a radical new idea, we ask, “how many resources will that cost?” If the idea would cost two team members and take a single day, we say, “go for it.” If the players like the changes, we keep them. Features that take more time and need more effort from more team members are heavily scrutinized by the entire team before deciding to move forward. It’s all about time allocation and work division.

That’s why we were able to develop ROBLOX Battle so methodically–we tracked metrics after each change was made, whether it was the implementation of new weapons, game types or levels. At the same time, this process took place in enough separate phases as to not scare away our core audience. This method of game development allowed for a nice, easy transition into what actually turned out to be a substantially different experience.

While this series of articles should provide some useful insight for up-and-coming game developers, there’s a lot more knowledge to be gained: ROBLOX Battle is now open source! Feel free to take a copy, and examine the architecture behind it. Create your very own levels, weapons, game modes, and see what works. Try tracking the statistics of players in your game. The possibilities are endless. From a coding perspective, Battle is one of the most complex games on ROBLOX. Please have a look, and find ways to expand the horizons even further–we’re always hanging in and out of the Games Design Forum, so that would be a great place to post your creations!