Issue #3 - Riot dev teams want to work in harmony

League (and also Riot) has grown hugely since 2008 and we could never have guessed that we’d end up with so many teams wanting to add so many features to the client. The original codebase was structured around a small, highly cohesive team and didn’t give the required level of autonomy and independence as the number of features teams grew.

This problem of dev teams colliding with each other wasn’t going away—it was getting worse over time as we added new features. In addition, we aspire to be a multi-game studio, which brings many other challenges and opportunities.

Incrementing towards an awesome architecture

There are a number of ways to solve for the above issues—for instance, we spent some serious time reorganizing the existing AIR code with the intention of staying on the (outdated) AIR browser. However, while exploring this option in depth, we realized that we could solve the above issues in a more powerful and sustainable way by changing our platform. We decided to do this despite being completely aware of the dangers of rewrites, because we believe the benefits to the player will outweigh the risks in the long-run.

As such, we chose to start again and build around Chromium Embedded Framework (CEF) as our front-end. This gives us a super strong HTML5 browser component that we can easily customize.

Let me now explain how we found our way to our final architecture.

Step A: JavaScript is the king of the world!

Our initial idea was to do all the work in JavaScript.

Since we were planning on building the UI using HTML and JavaScript, initially we felt that putting all the business and communications logic in JavaScript also would simplify the architecture and keep it uniform. (The impulse for this type of uniformity between UI and services lies at the heart of platforms like node.js.)

As such, we developed a simple and minimal C++ library allowing JavaScript to invoke remote calls via RTMP and handle asynchronous responses. We kept RTMP because it works well for our current scale, and because we didn’t want to change the communications protocol at the same time as changing the client.

Based on some internal prototyping, we decided to use ember.js as our single page application framework, and ember-orbit as our data layer.

What could possibly go wrong with something so straightforward? Well, quite a bit as it turned out.

Our JavaScript code became extremely complex due to the fact that we were handling all the asynchronicity in the web tier. Further, player state was being kept in JavaScript also, meaning that we couldn’t easily minimize down to a low memory footprint.

So, this architecture solved issue #1, but did nothing for issues #2 or #3. An internal developer satisfaction survey found that developing in this way was less productive than the AIR client. Ouch.

Step B: We rediscovered web apps (but on the desktop)!

Our next step was to provide microservices in C++, presenting the async game protocol as a set of REST resources.

We started thinking about how normal web applications are designed, and we realized we were missing a middle tier. We built a microservice layer (still running on the player’s desktop) to present the RTMP protocol as REST resources, to insulate the JavaScript from so much async. We used websockets to handle events back to the UI. We called this microservices layer the ”foundation.”

Below is a picture showing the Swagger UI for some of the foundation patcher resources. This arrangement greatly simplified the JavaScript code, and the devs were now able to use standard web techniques.

We also discovered an additional benefit of this architecture: when we minimized the client, we were able to kill the CEF UI (and all JavaScript VMs), leaving just the foundation. This was only possible because the foundation keeps canonical state—the entire UI can be reconstituted using GETs. The foundation only takes up around 20mb in memory, which is roughly the size of a few internet cat pictures. No one should ever need to kill the client again after getting in-game (as some players currently do to save memory).

This allowed us to start building towards the vision of an "always available" client, which addresses issue #2. The foundation can bring up a system tray notification, even in the background, to indicate a game invite or a friend request.

Step C: Still stepping on toes…

The next evolution was to add an extensibility architecture.

As teams started developing microservices and features, a new problem arose. These teams were often updating the same code, and colliding with each other.

Let me use an analogy to make the point about extensibility. Imagine if ten people were asked to write a single book. Suppose we asked them all to work on each chapter together (e.g. chapter two: “Playing League as a beginner”). It would be a nightmare because they would literally all be trying to change one another's sentences. Instead, it makes sense to give them each a single chapter to write (e.g. chapter five: “Masteries” and chapter six: “Runes”), allowing them to work independently on the same book. Of course, the challenge now is to make the book read cohesively.

We faced a similar challenge on the client update. Teams were adding their features, but too often they had to adjust shared code and frameworks. We needed an architectural pattern to keep them as independent as possible.

I’ve spent a lot of time working on extensibility architectures and I know how nicely they solve the developer collision problem. To keep our architecture simple we chose a variant on a tried and tested extensibility approach called plugins. A plugin is an independent unit of ownership, development, testing, and deployment. Each plugin respects semver and must indicate which other plugins and versions it depends on, which keeps the architecture explicit. We augmented this style with a fully explicit dependency graph, indicating how all the plugins were connected.

So, a UI screen or feature goes into what we call a front-end (FE) plugin, and C++ communications logic goes into a (somewhat confusingly named) back-end (BE) plugin. Note that both types of plugin live inside the new client - we used the terms front and back-end because the architecture mimics a typical client-server architecture, even if it is inside a single desktop process.

We then allow the set of plugins chosen for a player to be based on what entitlements that player has. Entitlements are used to control regional-specific features, and will also be used to give access to new games.

In the process of moving to this architecture, we had to unpack our monolithic JavaScript data layer. We were using ember-orbit to maintain a single connected graph of objects, representing our REST resources. This was a big source of headaches for us, as it still meant that other teams were able to access each other’s data via object links rather than service calls. This was way too much coupling.

So, we retired our use of Orbit and used a simple resource cache owned by each front-end plugin. Doing all this retired a metric shedload of complexity.

Step D: Ask N JavaScript developers for their favorite MVC framework. Get N+1 answers.

We allowed each feature team to choose their own JavaScript framework.

We now had a much more productive architecture and teams were able to make screens and services quickly, with clear separation of responsibilities.

However, we had a developer problem. Some of the teams really didn’t like Ember.js and wanted to use web components directly. Other teams already had their own web apps written in a variety of frameworks that they wished to host inside our new client.

At this point we were still reluctant to relax the "Ember.js for all” rule just for developer preference. Removing the assumptions related to Ember as the single embedded framework was going to take some serious work.

The turning point came when we realized that a client platform for a company the size of Riot cannot even force everyone onto the same version of Ember. It was painful, but we ended up architecting the client to allow each plugin to potentially have its own framework—the JavaScript framework became part of the domain of each front-end plugin.

Note that the key word above is potentially. We still use Ember for the majority of our features for consistency purposes, but this is no longer technically required.

The final architecture

The target architecture for the updated client now addresses our three original issues well. It’s an engine which allows multiple teams to independently deliver HTML5-based features without unnecessary dependencies, using supporting communications infrastructure to allow players to remain connected. It’s effectively a hosting engine for C++ microservices and JavaScript web apps, where the choice of plugins is personalized and dynamic, based on a player’s entitlements.

We repeated the developer satisfaction survey mentioned above, and found that the situation had improved by around 15% and was rising quickly. Nice!

One of the developers commented that we’d made a "datacenter in a desktop." From another perspective, I realized we'd created the game client equivalent of Atom, Github’s CEF-based extensible text editor based on the Electron framework. Interesting stuff, and I would never have guessed that we'd end up here at the start.

There are a lot of questions that this post hasn’t explicitly addressed, such as how we can ensure that features interact seamlessly and don't look stitched together, how we get game quality effects using HTML5, and how we allow for the independent deployment of features. These are fascinating areas in their own right, and we're happy to talk more about some of these things if there's a lot of interest. Hit us up in the comments!

© 2016 Riot Games, Inc. All rights reserved. League of Legends and Riot Games are trademarks, service marks, or registered trademarks of Riot Games, Inc.