First steps towards massive replays and spectators in Awesomenauts

One of the biggest challenges for us in the coming period is implementing replays and spectator mode in Awesomenauts. It might not be immediately obvious, but this is actually an incredibly complex problem to solve. We have the very first bits working now, so here is a video in which I show and explain what we have so far:



Our first target with replays is to get them working offline, so you store replays of your own matches on your own harddisk and you can email those files to your friends by hand. Storing and browsing replays through online servers comes after that and is still a long way away. We hope that the offline replays can go in beta before Christmas, but right now it seems like that might still be too early for how complex this all is.

So why are replays so difficult to build? There are a ton of reasons for this. One is that players need to be able to do live spectating. In high-level tournament matches, we currently sometimes get near to 1000 simultaneous viewers for a single match. This is on Twitch, so the bandwidth is handled by Twitch, but for our spectator mode we will need to make that possible ourselves. Handling that kind of user numbers and internet traffic is by itself already a big challenge.

Another problem is that players need to somehow get their replays to a server, live while playing. By itself this is not very difficult, but to keep this from disrupting normal online play, we need to do this in an extremely small amount of bandwidth. We are targeting to stream out a replay from one player to the server in only 1 to 2 kilobyte per second. The total size of a 20 minute match will then be around 2mb. Streaming all data in so few bits is an interesting challenge, but I think it is quite achievable.

Another problem is version management. We release lots of patches for Awesomenauts, and most patches change significant things in our gameplay logic. That means that if we rely on that gameplay logic for playing back replays, then stored replays will often not be playable anymore once we release a patch. A solution for this would be to add version management to the gameplay code, but that would add a kind a code complexity that we definitely don't want there. And I haven't even mentioned how sensitive to bugs that would be... Another solution would be to store legacy versions of the game to play older replays, but that would produce very high loading times when switching between reviews, and would mean loads of downloading to get all those older game versions.

Our solution for this version problem is to store only graphical data in the replay. This way if gameplay logic changes, it doesn't matter for stored replays, since those don't use the gameplay logic anyway. The demo shown above already works like this. However, this does mean that we cannot use any of the current gameplay code or network packages for the replays. We need to rebuild it all from the ground up in a different way. That is why the demo above still lacks animations and bullets and pickups and such: they have not been rebuilt for replays yet.

Another issue in the replays implementation is server bandwidth and loading times. If players view lots of replays, this will cost us a lot of bandwidth and thus a lot of money. Also, if players need to download the entire replay before they can watch it, then waiting times might be long. We even want players to be able to immediately skip to specific moments in the replay without downloading everything in between. With this in mind, the replay system again becomes a lot more complex.

There are several other challenges in implementing replays and spectator mode, but these are the most important. The video shown actually solves the basics of several of those already: there is quite a bit of interesting code architecture behind it. This is also why it took so long before we got anything to show at all: storing and playing character positions is by itself not all that difficult, but doing it in a way that will work with the above issues makes it a lot more time-consuming to implement. I expect I will be able to do a great talk or series of blogposts on this once the whole replays system has been built...

As a conclusion to this blogpost, I would like to say that this is a mightily interesting and fun challenge! We at the Ronimo coding team are greatly enjoying this task: coding puzzles are the best puzzles!