Monday, 7 September 2020

The games that inspired me most

A few years ago I was asked by a videogame magazine what games inspired me most. I had a lot of fun compiling a short list for them. Today I've updated that list. These aren't necessarily my favourite games (although they're close): Doom II isn't on this list for example. I'm going to focus not on why I had fun with a game, but why it inspired me, how it taught me something new. They all have some unique aspects that changed how I look at games and game development.

Command & Conquer

As a kid I was so much a fan of Command & Conquer that I even rebuilt some of its units with Lego. I started playing around with 3D computer graphics when I was only 13 years old and C&C was my big inspiration. I wanted to make 3D as awesome as in the cut-scenes for this game. Especially the video where a flame tank drives into a town and torches everything is still vividly in my mind. Many of the game designs that I wrote down on paper at this age were for RTS games inspired by C&C. 



Star Control II

This is easily my favourite game of all time. Great music, crazy humour, fun combat and a rich universe with deep backstories and tons of personality. What is most special though is that this is one of the few games I know where information and exploration are intertwined so strongly. Everything is there at the start of the game, but the universe is way too big to find things without knowing the coordinates of interesting planets. Gathering information and even buying it is a key feature here. In combination with the lack of invisible walls this makes exploration extremely rewarding and quite different from most open world games.


An interesting aside is that in the newly released Star Control: Origins (which I really like, by the way) some players complain about the universe being so empty that grinding through planets is boring. The problem here is that modern players have been taught that it's useful to randomly wander around the world, because most open world games these days make sure there are interesting things everywhere. That's fun of course, but that means that in a game like Fallout information isn't very important: by randomly walking around you'll likely find everything (although that does cost a lot of time). Star Control's exploration is totally different because the universe is so much bigger. I think the problem in Star Control: Origins is purely expectations: it isn't clearly communicated to players that this game is different.

Another aspect of Star Control II that has greatly influenced me is the music. Each alien race you encounter has a theme song that captures the personality of that race. Music in many games (as well as movies) mostly serves to set the mood and influence the viewers emotions. But I like music that takes the foreground and really adds to the personality of the game. Inspired by the alien themes from Star Control II, I pushed for having character theme songs in our game Awesomenauts as well. This resulted in Sonic Picnic making an amazingly diverse soundtrack for Awesomenauts, adding tons of personality.

Bioshock

In most games the story and the gameplay are told intertwined but are not truly related. Not in Bioshock: here the gameplay and the story revolve around a common theme. Both add meaningful insights to the concepts of freedom and choice that are explored. Bioshock is not just a marvellously fun game; it also made me ponder the philosophical themes it explores. Especially the twist halfway the game felt incredibly smart and exciting. I felt like I had personally fallen victim to exactly what had happened to the character in-game. One of the few games with true philosophical depth.


World of Goo

World of Goo is the ultimate example of how much creativity is needed to extend a simple and original concept to a full game. It seems to me like many of the participants of gamejams think that coming up with something original is the highest goal in game design. But much more difficult is to develop that original concept from a gimmick into a full game. World of Goo started out as a gimmicky mini-game and ended up as a glorious game. It adds enormous diversity and depth without adding a lot of features or complexity. This is true mastery of game design.


Deus Ex

The original Deus Ex is a great game in general and the combination of RPG + FPS is pure gold for my taste in games. However, the most unique thing in this game to me is the ending. Usually when games have different endings, they are very black-and-white. There are the good ending and the evil ending. Or maybe there were extra challenges and then there are a good ending and a super good ending. Not in Deus Ex: here the three endings all have up- and downsides. It's not clear which would be the best for humanity and the options are clever enough that I spent considerable time pondering each. Deus Ex has similarly deep philosophical thoughts as Bioshock, but while Bioshock is about the game manipulating the player, Deus Ex is about the player manipulating the world.


Dear Esther

A core aspect of game design to me is that a game should be fun without the graphics and the polish. If a game is already fun with just boxes, then it will be even better when visuals and sound are added. Dear Esther is a fantastic example of a game that proves me wrong. In fact, there's hardly any gameplay. But the storytelling, the music and the visuals all add up to an incredible experience. Dear Esther is also interesting in that it suggests a lot and explains very little. I played this game with my wife and we were constantly discussing what it all meant. The game continuously pushed our thoughts into different directions with new snippets of information.


Railroad Tycoon

I am a big fan of trains and always want to see more of them in games. I fondly remember Railroad Tycoon, Transport Tycoon and Sid Meier's Railroads. As a teen I wrote tons of pages of notes on an RTS that resolved around building railroads to transport resources. Trains could be armed to defend them and gather the resources needed to build an army. I doubt that game will ever happen, but anything with trains makes me all giddy with excitement. In fact, most of the game ideas I come up with today still have room for trains somewhere. I hope that in a few years I will finally have made at least one game that actually features trains!


Ori And The Blind Forest & Rayman Origins


I'm grouping these two games together because they have done a similar thing: they have both raised the bar for 2D art in games. 2D art is sometimes seen as lesser than 3D art, but Ori and Rayman have shown that it's possible for a 2D game to have the appeal of a big AAA production. At Ronimo we have studied the animation, special effects and level art in these two games in detail to see what techniques we could apply to our own games.


Braid


Of all the games in this list, Braid has probably influenced me most, and on several levels. For starters, Braid introduced us to the concept of self-publishing and indie development. Xbox Live Arcade had been around for a little while already but Braid, World of Goo and Castle Crashers really showed us the potential of indie and self publishing. When we started Ronimo we thought we needed to make disk based full price games. This was however not a path we were succeeding at so when we realised that we could do what these games were doing, we adjusted our plans and made a smaller game on our own instead. This became our first commercial release: Swords & Soldiers on Nintendo Wii.

Braid also influenced me on a more personal level. My interpretation of Braid's story is that it's not about an actual princess, but about the danger of looking for perfection. The main character has a girlfriend, but he breaks up with her because she isn't perfect. So he looks for the perfect woman, his 'princess'. But he doesn't find her because no one is perfect. The result is that he's alone and doesn't have his actually quite wonderful girlfriend anymore. In the epilogue the game also mentions other examples of the dangers of obsessing over perfection, like a scientist looking for the perfect science but creating the nuclear bomb as part of the Manhattan Project.


I don't know whether the story was truly meant like this, but this interpretation has greatly influenced me in my view on both creativity and life. Perfection is an impossible goal so striving for it only hampers you. Perfectionism causes you to spend too much time looking for an even better game concept to make, or endlessly iterating your game instead of releasing it and making another, or breaking up with your partner instead of making it work. Braid has truly changed my mindset in what to strive for in life and work.

So, that's it: the games that inspired me most! What games changed your view on games or the world?

Sunday, 9 August 2020

Three nasty netcode bugs we fixed around Blightbound's launch

In the weeks before and after the Early Access launch of our new game Blightbound we've been fixing a lot of bugs, including 3 very stubborn netcode bugs. They weren't any fun while we were looking for them for days, but now that we've found and fixed these they're actually quite intriguing. So today I'd like to share 3 of the more interesting bugs I've seen in the past few years.
While these bugs mostly say something about issues in our code, I think they're also nice examples of how complex low level netcode is, and how easy it is to make a mistake that's overlooked for years, until a specific circumstance makes it come to the surface. All three of these bugs were also in the code of our previous game Awesomenauts, but none of them actually occurred there due to the different circumstances of an Awesomenauts match compared to a Blightbound party.

The one thing all three of these bugs have in common is that they were very hard to reproduce. While they happened often enough to be game-breaking, they rarely happened in testing. This made them very hard to find. For the first two we found a way to reproduce them after a lot of experimenting, while for the final one we never got it ourselves until we had figured out the solution and knew exactly how to trigger it. Finding that one without being able to see it was some solid detective work by my colleague Maarten.

Endless loading screens number 1


This bug occasionally caused clients to get stuck in the loading screen forever. After adding additional logging we saw that the client never received the HostReady message. This is a message that the host sends to the clients to let them know that loading is done and they should get into the game. The client was forever waiting for that HostReady message. Upon further investigation it turned out that the host did actually send the message, but the client received it and then threw it away as irrelevant. Why was that? 

To answer that question, I need to give some background information. An issue with the internet is that it's highly unreliable. Packets can be lost altogether and they can also come in out-of-order or come in very late. One odd situation this can cause, is that a message from a previous level can come in once you're already in the next level, especially if loading times are very low. Applying a message from the previous level to the next level can cause all kinds of weird situations, so we need to recognise such an outdated message and throw it away. We do this with a simple level ID that tells us which level the message belongs to. If we get a message with the wrong level ID, we discard it.

That's simple enough, but we saw that both client and host were in the loading screen for the next level. Why then would the client discard the HostReady message as being from the wrong level? The reason turned out to be in our packet ordering system. Some messages we want to process in the order in which they are sent. So if one message goes missing, we store the ones after it and don't process them until the previous missing one is received and handled.

The bug turned out to be in that specific piece of code: the stored messages didn't store their level ID. Instead, when they were released, we used the level ID for the message that had just come in. This went wrong when a message from the previous level went missing and came in a second or two too late. By that time the HostReady message had already been received, but had been stored since it needed to be processed in order. Since the level ID from the previous room was used for all out-of-order messages, the HostReady message was processed as if it came from the previous room. Thus it was discarded.

I imagine reading this might be mightily confusing, so hopefully this scheme helps explain what went wrong:

Once we had figured this out, the fix was simple enough: store the level ID with each message that is stored for later processing, so that we know the correct level ID  when we get to it.

Endless loading screens number 2

The second loading screen bug we had around that time had a similar cause, but in a different way. When a client is done loading, it tells the host so in a ClientReady message. The host needs to wait for that. Similar to the bug I described above, the ClientReady message was received but discarded because it was from the wrong level. The cause however was a different one.

Sometimes the client starts loading before the host, for whatever reason. Since loading times between levels are very quick in Blightbound, the client can be done loading before the host starts loading. Once the client is done loading, it sends the ClientReady message. However, if the host isn't loading yet and is still in the previous room, then it discards the ClientReady message because it's from the wrong room. A moment later the host starts loading and when it's done loading, it starts waiting for the ClientReady message. Which never comes because it was already received (and discarded).

Here too the solution was pretty straightforward. We first changed the level ID to be incremental (instead of random) so that we can see in the level ID whether an incorrect message is from the next or the previous room. If it's from the previous room, we still discard the message. But if the message is from the next room, we store the message until we have also progressed to that room, and then process it.

The big desync stink


The final netcode bug I'd like to discuss today is rather painful, because unlike the others it still existed at launch. In fact, we didn't even know about it until a day after release. Once we knew it existed it still took us days to find it. In the end this one was fixed in update 0.1.2, six days after launch.

So, what was the bug? What players reported to us was that randomly occasionally the connection would be lost. In itself this is something that can happen: the internet is a highly unreliable place. However, it happened way too often, but never when we tried to reproduce it, not even with simulated extremely bad connections. Also, we have code that recognises connection problems, but this bug didn't trigger that code, so the game thought it was still connected and kept playing, with very odd results.

After a few days a pattern started to emerge: many players reported that this always happened once they had notoriety 3 or 4, when a legendary item was attainable. We didn't think that the bug was actually linked to notoriety or legendary items, because these are quite unrelated to netcode. Instead, we suspected something else: maybe the bug happened after playing together for a certain amount of time.

With this hypothesis in hand we started looking for anything where time matters. Maybe a memory leak that grew over time? Or, more likely: maybe some counter that looped incorrectly? As I described in the above issues, netcode can have many types of counters. I already described the level ID counter and the counter for knowing whether a message is received in-order. There are others as well.

The thing with data that is sent over the netwerk is that it needs to be very efficient. We don't want to send any bytes more than necessary. So whenever we send a counter, we use as few bits as possible. Why send a 32 bit number when 16 bits will do? However, when using smaller numbers they will overflow more quickly. For example, a 16 bit number can store values up to 65,535. Once we go beyond that, it loops back to 0.

Often looping back to 0 isn't a problem: the last time we received the lower packets is so long ago that we won't have to expect any overlap with the last time we used number 0. However, we do need to handle the looping numbers correctly and not blindly throw away number 0 because it's lower than number 65,535.

And this indeed turned out to be the problem. One of the counters we used was a 16 bit number and after about one hour it looped back. In that particular spot we had forgotten to include code to handle that situation. The result is that from there on the game received all messages but discarded them without applying them. Like in the cases above, the solution was simple enough to implement once we knew what the problem was.

One question remains: why wasn't this a problem in Awesomenauts? For starters, Awesomenauts matches are shorter and the connection is reset after a match. So the counter never reaches 65,535. In Blightbound however the connection is kept up in between dungeon runs, so if players keep playing together the number will go out of bounds. But there's more: if the bug would have happened in Awesomenauts, the game would have thought it was a disconnect and would have reconnected, resetting the counter in the process. In Blightbound we determine connection problems in a different way, causing the connection to not be reset in this particular case.


A lesson relearned


Netcode being hard is not new to us and getting weird bugs is inherent to coding such complex systems. However, there is one step we did for Awesomenauts that we didn't do here: auto testing. Since Blightbound uses a lot of netcode from Awesomenauts and Awesomenauts has been tested endlessly, we thought we didn't need to do extensive auto testing for this same code in Blightbound. This was a mistake: the circumstances in Blightbound turn out to be different enough that they cause hidden bugs to surface and wreak havoc.

We're currently running autotests for Blightbound to further stabilise the game. This already resulted in another netcode fix in update 0.1.2 and several obscure crash fixes that will be in the next major update.

To conclude I'd like to share this older but still very cool video of auto testing in Awesomenauts. It shows four computers each randomly bashing buttons and joining and leaving matches. The crazy movement of the Nauts is fun to watch. For more details on how we implemented auto testing in Awesomenauts, have a look at this blogpost.




Tuesday, 4 August 2020

Blightbound OUT NOW! Early Access on Steam!

Last Wednesday our new game Blightbound launched on Steam in Early Access! This is a super exciting moment as this is our first new IP to release in 8 years. We've been developing Blightbound for nearly 3 years now and it's really special to finally have the game live and see the community play it! Here's our launch trailer:



We have big plans for this game in Early Access. As the roadmap shows we have a ton of things in development to improve and expand the game. Since this is an Early Access release, there are plenty of fixes and improvements to be made to the game and the first two hotfixes have already released.

Through many years of expanding Awesomenauts we've gained a lot of experience running a live game and responding to the community and we plan on applying all of those lessons here. Robin and I did a talk on Games As A Service a few years back at Steam Dev Days and it's fun to be able to get back to that.

For example, during the first period of Awesomenauts live-ops we didn't really brand our patches and didn't distinguish between fixes and content drops. Here we're doing that right away: hotfixes happen in between and don't contain new content, while major updates get a name and branding and a strong dosis of new stuff to keep players interested and bring players back who had seen it all already. Updates will even come with their own accompanying short stories, written by Roderick Leeuwenhart. The launch story, named "Forging Ahead", is already live on our website and more will follow in the coming period.

Now that the game is out there, I'm also able to write more about it. In the coming period I hope to find the time to talk about how things were built, since there are tons of interesting topics to discuss! We've even already encountered our first set of blog-worthy networking bugs, so I think I'll start there in my next blogpost.

Sunday, 14 June 2020

Blightbound gameplay revealed!

Yesterday during the PC Gaming Show the gameplay for our new game Blightbound was revealed! We've been working on this game for nearly 3 years now, so it's really cool to finally show it to the world.



Blightbound is a multiplayer dungeon crawler that tasks three heroes to venture down from their mountain refuge to face the abominations of the Blight - a mysterious and corrupting fog that enshrouds the land.

I've personally been having a lot of fun with the tech for the characters. Since Awesomenauts characters are sprites they can't be big and they can't be really high framerate: that would just use too much texture memory. We wanted to fix these problems for Blightbound so I've implemented a new system that does allow us to do screen-filling bosses at high framerate. I've also implemented tech for doing real-time lighting on 2D characters with real volume and depth, instead of just lighting them as flat cut-outs as would be the obvious way to light 2D elements in a 3D world.

And that's just a few of the things I've personally worked on. Ronimo is currently 29 people so there are tons of interesting things the rest of the team has been making for Blightbound! This is by far the biggest game we've made so far and I'm really happy with how it's all turning out.

Last month we also released a story trailer:



There's a lot of interesting development stuff so I hope I can find the time to write a series of blogposts on specific aspects of Blightbound development in the future.

Tuesday, 9 June 2020

5 tips for reducing stress as a game company owner

Being a company owner can be a stressful occupation. Your decisions can make the company successful or run it into the ground. There's plenty one can worry about: the game market is constantly changing, so what's the best strategy to follow? How to handle when a co-owner wants something different than you? Mistakes can cost you both your livelihood and your dreams: after all, most people who start a game company do so because they have a passion for doing their own thing. Oh, and if you have employees, you're responsible for their fate as well!

While such topics can indeed cause a lot of stress, I have been relatively relaxed under all of that for the past 13 years. So I was wondering: do I have any mental tricks that help me cope with all of it? Turns out I do! Here are 5 things that I do that help me experience less stress. I hope sharing these can help make entrepreneurship a slightly more chill experience for others as well.

Before I continue I should mention that there are many other factors that determine how stressful it is to run a business. For example, I've always made sure to live slightly cheaper than I could afford so that I could build up savings for cases of economic hardship. This post is not about such financial choices. This post is also not about how a successful game launch or signing a good publishing deal will alleviate stress. Nor is it about raging communities review-bombing your game over a single removed feature or seeing company finances being only a few months from bankruptcy. At Ronimo we've experienced all of these things and much more over the past 13 years and while the things I'll discuss in this post have definitely reduced the stress a lot for me personally, it's been an emotional rollercoaster nevertheless! This post however is about coping with the day-to-day, constant pressure of running a business.

Also, one more side note: I wrote most of this post half a year ago and held back on posting it because one of my colleagues got a burn-out just before I got around to publishing this article. This post isn't about that colleague at all, but it seemed rather insensitive to post anything about stress right after that so I shelved this one. Now that that's a little bit less recent I feel I can post this. In a way it's even more reason to post this, since it shows just how stressful things can be!

1. Figure out why the worst-case is still nice


To me the most important element of being relaxed when I'm afraid things might go wrong, is to figure out why the worst case would actually still be pretty nice. This way regardless of whether something is a success or not, I can always be happy about it. One of the easiest examples for this is that if your game fails to sell well, at least you can be proud of finishing and releasing your own game. That's pretty amazing by itself!

The point where this made the biggest difference for me, was when we started Ronimo. Something I wondered about back then was: what if we would work on a game for a year or two but then can't find a publisher so it would end there? No game released, no income for two years, company gone. That's pretty bad, right? Well, actually, to me back then even that scenario wasn't a nightmare. Making games was (and is) my dream job and making my own games with my friends is one of the best things in the world. In the worst case I would still have done the coolest thing for a year or two. That's pretty neat even if it hadn't been a success!

Another example of this way of thinking is more recent. The owner of another studio told me they were really stressed out over the idea that if their company would go bankrupt, their employees would loose their jobs. That's a huge responsibility and indeed a horrible thing if it happens! However, even here I would argue that the worst case still has a silver lining. If you had never started your company, those people wouldn't have had that job in the first place. Even if it ends, you as the studio owner have provided them with an awesome job for years. You'll have given them a chance to gather a lot of experience that will go on their resume and with which they can get their next job. This may sound crude and even though it stinks if it ends, being able to have done that even for a few years is already an amazing thing to have achieved!

2. Better to make a wrong decision than no decision


An easy trap to fall into when being responsible for big decisions is to analyse them into infinity. "Maybe if we gather some more information, we can make a better choice." "Maybe if we discuss it a bit longer, we'll all agree on the next decision." While it's certainly true that important decisions need to be researched and discussed thoroughly, it's also important to act. Try things. Experiment. Don't get stuck in decision paralysis.

As Asimov wrote in his Foundation series (which just so happen to be among my favourite books):

"To succeed, planning alone is insufficient. One must improvise as well."

Often it's better to try something and see whether it works, than to endlessly think about what the best choice is. In game design this is an obvious rule that most people know: prototype, experiment, test, iterate. What you may not realise is that this same mindset can be applied to business as well. The key here is to not just act, but also constantly evaluate whether what you're doing works. If it doesn't, then change it. Don't be afraid to make mistakes and don't be afraid to admit that you've made a mistake. It's far better to fix it afterwards, than to pretend it didn't happen.

For example, we released many updates to Awesomenauts and often we would debate how to communicate about them. What would be the best marketing strategy for an update? However, far better than that was to simply try things and if it didn't work, try something different with the next update.

Another example is a lot more mundane. At Ronimo our daily lunch with the entire team is a coveted tradition (pro-Corona, that is). However, now that we're growing it becomes impractical to let the entire team lunch at the exact same time. So it was suggested that we should spread the lunch period so that people don't all lunch at the exact same time anymore. Some of the owners of the company (including me) really disliked this idea since it seemed to go pretty strongly against Ronimo's very DNA. However, instead of debating this forever, it was decided to just give it a try and evaluate afterwards. Now that we've tried this, it turns out this not only solves the overcrowded lunch, but also solves the issue that the same cliques formed daily at the lunch table. Now that people are moving in and out of the lunch, where people sit is much less static. Even I now think the spread lunch period is an improvement! If we had tried to make a final decision on this topic before trying it, then I doubt it would have happened at all.

3. Ask lots of advice


Whatever you're doing, someone is bound to have tried something similar before, especially when it comes to the business side of running a game studio. Hearing their experiences is an invaluable source of information for making better decisions yourself.

You may think the games industry is pretty closed off, with all those NDAs (Non-Disclosure Agreements), press embargoes and secretiveness around new games. However, towards fellow devs many game studios are a lot more open. I've often emailed other studios, asking them how they approached something, and often I got really in-depth answers, sometimes even including offers for Skype calls. Of course this is easiest if you know someone at the studio or know someone who can introduce you, but in some cases I've even cold-emailed someone I didn't know at all and still got an insightful answer. And if you think that only goes for indie developers, think again: I know some people at AAA studios and even they were happy to give us advice on topics we struggled with ourselves.

Just keep in mind: be respectful of other people's time and don't go prying just out of curiosity. If you have a real question, ask it and learn from their experiences.

You may wonder how to do something back for people who share their knowledge. Often that's difficult because in many cases the person who gives advice is more experienced than you are or might just not be in need of your own knowledge. In my opinion that's fine. Instead of doing something for them, pay it forward: help someone else when you can and ask nothing in return.

4. Don't feel responsible for everything


A common pitfall when running your own company is that you feel responsible for everything. After all, it's your own company, or at least it partially is. While this may work at first, it's also a great way to get a burn-out, especially as the company grows. Once you employ a dozen people, it's impossible to check all the details of everything they do. Trying to do so costs too much time and distracts you from your own work. You'll simply have to trust them to do their work well and only check on some parts of it.

My own approach to this is that when a new programmer joins our team, during the initial period I review all their code. Then, as they get more experienced with our way of working and thus the amount of feedback I give decreases, there comes a point where I rarely check their code anymore at all. The key in my experience is that while that first period might be time-consuming and maybe even frustrating for both parties, it sets clear expectations as to what I expect in terms of code quality, working method and coding style.

Letting go can be really hard. The boss of another studio recently had a nice way of explaining why you need to do so anyway. He said something along these lines: "Previously I did everything myself. Now I employ dozens of people and don't have the time to code much myself anymore. This was frustrating, until I realised that all those people are working for me. Previously I could spent months making one thing. Now I tell all those people what to work on, and a few months later many things are done. Much more than I could ever do on my own. In a sense, while I don't do those things myself anymore, I've gotten more productive than ever."

5. Accept that things won't always go your way


Most people who start a game company do so with a few co-founders: often there are 2 to 4 founders. In our case we went a little over-the-top and started the company with no less than 7 founders, all with equal ownership rights over Ronimo Games. I've often been asked how we managed to make this work: seven captains on a ship is a whole lot and they're bound to all steer the boat in a different direction. That's bound to produce a lot of friction.

The reason this has worked for us for over a dozen years now, is that having such a big group of founders meant that we immediately ran into disagreements but needed to make decisions anyway. So we decided that if we don't agree on something, we'll just decide by voting. In practice that means that a lot of times we make decisions that I personally don't agree with. Realising this early on also made me accept this early on. Once you've accepted that many decisions will not be exactly what you want them to be, it becomes a lot easier to cope with this.

I expect smaller groups of founders will often have this problem to a lesser degree: it's easier for 3 people to share the same opinion than for 7. However, that also means that when you don't agree, a group of 3 founders might also be more likely to get into a fight over it, resulting in lots of frustration and stress. So I think that even for smaller groups of founders, it's crucial to accept early on that even though it's your own company, it will not always go according to your own plan.

Note that this mindset will also help when dealing with employees. To make talented people shine, you have to give them some room to express their own creativity and ingenuity. This also means occasionally letting something happen even though you disagree with it. As a boss you may technically have the right to force your own decisions on your employees, but sometimes it's better not to.

Regardless of your mindset, there's bound to be stress involved when starting and running your own (game) company. However, the things I've listed in this post have greatly helped me cope with it all and have made me relaxed most of the time. What are your tricks to reduce stress and stay chill?

Sunday, 24 May 2020

Then the Halls Were Empty... and I Turned It On!

I've finished a new song! It's an instrumental that tells a little story. This composition is the centrepiece for my album: here the gate that it's all about is opened for the first time, into who-knows-where. After this the real drama starts: the next song is about how the entire facility needs to be destroyed in order to stop creatures from flooding in!



The first 2 minutes of this song actually started as the soundtrack for a cancelled Ronimo game, all the way back in 2012. I wrote a few short pieces for that and this one made it in. I started by building a rhythm from workshop sound effects, like a tape-measure sliding back in, and then composed the rest from there. While the atmosphere worked really well with our game prototype, the rhythmic sound effects turned out to be problematic: the game also had real sound effects and that didn't combine well. So I simply removed the rhythmic part from my song and then it sounded good in the game. I think this is a beautiful example of how creativity and iteration work: the things you make serve a purpose in the process, even those that get scrapped in the end.

When that game was cancelled, this track went into the fridge. Nevertheless it remained one of my favourite tracks and I came back to listen to it quite often. I wanted to do something with it, but didn't know what. Back then it was only 1:50 minutes long and that felt too short to be 'finished'.

Then I heard Home By The Sea by Genesis. I'm a huge fan of Genesis so it's not rare for me to listen to their music, but this particular time I suddenly realised that I could try doing the same structure for my own song. Home By The Sea is basically two songs in one: it starts as a pop song but halfway it quiets down and then at 5:07 a big beat sets in and it becomes an amazing instrumental. I wanted a similar transformation in my own song! So I tried emulating that vibe, but I failed and it doesn't sound anything like the original. In this case however it did capture something else: I really liked the resulting sound! To me that's another great example of how creativity can work: take an inspiration and then morph it into something completely different to make it your own.

I continued from there, turning it into a 6:25 minutes long piece. That actually makes it my longest composition since I was in a prog-rock band over 15 years ago.

For the title I drew some more inspiration from Genesis. The original title of my song was "And Then The Halls Were Empty", but the more active second part didn't fit that quiet title anymore, so I needed to add something. Genesis has a wonderful pair of songs called "Unquiet Slumbers for the Sleepers..." and "...In That Quiet Earth". I figured I could do something similar with the title of my song, hence the ellipsis (...) in my title. (Note that the second half of the title also sounds a whole lot like another Genesis song, but that's actually a coincidence that I didn't notice until I started writing this blogpost.)

The inspiration for the final bit (starting at 3:44) comes from the song Prelude 1 by Floex and Tom Hodge. Here the result is closer to the inspiration, but I feel my composition is different enough that it's become its own thing. I actually met Floex (Tomáš Dvořák) at a conference a few years ago and was fan-boying to the max, since his soundtrack for Machinarium is one of my favourite game soundtracks. Especially the second half of The Glasshouse With Butterfly is mind-blowingly beautiful. Turns out Tomáš Dvořák is also a very friendly guy to talk to. :)

Since I've started the tradition of creating cello sheet music for all of my songs, I've done so for this one as well. From the perspective of the cello this isn't my most interesting composition, but if you'd like to play the cello parts, you can find sheet music and an MP3 to play along to (without the cello) on music.joostvandongen.com. This website also has sheet music for all my other compositions.

This song actually completes the line-up for my album, which will be called "The Ageless Gate - A Cello Tale". All 13 songs are now complete, making for a grand total of 47:03 minutes of music. The only steps left now are mixing/mastering and making things like a cover and booklet, since I want to make a proper CD out of this one (as well as put it on streaming services like Spotify, of course).

Sunday, 1 March 2020

How we made particles twice as fast through cache optimisation

Cache optimisation is one of the most hardcore, low-level topics in game programming. It's also a topic that I haven't been very successful with myself: I tried optimising for cache efficiency a few times but never managed to achieve a measurable improvement in those specific cases. That's why I was quite intrigued when our colleague Jeroen D. Stout (famous as the developer of the experimental indie game Dinner Date) managed to make our particles twice as fast. What's the trick? Jeroen was kind enough to tell me and today I'll share his approach. Since it's such a hardcore topic, I'll start by giving a general overview of what cache optimisation is in the first place.



Let's have a look at computer memory, or RAM. Computers these days have gigabytes of very fast memory that's used to store whatever the game or program needs to use on the fly. But is that memory truly so fast? Physically, it's a separate part of the computer. Whenever the processor needs to get something from memory, it needs to be transported to the CPU. That only takes a tiny bit of time, but a modern processor can perform billions of operations per second. At those kinds of speeds, a tiny bit of time is actually a lot of operations that the processor could have done during that time. Instead, it's waiting for something to come out of memory.

This is where cache comes in. Cache is a small amount of even faster memory that's directly on the processor and is thus a lot faster to access than normal memory. So whenever something is already in cache, the processor doesn't have to wait for it (or at least not as long) and can keep working. According to this article cache can be up to 27 times faster than main memory.



As programmers we generally don't have direct control over what's in cache: the processor tries to utilise the cache as efficiently as possible by it's own volition. However, depending on how we access memory, we can make it a lot easier for the processor to use cache efficiently. This is where cache optimisation comes in.

The goal of cache optimisation is to structure our code in such a way that we use cache as much as possible and need to wait for memory as little as possible. Whenever the processor needs to get something from memory and it's not already available in cache, that's called a cache miss. Our goal is to avoid cache misses as much as possible.

To be able to avoid cache misses, we need to know a little bit about how cache works. I'm no specialist in this field, but we only need a small bit of knowledge to already understand how a lot of cache optimisations work. These are the basic ideas:
  • Data is transferred from memory to cache in blocks. So whenever you're using something, the memory directly around that is likely also already available in cache.
  • Cache is often not cleared immediately. Things you've just used have a good chance of still being available in cache.
  • The processor tries to predict what you'll use next and get that into cache ahead of time. The more predictable your memory access, the better cache will work.

These three concepts lead to a general rule for writing code that's efficient for cache: memory coherence. If you write your code in such a way that it doesn't bounce all over memory all the time, then it will likely have better performance.

Now that the basics are clear, let's have a look at the specific optimisation that Jeroen did that halved the CPU usage of our particles.



To avoid dumping huge amounts of code here, I'm going to work with a highly simplified particle system. Please ignore any other inefficiencies: I'm focusing purely on the cache misses here. Let's have a look at a very naive approach to implementing a particle system:

struct Particle
{
   Vector3D position;
};

class ParticleSystem
{
   std::vector<Particle*> particles;

   void update()
   {
      for (int i = 0; i < particles.size(); ++i)
      {
         particles[i]->position += speed;
         if (shouldDestroy(particles[i]))
         {
            delete particles[i];
            particles.erase(particles.begin() + i);
            --i;
         }
      }
      while (shouldAddParticle())
      {
         particles.push_back(new Particle{});
      }
   }
};

This code is simple enough, but if we look at it from a cache perspective, it's highly inefficient. The problem is that every time we create a new particle with new, it's placed in a random free spot somewhere in memory. That means that all the particles will be spread out over memory. The line particles[i]->position += speed; is now very slow, because it will result in a cache miss most of the time. The time the processor spends waiting for the particle's position to be read from memory is much greater than the time that simple addition takes.



I knew that this would give performance problems, so I immediately built the particles in a more efficient way. If we know the maximum number of particles a specific particle system can contain, then we can reserve a block of memory for that on startup and use that instead of calling new all the time.

This results in a bit more complex code, since we now need to manage that block of memory and create objects inside it using placement new. In C++, placement new allows us to call a constructor the normal way, but use a block of memory that we already have. This is what the resulting code can look like:

struct Particle
{
   Vector3D position;
};

class ParticleSystem
{
   unsigned char* particleMemory;
   std::vector<Particle*> particles;
   int numUsedParticles;

   ParticleSystem():
      numUsedParticles(0)
   {
      particleMemory = new unsigned char[maxCount * sizeof(Particle)];
      for (unsigned int i = 0; i < maxCount; ++i)
      {
         particles.push_back(reinterpret_cast<Particle*>(
            particleMemory + i * sizeof(Particle)));
      }
   }

   ~ParticleSystem()
   {
      delete [] particleMemory;
   }

   void update()
   {
      for (int i = 0; i < numUsedParticles; ++i)
      {
         particles[i]->position += speed;
         if (shouldDestroy(particles[i]))
         {
            //Call the destructor without releasing memory
            particles[i]->~Particle();

            // Swap to place the dead particle in the last position
            if (i < numUsedParticles - 1)
               swap(particles[i], particles[numUsedParticles - 1]);

            --numUsedParticles;
            --i;
         }
      }
      while (shouldAddParticle() && bufferNotFull())
      {
         //Placement new: constructor without requesting memory
         new(particles[numUsedParticles]) Particle{};
         ++numUsedParticles;
      }
   }
};

Now all the particles are close to each other in memory and there should be much fewer cache misses when iterating over them. Since I built it this way right away, I'm not sure how much performance I actually won, but I assumed it would be pretty efficient this way.

Nope.

Still lots of cache misses.

In comes Jeroen D. Stout, cache optimiser extraordinaire. He wasn't impressed by my attempts at cache optimisation, since he saw in the profiler that the line particles[i]->position += speed; was still taking a disproportionate amount of time, indicating cache misses there.

It took me a while to realise why this is, but the problem is in the swap-line. Whenever a particle is removed, it's swapped with the last one to avoid moving all particles after it one forward. The result however is that as particles are removed, the order in which we go through the particles becomes very random very quickly.



The particles in our example here are really small: just a single Vector3D, so 12 bytes per particle. This means that even if we go through the particles in random order, we might occasionally still be staying in the same cache line. But a real particle in the Secret New Game (tm) that we're developing at Ronimo has much more data, like speed, scale, orientation, rotation speed, vibration, colour, and more. A real particle in our engine is around 130 bytes. Now imagine a particle system with 100 particles in it. That's a block of memory of 13,000 bytes. Hopping through that in random order is much more problematic for cache!

Thus, Jeroen set out to make it so that updating particles goes through memory linearly, not jumping around in memory at all anymore.

Jeroen's idea was to not have any pointers to the particles anymore. Instead we have that big block of memory with all the particles in it, and we store the indices of the first and last currently active particles in it. If a particle dies, we update the range of living particles and mark the particle as dead. If the particle happens to be somewhere in the middle of the list of living particles then we simply leave it there and skip it while updating.

Whenever we create a new particle, we just use the next bit of memory and increment the index of the last living particle. A tricky part here is what to do if we've reached the end of the buffer. For this we consider the memory to be a ring buffer: once we reach the end, we continue at the start, where there's room since those particles will have died by now. (Or, if there's no room there, then we don't create a particle since we don't have room for it.)

The code for this isn't that complicated, but it's a bit too much to post here. Instead, I'm just going to post this scheme:



Compared to my previous version, Jeroen managed to make our particles twice as fast with this approach. That's a pretty spectacular improvement and shows just how important cache optimisation can be to performance!

Interestingly, my first hunch here is that this optimisation is a bad idea since we need to touch all the dead particles in between the living ones to know that they're dead. Worst case, that's a lot of dead particles. In fact, when looking at computational complexity, the worst case has gone from being linear in the number of currently existing particles, to being linear in the number of particles that has ever existed. When I studied computer science at university there was a lot of emphasis on Big O Notation and complexity analysis, so to me this is a big deal.

However, as was mentioned above, accessing cache can be as much as 27 times faster than accessing main memory. Depending on how many dead particles are in the middle, touching all those dead particles can take a lot less time than getting all those cache misses.

In practice, the lifetimes of particles don't vary all that much. For example, realistic particles might randomly live between 1.2 seconds and 1.6 seconds. That's a relatively small variation. In other words: if there are dead particles in between, then the ones before them will also die soon. Thus the number of dead particles that we need to process doesn't become all that big.

The big lesson this has taught me personally, is that cache optimisation can be more important for performance than Big O Notation, despite that my teachers at university talked about Big O all the time and rarely about cache optimisation.

EDIT: Several commenters on Twitter and Reddit suggested an even simpler and faster solution: we can swap the dead particles towards the end of the list. This removes the overhead of needing to skip the dead particles and also removes the memory that's temporarily lost because dead particles are in the middle. It also simplifies the code as we don't need to handle that ring buffer anymore.

For further reading, I would like to again recommend this article, as it explains the principles behind cache lines very well and shows some clever tricks for optimising to reduce cache misses.

As a final note, I would like to also stress the unimportance of cache optimisations. Computers these days are super fast, so for the vast majority of code, it's fine if it has very poor cache behaviour. In fact, if you're making a small indie game, then it's probably fine to ignore cache misses altogether: the computer is likely fast enough to run your game anyway, or with only simpler optimisations. As your game becomes bigger and more complex, optimisations become more important. Cache optimisations generally make code more complex and less readable, which is something we should avoid. So, use cache optimisations only where they're actually needed.

Have you ever successfully pulled off any cache optimisations? Are there any clever tricks that you used? Please share in the comments!