The skill design system in Awesomenauts

(Note: Swords & Soldiers is currently on sale on Steam! 50% discount! ^_^ )

At Ronimo Games we always try to make innovative games, and in my opinion one of the biggest challenges when doing so is how to enable the game designers to do their work. Game designers work with things like game mechanics and control schemes, but you cannot know whether what you designed on paper is a good idea, until you played it and experimented with. It requires programming to create something playable, so unless a game designer is also a programmer, he cannot really do much on his own.

For artists, this problem has pretty much been solved: graphics tools are so powerful that artists can create most things without need of a programmer (except maybe for polish and performance). For game design however, this is essentially an open problem. There are many methods to empower game designers, but none are ideal. To name a few: scripting lanuages, event based systems, tools like Gamemaker and Virtools, and the most used of them all: just copying everything from existing games and not even trying to be innovative in any way (this also solves a lot of other issues, like the problem of being creative and taking pride in your work).

The approach we use to enabling game designers to create stuff is not ideal either, but it does work really well for us, so I'd like to write a couple of blog posts about the systems that allow our game designers to create game mechanics. Today is the first, and it is about our skill system.

To us it all starts with the coders and the game designers working together closely. Now whenever a programmer builds something, we set it up as flexible as possible, and make settings to control it. Things like speed, size, health, cooldowns and whatever else we can think of are put in settings and our game designers can edit those. Last year I wrote a post about our settings system in Swords & Soldiers that explains the basic idea further.

However, the way we approached this in Swords & Soldiers was also quite limited. All actual behaviour was programmed specifically per unit, so the only thing our designers could do, was tweak them. If they wanted to create a new unit, or even combine properties from existing units, they had to ask the programmer (me) to implement this for them.



In Awesomenauts we set out to make things more flexible. A character in Awesomenauts mainly consists of a list of skills, and which button needs to be pressed to use that skill. We have programmed around 20 skills (things like Shoot, Hit, Jump, Dash and Shield), and designers can create new versions of each skill with separate settings. Designers can also attach several skills to one button to create combinations, so maybe there is a combo skill that shoots a bullet and starts a shield at the same time.



In this trailer you can see how Sheriff Lonestar's set of skills works out in the actual game:



Each of the skills refer to a group of settings that define the exact properties of that skill. So there can be many different Shoot skills, each with different settings. This way some become grenades, other bullets, lasers or mines, and some even spawn new creeps.



Now this is really flexible, and I guess it is quite obvious that building it like this works much better than hardcoding the entire character as we did in Swords & Soldiers. The trick to making this really flexible, though, is to make lots of settings. Throughout development of Awesomenauts, we have constantly been adding new settings to make the existing skills more flexible, so that more things could be made with them.

However, it may seem so, but working like this is not an Obvious Good Idea (tm). It has some serious downsides as well, which all boil down to one thing: skills can be combined in any way and this makes all character code infinitely more complex. If you have 20 skills, then designers can already make 400 combinations of two skills happening at the same time. You can see where this goes once every character has 10 skills...

So implementing this system cost us an enormous amount of time to get all the combinations to behave correctly. A good example of the complexity here is animations. How to choose which animation to play when several skills are active at the same time? There are so many combinations that we kept bumping into situations where an animation did not look good or work correctly because some other skill interrupted it in the wrong way. This is very solid in the final game, but it took us a lot of work to get there. Unfortunately, it is still very fragile code (i.e. it is easy to introduce bugs when working with it).

Another downside of this system is that if you want it to be really flexible, then you need dozens of settings for each skill. The more settings there are, the more difficult it becomes for the game designer to keep track of what each does exactly. During development it happened quite often that our designers asked for a feature that was already there.

Of course, the upside of this is that with so many settings, there are many combinations that achieve totally unique things. Especially the Shoot and Shield skills have been applied by our game designers to do totally different things than what I expected when we programmed them.

So choosing between the flexibility in Awesomenauts and the very inflexible system in Swords & Soldiers, I actually don't think the flexible system always wins. I would even say the system in Awesomenauts might be up to three times as much work.

However, if, like we did in Awesomenauts, you want to spend a lot of time iterating on the game design to find the most fun and varied units, then more flexibility is definitely necessary. So I think we made the right choice for Awesomenauts. It is a choice that cost us a lot of development time, but that also improved the game's quality enormously.

The lamest bug we ever encountered

Yesterday we resolved by far the lamest 'bug' I have ever encountered. We spent almost two whole days with the entire programming team (5 coders) trying to find it, and once we discovered what was happening, it felt so incredibly lame, that I figured it would be a nice story to share. So here's what happened!

In the past half year we have seen a really rare bug occur about once per month in Awesomenauts on the Playstation 3: suddenly the game would freeze for anywhere between 10 to 100 seconds, and then continue normally as if nothing weird had happened. We always have a PC connected to collect logs from the game. However, the log printed nothing interesting and showed simply the framerate, which is printed once per second:

13:48:13 60fps
13:48:14 60fps
13:49:21 1fps
13:49:22 60fps

From this we concluded that the issue must be in some part of code that does not log anything. That leaves 99% and with a codebase of at least 150,000 lines, that is a lot!

Since this issue was so rare and we hadn't seen it in months, we had closed it. We thought it was an anomaly that we couldn't find or fix and that had somehow disappeared.

Until we were testing our latest build last Thursday. We were playing on seven Playstation 3 devkits, when suddenly 4 of them froze for about 90 seconds. Argh! We thought this really difficult bug had magically disappeared, and now it was back, with a vengeance! We cannot leave a bug in that happens so massively on all these consoles at once!

Again, no relevant logging. Yet something really interesting nevertheless: these Playstations were in different online matches, so they weren't even talking to each other! So how could they all freeze at the exact same time? We concluded that the only thing they had in common, was that they all talk to the same Sony matchmaking servers, so we started investigating all our code related to that. However, we couldn't find anything relevant.

So we added a lot more logging, and did some really advanced stuff to get more info on the stack during the freeze (which is difficult to get from an executable that has been stripped of all debugging info), and started playing again. I let the Playstations perform our automatic testing all night, and the bug didn't happen. Then we played the game for five more hours with the entire team and BAM!, it finally happened again on two consoles!

This time, we had more info, and it turned out that the spot where the game froze was different on both freezing consoles, and both did not contain any calls to Sony's matchmaking servers. In fact, it was in between two logging calls, in a spot where nothing relevant was happening. So we concluded there were only two possible causes: either other threads were hogging the entire CPU (due to how the scheduling system on the Playstation 3 works, high priority threads can do this permanently), or the logging itself was broken.

So we started experimenting around that, and now we quickly found the cause of this 'bug': when the PC that is tracking logs goes into sleep mode, the connected consoles freeze a little while later; once the PC is awoken once more, the consoles continue as well a little later. The PC that was tracking the logs automatically went into sleep mode after not touching it for 30 minutes. This only happened during extensive play-testing, because people normally actually use that PC. So it wasn't even a bug in our code! ARGH! Especially since logging is turned off in the release build...

This may all seem really obvious in hindsight, but in general when we have a bug/freeze/crash it is in our own code, not in one of the tools we use. With such a big codebase, it is easy to not even think about something else. Also, in the chaos of 14 people playing the game on 7 consoles, it is easy to overlook one specific PC going into Sleep mode a little bit before the consoles freeze... Also, I still don't know why not all consoles connected to that PC froze.

So, this hard to find and very persistent bug turned out not to even be a bug! And what do we learn from this? Two things:
1. Really nasty bugs are often in a totally different spot than where you are looking.
2. Sometimes you encounter the lamest and most frustrating stuff while programming games!

QQQ

In his talk at Quakecon this year, legendary programmer John Carmack (Keen, Wolfenstein, Doom, Quake, Rage) talked about code quality, and how even the very best programmers make lots of mistakes. So he says that every programmer should think about ways and methods that make mistakes happen less. This is something that I totally agree with, and this kind of thinking is the basis of the coding standard and methodology that we use at Ronimo Games.

One of the nicer things that have crept into our coding standard over the years is a tiny little trick that keeps a lot of issues from happening. It is pretty obvious, but I know a lot of programmers who don't do something like this, so I figured it would be useful to mention this one today. :)

Often when you are programming something, you quickly make some changes somewhere else to test one specific situation. For example, maybe you are testing the graphics of a weapon against a specific enemy. If that enemy dies all the time, you need to find another one to test on, so a quick change could be to decrease the damage of that weapon against that specific enemy by 90%. Now the enemy stays around a lot longer and you can test more efficiently.

However, once you are done working on that weapon, it is really easy to forget to remove your little hack. Later someone else spends hours searching through code why that enemy dies so slowly. Only to discover that forgotten hack of yours.

Many coders would say now that this is sloppy programming and you should just be a better programmer and remember these things. However, while programming, there are so many things to consider that it is easy to forget one once in a while, no matter how good you are. It is better to design your working method around the idea that you do make mistakes, then to simply say you should be a better programmer.

Now since these little hacks are a great help when programming, everyone I know uses them a lot. So I came up with a really simple trick to keep track of them. Whenever you do a little hack like that, you add a comment with the text "QQQ". So like this:

if (enemy->getName() == "UrQuan") damage *= 0.1f; //QQQ

All coders at Ronimo do this ('voluntarily'). Before we commit our newly written code to the source server, we search for QQQ to see whether we forgot anything. That gives us a list of all the temporary hacks that are still in there. And because we all use QQQ, and not something different for each programmer, we can quickly find QQQs that someone else forgot to remove.

Since QQQ is basically just a marker to remember something, I also use it when I realise I need to remember to rework or check something later on.

So why the letters "QQQ"? This specific term has the benefit that it never occurs in any normal sentence, and is short enough to quickly put in while working. I actually stole it from my brother and sister, who are both researchers and put it in their texts when they need to fill something in later.