Building hundreds of upgrades for Awesomenauts

In Awesomenauts, players can buy hundreds of different upgrades. There are around 150 unique ones, and most have two or three upgrade levels, so in total there are probably around 300 upgrades in the game. We wanted our designers to be able to create those themselves, without any work from the programming team for specific upgrades. At the same time, we also wanted the upgrade system to be super flexible, so that the upgrades would be very diverse, and not just all be cooldown reductions and damage increases.



When we first started looking for a way to build this, I was really at a loss. How to add a flexible system for upgrades to our settings system? It took some time, but coding intern Daan and I came up with a system that is easy to use and so flexible, that we ended up not just using it for upgrades, but also for temporary boosts and areas with modified mechanics, like low-gravity areas.



The idea is that for every setting, our designers can create modified versions for when a character has certain upgrades. Each setting always has one base value. Then as many modified versions can be added as desired, simply by copying the setting and adding @ and the name of the upgrade.



The @ sign works for many things, but it has the problem that it is problematic to have different upgrades for the same setting: what should happen if you have both upgrades? So we also introduced the + sign, which means that that upgrade's value is added to the original value. This way several upgrades can influence the same value.



This system does not just allow for simple upgrades that upgrade a single value, but can also be used for more complex stuff. Especially since our designers can give a player a temporary upgrade when he presses a button. So let's say we want to create a skill that temporarily makes the player deal twice as much damage, but also make him half as fast and twice as big. A designer could simply use the name of a single upgrade to modify all of these values at the same time.

Of course, the more flexible and complex things get, the easier it is for our designers to mess up. We have had quite a few bugs where certain combinations of upgrades behaved in a different way than our designer intended, simply because they chose the wrong values. The plus side of these kinds of bugs is that our designers can fix them themselves, so as a programmer I am okay with that... ;)

The upgrade system also gives our artists a lot of flexibility. For example, if a bullet does more damage, our artists can upgrade its graphic and its explosion to use a different animation.



The upgrade system described in this blogpost looks pretty obvious to me now, but it definitely was a revelation to us when we came up with it two years ago, and I still think it is the most elegant solution we came up with for any problem at Ronimo Games so far (although I suppose this system is not unique and other RPGs probably use something very similar). It is incredibly flexible and really easy to use, and allows our game designers to make almost any upgrade they want. And the good thing is that it is not specific to any games, so anything we make with the Ronitech (our engine) will have this feature for our designers! ^_^

Next week, I'll explain the technical side of how we made this upgrade work in C++!

The skill design system in Awesomenauts

(Note: Swords & Soldiers is currently on sale on Steam! 50% discount! ^_^ )

At Ronimo Games we always try to make innovative games, and in my opinion one of the biggest challenges when doing so is how to enable the game designers to do their work. Game designers work with things like game mechanics and control schemes, but you cannot know whether what you designed on paper is a good idea, until you played it and experimented with. It requires programming to create something playable, so unless a game designer is also a programmer, he cannot really do much on his own.

For artists, this problem has pretty much been solved: graphics tools are so powerful that artists can create most things without need of a programmer (except maybe for polish and performance). For game design however, this is essentially an open problem. There are many methods to empower game designers, but none are ideal. To name a few: scripting lanuages, event based systems, tools like Gamemaker and Virtools, and the most used of them all: just copying everything from existing games and not even trying to be innovative in any way (this also solves a lot of other issues, like the problem of being creative and taking pride in your work).

The approach we use to enabling game designers to create stuff is not ideal either, but it does work really well for us, so I'd like to write a couple of blog posts about the systems that allow our game designers to create game mechanics. Today is the first, and it is about our skill system.

To us it all starts with the coders and the game designers working together closely. Now whenever a programmer builds something, we set it up as flexible as possible, and make settings to control it. Things like speed, size, health, cooldowns and whatever else we can think of are put in settings and our game designers can edit those. Last year I wrote a post about our settings system in Swords & Soldiers that explains the basic idea further.

However, the way we approached this in Swords & Soldiers was also quite limited. All actual behaviour was programmed specifically per unit, so the only thing our designers could do, was tweak them. If they wanted to create a new unit, or even combine properties from existing units, they had to ask the programmer (me) to implement this for them.



In Awesomenauts we set out to make things more flexible. A character in Awesomenauts mainly consists of a list of skills, and which button needs to be pressed to use that skill. We have programmed around 20 skills (things like Shoot, Hit, Jump, Dash and Shield), and designers can create new versions of each skill with separate settings. Designers can also attach several skills to one button to create combinations, so maybe there is a combo skill that shoots a bullet and starts a shield at the same time.



In this trailer you can see how Sheriff Lonestar's set of skills works out in the actual game:



Each of the skills refer to a group of settings that define the exact properties of that skill. So there can be many different Shoot skills, each with different settings. This way some become grenades, other bullets, lasers or mines, and some even spawn new creeps.



Now this is really flexible, and I guess it is quite obvious that building it like this works much better than hardcoding the entire character as we did in Swords & Soldiers. The trick to making this really flexible, though, is to make lots of settings. Throughout development of Awesomenauts, we have constantly been adding new settings to make the existing skills more flexible, so that more things could be made with them.

However, it may seem so, but working like this is not an Obvious Good Idea (tm). It has some serious downsides as well, which all boil down to one thing: skills can be combined in any way and this makes all character code infinitely more complex. If you have 20 skills, then designers can already make 400 combinations of two skills happening at the same time. You can see where this goes once every character has 10 skills...

So implementing this system cost us an enormous amount of time to get all the combinations to behave correctly. A good example of the complexity here is animations. How to choose which animation to play when several skills are active at the same time? There are so many combinations that we kept bumping into situations where an animation did not look good or work correctly because some other skill interrupted it in the wrong way. This is very solid in the final game, but it took us a lot of work to get there. Unfortunately, it is still very fragile code (i.e. it is easy to introduce bugs when working with it).

Another downside of this system is that if you want it to be really flexible, then you need dozens of settings for each skill. The more settings there are, the more difficult it becomes for the game designer to keep track of what each does exactly. During development it happened quite often that our designers asked for a feature that was already there.

Of course, the upside of this is that with so many settings, there are many combinations that achieve totally unique things. Especially the Shoot and Shield skills have been applied by our game designers to do totally different things than what I expected when we programmed them.

So choosing between the flexibility in Awesomenauts and the very inflexible system in Swords & Soldiers, I actually don't think the flexible system always wins. I would even say the system in Awesomenauts might be up to three times as much work.

However, if, like we did in Awesomenauts, you want to spend a lot of time iterating on the game design to find the most fun and varied units, then more flexibility is definitely necessary. So I think we made the right choice for Awesomenauts. It is a choice that cost us a lot of development time, but that also improved the game's quality enormously.

The lamest bug we ever encountered

Yesterday we resolved by far the lamest 'bug' I have ever encountered. We spent almost two whole days with the entire programming team (5 coders) trying to find it, and once we discovered what was happening, it felt so incredibly lame, that I figured it would be a nice story to share. So here's what happened!

In the past half year we have seen a really rare bug occur about once per month in Awesomenauts on the Playstation 3: suddenly the game would freeze for anywhere between 10 to 100 seconds, and then continue normally as if nothing weird had happened. We always have a PC connected to collect logs from the game. However, the log printed nothing interesting and showed simply the framerate, which is printed once per second:

13:48:13 60fps
13:48:14 60fps
13:49:21 1fps
13:49:22 60fps

From this we concluded that the issue must be in some part of code that does not log anything. That leaves 99% and with a codebase of at least 150,000 lines, that is a lot!

Since this issue was so rare and we hadn't seen it in months, we had closed it. We thought it was an anomaly that we couldn't find or fix and that had somehow disappeared.

Until we were testing our latest build last Thursday. We were playing on seven Playstation 3 devkits, when suddenly 4 of them froze for about 90 seconds. Argh! We thought this really difficult bug had magically disappeared, and now it was back, with a vengeance! We cannot leave a bug in that happens so massively on all these consoles at once!

Again, no relevant logging. Yet something really interesting nevertheless: these Playstations were in different online matches, so they weren't even talking to each other! So how could they all freeze at the exact same time? We concluded that the only thing they had in common, was that they all talk to the same Sony matchmaking servers, so we started investigating all our code related to that. However, we couldn't find anything relevant.

So we added a lot more logging, and did some really advanced stuff to get more info on the stack during the freeze (which is difficult to get from an executable that has been stripped of all debugging info), and started playing again. I let the Playstations perform our automatic testing all night, and the bug didn't happen. Then we played the game for five more hours with the entire team and BAM!, it finally happened again on two consoles!

This time, we had more info, and it turned out that the spot where the game froze was different on both freezing consoles, and both did not contain any calls to Sony's matchmaking servers. In fact, it was in between two logging calls, in a spot where nothing relevant was happening. So we concluded there were only two possible causes: either other threads were hogging the entire CPU (due to how the scheduling system on the Playstation 3 works, high priority threads can do this permanently), or the logging itself was broken.

So we started experimenting around that, and now we quickly found the cause of this 'bug': when the PC that is tracking logs goes into sleep mode, the connected consoles freeze a little while later; once the PC is awoken once more, the consoles continue as well a little later. The PC that was tracking the logs automatically went into sleep mode after not touching it for 30 minutes. This only happened during extensive play-testing, because people normally actually use that PC. So it wasn't even a bug in our code! ARGH! Especially since logging is turned off in the release build...

This may all seem really obvious in hindsight, but in general when we have a bug/freeze/crash it is in our own code, not in one of the tools we use. With such a big codebase, it is easy to not even think about something else. Also, in the chaos of 14 people playing the game on 7 consoles, it is easy to overlook one specific PC going into Sleep mode a little bit before the consoles freeze... Also, I still don't know why not all consoles connected to that PC froze.

So, this hard to find and very persistent bug turned out not to even be a bug! And what do we learn from this? Two things:
1. Really nasty bugs are often in a totally different spot than where you are looking.
2. Sometimes you encounter the lamest and most frustrating stuff while programming games!

QQQ

In his talk at Quakecon this year, legendary programmer John Carmack (Keen, Wolfenstein, Doom, Quake, Rage) talked about code quality, and how even the very best programmers make lots of mistakes. So he says that every programmer should think about ways and methods that make mistakes happen less. This is something that I totally agree with, and this kind of thinking is the basis of the coding standard and methodology that we use at Ronimo Games.

One of the nicer things that have crept into our coding standard over the years is a tiny little trick that keeps a lot of issues from happening. It is pretty obvious, but I know a lot of programmers who don't do something like this, so I figured it would be useful to mention this one today. :)

Often when you are programming something, you quickly make some changes somewhere else to test one specific situation. For example, maybe you are testing the graphics of a weapon against a specific enemy. If that enemy dies all the time, you need to find another one to test on, so a quick change could be to decrease the damage of that weapon against that specific enemy by 90%. Now the enemy stays around a lot longer and you can test more efficiently.

However, once you are done working on that weapon, it is really easy to forget to remove your little hack. Later someone else spends hours searching through code why that enemy dies so slowly. Only to discover that forgotten hack of yours.

Many coders would say now that this is sloppy programming and you should just be a better programmer and remember these things. However, while programming, there are so many things to consider that it is easy to forget one once in a while, no matter how good you are. It is better to design your working method around the idea that you do make mistakes, then to simply say you should be a better programmer.

Now since these little hacks are a great help when programming, everyone I know uses them a lot. So I came up with a really simple trick to keep track of them. Whenever you do a little hack like that, you add a comment with the text "QQQ". So like this:

if (enemy->getName() == "UrQuan") damage *= 0.1f; //QQQ

All coders at Ronimo do this ('voluntarily'). Before we commit our newly written code to the source server, we search for QQQ to see whether we forgot anything. That gives us a list of all the temporary hacks that are still in there. And because we all use QQQ, and not something different for each programmer, we can quickly find QQQs that someone else forgot to remove.

Since QQQ is basically just a marker to remember something, I also use it when I realise I need to remember to rework or check something later on.

So why the letters "QQQ"? This specific term has the benefit that it never occurs in any normal sentence, and is short enough to quickly put in while working. I actually stole it from my brother and sister, who are both researchers and put it in their texts when they need to fill something in later.

Monster Truck VS Zombies - A cancelled Ronimo project

At Ronimo Games we make lots of stuff that is never released or shown outside our office. Kind of a pity, really, because quite a lot of that is really cool! So I am really happy that today I can show one of the cancelled projects that I personally like most: the insane mockup trailer we made in May 2009 for Monster Truck VS Zombies!



This is a pitch video we did for a big mobile publisher (can't say who, since that's confidential). They wanted fresh games for something, and in a couple of days we came up with a concept, made concept art and produced a pitch video to show all of that.

This was to be a really small game, so we came up with a slightly simpler concept than what you usually see from Ronimo. The idea was to make a side-scrolling stunt-driving game, much like Excite Bike and Trials. You drive a giant monster truck and the goal is to kill zombies through stunt kills. After each level you can buy upgrades and weapons for your truck and driver. But really, the video says at all, so just have a look:



The music is Sad Man's Tonque by Volbeat. Since this is just a pitch video and not an actual product, we used an existing song that fits the game.

The art in this video was of course made by our art team, but I got lucky and could animate the gameplay sequence in the middle (the part with the stick figures ;) ) and do the video montage. Since I am lead programmer at Ronimo, it rarely works to let me do art things: it is impractical for our planning and our artists are way better at it than I am. However, I really enjoy the rare moments when there is an opportunity for me to do some 3D art, animation or lighting! :)

As far as pitch-videos for new concepts go, I think this is a pretty good one. I think it communicates both the style and the actual gameplay pretty well. Through the choice of music and the logo and concept art, it gives a good feel for how the player will experience it.

The gameplay mock-up video in the middle shows what the actual gameplay will be. We deliberately made this with stick figures, so that it is absolutely clear that this is not the graphical style the game would have. Since we wanted to make this video in only a couple of days, trying to show what the real graphics would look like in the final game during gameplay would not have worked: it is simply too much work to make that on a high enough quality level. In practice, it is always difficult for a viewer to look beyond mediocre graphics and realise that the real game is going to look better. So I think if you want to pitch something, it is a good idea to make things either look/play absolutely fantastic, or make it super clear that only parts of what you are seeing represent the final product.

So since the gameplay animation didn't show the graphical style of the game, our artists instead made two concept art pieces for that. One is the 'standard' concept art image. It features the main elements of the game and looks as cool as possible:



However, this kind of concept art is not very useful for anything but setting the mood and character. At no point in the game will you play anything that looks even remotely like this. After all, the game itself is side-scrolling and shows the monster truck from quite a distance.

This is a pitfall that lots of concept artists seem to fall for. How often does a concept artist design a character specifically to look good from the front, even though the game is third person, so you look at his back all the time? So at Ronimo our artists mostly make concept art from exactly the perspective of the actual game. So when we decide on a style or design, we decide based on what things will actually look like in the game. So for Monster Truck VS Zombies, our artists also drew an image of what the actual game would look like in the end:



Monster Truck VS Zombies was cancelled and nothing was ever made for it besides this trailer. So what went wrong with this game? A couple of things. First: the publisher didn't like it, so there was no financing. Then there was the platform: we had never worked with that mobile device before, so doing a new device was going to make the coding team too much of a bottleneck. Especially since I was still the only programmer at the time.

Finally, and this killed a lot of the enthusiasm in our team for this project: we discovered a Flash game that was almost exactly the same. A side-scrolling stunt-racer where you kill zombies with a monster truck. That Flash game was so badly made that it was no fun at all, but still, seeing something so literally the same is far from inspiring. We always try to make something that is at least slightly original and innovative, and this wasn't that any more after we saw that video.

Anyway, I still really like this little over-the-top pitch video for Monster Truck VS Zombies, so it is really nice to finally show it here! :)

Making an HD 2D game look good on an SD television

Awesomenauts and Swords & Soldiers are both 2D games that have been designed for high resolution HD screens. So how do they look on old CRT televisions with their low SD resolutions? The obvious answer seems to be that they should automatically look great: the art is super crisp, because it was designed for a higher resolution. However, what you may not realise, is that too crisp is not a good thing!

The problem lies in anti-aliasing, and in graphical details. Anything that the artists have drawn that is only 1 pixel wide in HD, will start to flicker on an SD screen. This happens for the simple reason that not every pixel will be shown on an SD TV. The art has been made for 1920x1080 pixels on the screen, and only 720x576 of those will be shown on an SD television. As a character moves over the screen, his details will flicker in and out of view as they are sometimes on a pixel, and sometimes in between two pixels.



This is especially problematic for anti-aliasing: in a 2D game, anti-aliasing of edges is drawn in Photoshop and doesn't require any work from the game. But on an SD television, the pixels that make the anti-aliasing may fall in between two screen pixels, meaning that anti-aliasing is often entirely lost.



(Note that this is a problem that is specific to 2D games: in 3D mipmapping is used to solve this problem, and anti-aliasing is rendered in real-time by the videocard.)

So when we render our HD 2D graphics to an SD television, they look noisy and too crisp. Now how do we solve that?

The solution is actually pretty simple. To get an image that is both smooth and sharp in SD, we use an easy trick in both Awesomenauts and Swords & Soldiers: on SD televisions, the game is rendered at a higher resolution than what is shown on the screen, and then sampled back to the lower screen resolution. This is a standard technique in non real-time rendering and is called supersampling.

So first we render the game at a higher resolution (1080x864 instead of 720x576). Then this high resolution image is rendered to the screen by sampling the area beneath each screen pixel. For each screen pixel, I take four samples from the high resolution image and use the average of that. This way really small details are not lost. Instead, they become more subtle, since they are averaged with the pixels around them.



So what resolution do we use for the high resolution image that we sample from? Easiest would be to use twice the screen resolution, but that requires rendering at 1440x1152, which is really high and wastes a lot of performance. I experimented a bit, and it turns out that rendering at 1.5 times the resolution is already enough to get smooth graphics.

This is a good thing, because twice times the resolution does tend to become quite expensive. In fact, at some point SD TVs had a lower framerate than 720p HD TVs, which is why I started experimenting with the exact resolutions to find a better trade-off between performance and quality. At 1.5 times the resolution, the framerate is smooth again.



The results look good, and they mean that I don't have to bother with storing all textures on different resolutions for HD and SD televisions, which keeps our art pipeline simple.

This post may seem to be about a really minor detail, but on old SD televisions, this really makes a big difference. Without this technique, both Swords & Soldiers and Awesomenauts didn't look good on low resolutions.

I find it really interesting that if you want to make a quality game, you really need to worry about all kinds of seemingly small details. All those little things in coding, art, sound and design are what makes a game polished, and this post has been about just one little example of the kind of polishing we do on our games.

Next week, I expect to have something really cool to show here, so see you then! ^_^

How a lack of art influences prototyping results

It is common knowledge among modern game designers that you have to prototype your game designs early and often. Yet when you do that, you usually don't have any art to put in your prototypes yet: the art team has probably also only just started development. Some game designers might say this is actually a good thing: your game design should be fun even when played with just coloured squares or simple icons, so you should start there and only add art and sound once the abstract core gameplay is fun. I used to agree with that in most cases, but during development of Awesomenauts, I learned that working from such abstract prototypes can also steer your design in certain directions, which is not always an improvement.

What really made me realise this is the jumping mechanics in Awesomenauts. Since we didn't have any art for this new game yet at that point, we made the very first gameplay prototypes for Awesomenauts using characters from Swords & Soldiers. However, these characters don't have any jumping animations, since Swords & Soldiers doesn't feature any jumping.



Now at some point our game designers wanted to add some weight to the movements of the characters, so they proposed that the characters would need to really push off the ground when jumping, and really land after a jump instead of immediately continuing to walk. Since we didn't have any art for this yet, I implemented this by simply adding a short pause before jumping and after landing. So when you press jump, the character takes 0.5 seconds before going up, and after landing he pauses 0.5 seconds before continuing to walk.

In terms of pure game mechanics, this is the entire thing. The only difference between this and what would be in the final game, is that these 0.5 seconds pauses would have specific animations for pushing off and landing. So in terms of abstract game prototyping, this was a complete prototype with all the mechanics in it.

Yet is played horribly: it simply felt like the game's controls were slow and lagging behind, not like the character had more weight and presence in the world. So we threw out the pauses after a couple of short internal playtests.

At this point I realised that the reason these pauses felt bad was not that the mechanic in itself was wrong. The reason it felt bad was that there were no animations. If there is no animations, then the 0.5 seconds pause before jumping is simply a delay on your controls. You press the jump button and nothing happens for a little while. Yet when there are animations, something does happen immediately: the character bends through his knees and pushes off the ground. The delay between pressing the button and moving up is the same with and without animation, yet the feedback is way better when there is animation, because the character immediately reacts to your button press, even though he doesn't immediately start moving up.

To me this was an interesting realisation: it is not always possible to use abstract prototyping to see whether a mechanic works! Sometimes you need proper art and animations to know whether the gameplay is any good.

Note that in this case, squashing the character before and after a jump would already have been a big improvement, because this way at least the graphical feedback would have been immediate.

So the prototype was not a good enough test. But was not having these extra animations a good choice? I still think it was: Awesomenauts is a very fast skill-based game and quick and direct controls are incredibly important, so we would have wanted immediate jumping anyway. So despite the incomplete test, I definitely think we made the right choice.



Of course, this is totally different for each game: for a game like Assassin's Creed, the subtle animations really make the game, and make it feel like the character is really climbing those walls and has real weight. So it really depends on the game whether the delay that is added by in-between animations is a good thing or not.



So whether you have animations in your prototype or not greatly influences the core feel of your gameplay. The same applies to many other assets. Here are a couple of examples of situations where prototyping with or without sound and graphics can make a big difference on what the conclusions of the prorotype are:

-Games are way more fun with sound. Half the intensity and thrill you feel while playing a game comes from the audio. However, often sound designers are contracted rather late in development, so you usually don't have the real sound effects during the first half of development. For Swords & Soldiers, we fixed this in an easy way: we simply copied the sound effects from Warcraft III and used those temporarily. As soon as the real sound effects by Sonic Picnic were created, we threw out all the Warcraft III sound effects. But until that time, we had a much better feel for the actual game, because we actually had any sound at all.

-When looking for the right style of music for a game, a common quick prototyping method is to simply put on the music outside the game and then start playing to see whether it feels right. This is of course a good idea, but it also influences the attachment between the music and the game negatively. The problem is that the timing of the music is not linked to the game when you do this, because you have to start the track outside the game. It works much better if the music is started at the exact same time as a level, because now the music is not part of everything outside the game any more. I noticed this with several games now: music never really feels right until the timing of starting and stopping the music is right. Adding this timing to a prototype is usually really simple and helps a lot.

-A really obvious example is guns in a shooter. The most important thing for a gun in a shooter is that it feels right and has serious impact. Since 80% of the feeling of impact for a gun comes from the graphics and the sound, and only 20% from the actual gameplay and damage of the gun, it is impossible to really prototype guns if you don't add any audio and visuals. However, creating those well is a lot of work, so I guess you have to start without, but until you have sound, I don't think you can really tell whether a certain concept for a gun is an awesome idea or not.

-The Aztec Sun Giant and the Viking Frost Hammer in Swords & Soldiers are rather similar soldiers in terms of abstract game design: slow, specialist units that deal area of effect damage and shortly put the enemy out of battle (through respectively knockback and stun). When playing the game without graphics, one might conclude that they are too similar, except for their large difference in price and health. So maybe more variation is needed? No, not really: when the art is added, these two units feel really unique and different, making them work really well in the game. Again, prototyping without art might not have been a good idea for these two units.



The most important realisation from this whole topic is that the prototyping method greatly determines the outcome. Prototyping gameplay with or without graphics and audio gives different conclusions about which mechanics work, and it is important to realise that this is the case. Of course, I am absolutely in favour of prototyping early and often, even if you don't have any art yet, but I do think trying to add some sound and graphics as early as possible greatly helps, even if they have been ripped from other games for the time being.

There is a more general lesson here as well. This post was just about graphics and audio, but I think the prototyping method always influences the outcome. Paper prototyping, prototyping in editors from other games: always keep in mind that the prototyping method used has some kind of influence on the outcome!

PS. Woohoo, Proun has been nominated for the Dutch Game Awards in a whopping three categories: Best Visual Design, Best Original Game Design and Best Audio Design & Music! Incredibly cool! Only one other game was nominated in that many categories: Killzone 3. That is one really big and really good game, so it is an honour to get the same number of nominations! However, since Killzone 3 was developed by something like 200 people, and Proun by only two, Proun has a hundred times more nominations per person than Killzone 3! [insert evil laughter here] Some other awesome games that got nominations are Radical Fishing (by our neighbours Vlambeer) and Gatling Gears.