An easy solution to the safe frame problem

One of the things I hated most while developing Swords & Soldiers for the Wii, was the safe frame. CRT televisions (you know, those old, big televisions that some people still have) don't show the entire screen: they simply leave out the screen's edges. I don't know exactly why this is traditionally done, but the amount of screen that is left out varies per television and can be pretty large on some of the worst TVs.



Surprisingly, many modern LCD/LED/Plasma televisions are by default set to also leave out the edges. They just zoom in a bit. However, since there is no technical reason why this is, these televisions come with a setting to turn that irritating behaviour off.

Now the good thing, is that there is a limit to how much of the edges a television actually cuts off. The area that you can be sure will be shown on every television, is called the safe frame or safe area.



Since games need to be playable on any television, Xbox, Playstation and Wii all have requirements that the game should be playable even if everything outside the safe frame is cut off. So crucial information or interface elements can not be positioned outside the safe frame. It is of course a good thing that Microsoft, Sony and Nintendo force developers to make games that are playable for everyone, but this is pretty horrible for us as developers: usually you want the interface to be as far to the sides of the screen as possible to leave lots of playing field, but that is not possible any more.



Luckily for me, our art team had to deal with this and not me. So they moved the interface elements further from the edges of the screen. Now the total screen (as is seen on televisions that don't cut off the edges), looks like this:



Note that it actually takes quite a lot of time and effort to do this correctly: in our office we have no television that has the worst possible safe frame, so instead we made a paper frame to put in front of the screen and 'emulate' the worst possible safe frame. Everything needed to be tested with this piece of paper in front of the screen.

Moving the interface away from the edges works, but it is pretty limiting for interface design, and it is also irritating work. While making Swords & Soldiers, we thought we had to do this, but right after the game was done, we discovered that some games solve this in a much easier way. So easy, that at the time it almost felt like a trick.

So, to keep you, the reader, from spending a lot of time on the safe frame, here is the simple trick that our new game Awesomenauts and some other games use instead:

Allow the user to set the zoom of the screen.



That's it. In OpenGL and DirectX, you can just create smaller viewports and leave the borders of the screen black. So on televisions that cut off a lot of pixels at the borders, you just render the game to a smaller part of the screen. Everything outside that is black. As long as the user sets this correctly (which he is forced to do when he starts the game for the first time), everything will always be shown, so you can put your interface elements wherever you like.



The only real downside this solution has, is that the resolution of the game is now pretty much dynamic. So rendering pixel-precise effects becomes a bit more difficult now. On the other hand, it also has a hidden benefit: since you render to a smaller portion of the screen, people with CRT televisions might get a better framerate...

This really is a simple trick, but for those who didn't know it yet: be glad you do now! ^_^

Follow your heart, not your wallet!

Last November my game Proun won a Dutch Game Award for Best Original Game Design. When I received the award, I grabbed the microphone and did a short speech, which Control Magazine then asked me to write that into a column. Which I did! The Dutch version is currently on their website and in the magazine, and here is the English translation.

The Dutch games industry is doing really well. More developers than ever before! But still, if you look for truly special Dutch games, you will find painfully few. Testament to this is that at the last Dutch Game Awards, two student games and a hobby game were nominated for Best Original Game Design. Where are the companies? So many developers, so few people who create their own concepts. It's as if we have all become entrepeneurs, and left our passion and creativity at the door.

I think this is because many Dutch developers are afraid to follow their heart. With all these talented people, there must be so many good ideas floating around! Yet almost everyone is doing work for hire. Within budget, finished on time, a certain income. Hardly any creative freedom and everyone will have forgotten your game two months later.

Of course, there are exceptions. Just think about Ibb & Obb, Dinner Date, Super Crate Box, Killzone (yes, that one as well!), and of course our own Awesomenauts. And don't get me wrong: there is nothing wrong with doing work for hire. It is an important part of our industry and it is a good thing that we Dutch are as good at it as we are. But it has been taken a step too far: most developers seem to only care about a steady income.

And of course, building your own concepts also means more risk, and will fail once in a while. But I am certain that more truly special games would be born in the Netherlands, and more international success would follow!

Because truly great games are made from the heart, not from the wallet!

The code of Awesomenauts' upgrade system

Last week I explained how our designers can create tons of upgrades for Awesomenauts using our Settings system. However, I didn't say a thing about how we actually made that work. We used some fun template and functor tricks there, so I figured it would be nice to show how it was built to work elegantly, without making our code a big mess of checks for upgrades.

Warning: this blogpost contains hardcore programming awesomeness! If you are an artist or game designer and fear templatised meta-programming functors might disintegrate your brains and/or explode your head, then I advice skipping this specific blogpost and coming back next week!

First lets have a look at how things would work without upgrades. Note that to keep things readable, I only show the essential bits of code, and I have greatly simplified all the code examples here.



The function loadSettings() translates from a variable name to a string, so that we can actually load the variable from a text-file. Doing this for every setting in the game is a bit cumbersome, but necessary because variable names in C++ are not real things and become simple memory addresses when the code is compiled. Some spot is needed where we have the variable name as a string in code, and I chose to put all of those in a single place in the function loadSettings(). I guess it might be possible to write a macro that does the same thing, but I don't know about that myself.

The key thing to notice here is how easy it is to use the settings in gameplay code, as can be seen in the function handleWalking(). We are going to add upgrades to this and out most important goal for that, is to keep the gameplay code this simple.

Since the upgrade system requires each setting to be able to have several upgrades and combine them, we switch the settings from a simple float or bool to a real class.

I want basically the same behaviour from floats, bools, strings and any other types that need to be upgradeable, so I used templates for this. Experienced programmers probably know what templates are, but for those who don't: in the code below the type T is left open and it can be interpreted as whatever we say it is. So we can make an UpgradeableSetting where T is a float, or where T is a bool, or something else. C++ can compile this code with any type we want it to.



The load() function fills baseValue and upgrades by looking for all the occurrences of the setting in the settings file.

The get() function returns the value that this setting has for the Character that is passed to it. The Character is needed here, because we need to check which upgrades he has and return a different value depending on that. Its code is rather simple, but seeing it might help understanding how this works:



Note that this code only handles upgrades that replace the standard value, so upgrades that are marked with @ in the settings file. For simplicity, I have left out upgrades with + (which add to the original value) that I discussed last week.

Now that we have this, we can make some slight modifications to our gameplay code to use the upgraded value:



This all works fine, but I am not all too happy with that function get(). Normally it would be perfect, but there are so many places in the gameplay code where settings are used, that I would rather not add get() to all of those.

Now C++ happens to have a nice solution for this. I rarely see it used outside STL, but it has an absolutely awesome name: "functor". I love that name! Got to be a Decepticon! Anyway, a functor is a construction where you can call an object as if it is a function. With a functor, our little class looks like this:



Now we can do the exact same gameplay code as before, but we can simply leave out the get call. For the rest, this code does the exact same stuff. Because we use the settings so often, this is a nice improvement.



Since we have so many settings in our game, typing the entire type every time is too much work, since that would be something like UpgradeableSetting every time. We simply solved this using typedefs. At Ronimo we normally try to avoid typedefs, because they hide the real type of a value and I would rather know exactly what I am dealing with. However, because there are thousands of settings, I consider this an exception and use typedefs anyway:



Now that we have this new class, let's have a look at the same code we had before, but now with the upgrade system added to it. Note how it has hardly changed, but can do so much more now!



So, that's it. It's just a little bit if code, but this makes the settings system way more powerful and flexible. It is definitely a great improvement to the RoniTech (the engine we created here at Ronimo Games and that we used to build Awesomenauts). In fact, its uses are so diverse that we also use it for many other things than the upgrades that the player can buy in the shop. So let's conclude this blogpost with this nice example of that:

Building hundreds of upgrades for Awesomenauts

In Awesomenauts, players can buy hundreds of different upgrades. There are around 150 unique ones, and most have two or three upgrade levels, so in total there are probably around 300 upgrades in the game. We wanted our designers to be able to create those themselves, without any work from the programming team for specific upgrades. At the same time, we also wanted the upgrade system to be super flexible, so that the upgrades would be very diverse, and not just all be cooldown reductions and damage increases.



When we first started looking for a way to build this, I was really at a loss. How to add a flexible system for upgrades to our settings system? It took some time, but coding intern Daan and I came up with a system that is easy to use and so flexible, that we ended up not just using it for upgrades, but also for temporary boosts and areas with modified mechanics, like low-gravity areas.



The idea is that for every setting, our designers can create modified versions for when a character has certain upgrades. Each setting always has one base value. Then as many modified versions can be added as desired, simply by copying the setting and adding @ and the name of the upgrade.



The @ sign works for many things, but it has the problem that it is problematic to have different upgrades for the same setting: what should happen if you have both upgrades? So we also introduced the + sign, which means that that upgrade's value is added to the original value. This way several upgrades can influence the same value.



This system does not just allow for simple upgrades that upgrade a single value, but can also be used for more complex stuff. Especially since our designers can give a player a temporary upgrade when he presses a button. So let's say we want to create a skill that temporarily makes the player deal twice as much damage, but also make him half as fast and twice as big. A designer could simply use the name of a single upgrade to modify all of these values at the same time.

Of course, the more flexible and complex things get, the easier it is for our designers to mess up. We have had quite a few bugs where certain combinations of upgrades behaved in a different way than our designer intended, simply because they chose the wrong values. The plus side of these kinds of bugs is that our designers can fix them themselves, so as a programmer I am okay with that... ;)

The upgrade system also gives our artists a lot of flexibility. For example, if a bullet does more damage, our artists can upgrade its graphic and its explosion to use a different animation.



The upgrade system described in this blogpost looks pretty obvious to me now, but it definitely was a revelation to us when we came up with it two years ago, and I still think it is the most elegant solution we came up with for any problem at Ronimo Games so far (although I suppose this system is not unique and other RPGs probably use something very similar). It is incredibly flexible and really easy to use, and allows our game designers to make almost any upgrade they want. And the good thing is that it is not specific to any games, so anything we make with the Ronitech (our engine) will have this feature for our designers! ^_^

Next week, I'll explain the technical side of how we made this upgrade work in C++!

The skill design system in Awesomenauts

(Note: Swords & Soldiers is currently on sale on Steam! 50% discount! ^_^ )

At Ronimo Games we always try to make innovative games, and in my opinion one of the biggest challenges when doing so is how to enable the game designers to do their work. Game designers work with things like game mechanics and control schemes, but you cannot know whether what you designed on paper is a good idea, until you played it and experimented with. It requires programming to create something playable, so unless a game designer is also a programmer, he cannot really do much on his own.

For artists, this problem has pretty much been solved: graphics tools are so powerful that artists can create most things without need of a programmer (except maybe for polish and performance). For game design however, this is essentially an open problem. There are many methods to empower game designers, but none are ideal. To name a few: scripting lanuages, event based systems, tools like Gamemaker and Virtools, and the most used of them all: just copying everything from existing games and not even trying to be innovative in any way (this also solves a lot of other issues, like the problem of being creative and taking pride in your work).

The approach we use to enabling game designers to create stuff is not ideal either, but it does work really well for us, so I'd like to write a couple of blog posts about the systems that allow our game designers to create game mechanics. Today is the first, and it is about our skill system.

To us it all starts with the coders and the game designers working together closely. Now whenever a programmer builds something, we set it up as flexible as possible, and make settings to control it. Things like speed, size, health, cooldowns and whatever else we can think of are put in settings and our game designers can edit those. Last year I wrote a post about our settings system in Swords & Soldiers that explains the basic idea further.

However, the way we approached this in Swords & Soldiers was also quite limited. All actual behaviour was programmed specifically per unit, so the only thing our designers could do, was tweak them. If they wanted to create a new unit, or even combine properties from existing units, they had to ask the programmer (me) to implement this for them.



In Awesomenauts we set out to make things more flexible. A character in Awesomenauts mainly consists of a list of skills, and which button needs to be pressed to use that skill. We have programmed around 20 skills (things like Shoot, Hit, Jump, Dash and Shield), and designers can create new versions of each skill with separate settings. Designers can also attach several skills to one button to create combinations, so maybe there is a combo skill that shoots a bullet and starts a shield at the same time.



In this trailer you can see how Sheriff Lonestar's set of skills works out in the actual game:



Each of the skills refer to a group of settings that define the exact properties of that skill. So there can be many different Shoot skills, each with different settings. This way some become grenades, other bullets, lasers or mines, and some even spawn new creeps.



Now this is really flexible, and I guess it is quite obvious that building it like this works much better than hardcoding the entire character as we did in Swords & Soldiers. The trick to making this really flexible, though, is to make lots of settings. Throughout development of Awesomenauts, we have constantly been adding new settings to make the existing skills more flexible, so that more things could be made with them.

However, it may seem so, but working like this is not an Obvious Good Idea (tm). It has some serious downsides as well, which all boil down to one thing: skills can be combined in any way and this makes all character code infinitely more complex. If you have 20 skills, then designers can already make 400 combinations of two skills happening at the same time. You can see where this goes once every character has 10 skills...

So implementing this system cost us an enormous amount of time to get all the combinations to behave correctly. A good example of the complexity here is animations. How to choose which animation to play when several skills are active at the same time? There are so many combinations that we kept bumping into situations where an animation did not look good or work correctly because some other skill interrupted it in the wrong way. This is very solid in the final game, but it took us a lot of work to get there. Unfortunately, it is still very fragile code (i.e. it is easy to introduce bugs when working with it).

Another downside of this system is that if you want it to be really flexible, then you need dozens of settings for each skill. The more settings there are, the more difficult it becomes for the game designer to keep track of what each does exactly. During development it happened quite often that our designers asked for a feature that was already there.

Of course, the upside of this is that with so many settings, there are many combinations that achieve totally unique things. Especially the Shoot and Shield skills have been applied by our game designers to do totally different things than what I expected when we programmed them.

So choosing between the flexibility in Awesomenauts and the very inflexible system in Swords & Soldiers, I actually don't think the flexible system always wins. I would even say the system in Awesomenauts might be up to three times as much work.

However, if, like we did in Awesomenauts, you want to spend a lot of time iterating on the game design to find the most fun and varied units, then more flexibility is definitely necessary. So I think we made the right choice for Awesomenauts. It is a choice that cost us a lot of development time, but that also improved the game's quality enormously.

The lamest bug we ever encountered

Yesterday we resolved by far the lamest 'bug' I have ever encountered. We spent almost two whole days with the entire programming team (5 coders) trying to find it, and once we discovered what was happening, it felt so incredibly lame, that I figured it would be a nice story to share. So here's what happened!

In the past half year we have seen a really rare bug occur about once per month in Awesomenauts on the Playstation 3: suddenly the game would freeze for anywhere between 10 to 100 seconds, and then continue normally as if nothing weird had happened. We always have a PC connected to collect logs from the game. However, the log printed nothing interesting and showed simply the framerate, which is printed once per second:

13:48:13 60fps
13:48:14 60fps
13:49:21 1fps
13:49:22 60fps

From this we concluded that the issue must be in some part of code that does not log anything. That leaves 99% and with a codebase of at least 150,000 lines, that is a lot!

Since this issue was so rare and we hadn't seen it in months, we had closed it. We thought it was an anomaly that we couldn't find or fix and that had somehow disappeared.

Until we were testing our latest build last Thursday. We were playing on seven Playstation 3 devkits, when suddenly 4 of them froze for about 90 seconds. Argh! We thought this really difficult bug had magically disappeared, and now it was back, with a vengeance! We cannot leave a bug in that happens so massively on all these consoles at once!

Again, no relevant logging. Yet something really interesting nevertheless: these Playstations were in different online matches, so they weren't even talking to each other! So how could they all freeze at the exact same time? We concluded that the only thing they had in common, was that they all talk to the same Sony matchmaking servers, so we started investigating all our code related to that. However, we couldn't find anything relevant.

So we added a lot more logging, and did some really advanced stuff to get more info on the stack during the freeze (which is difficult to get from an executable that has been stripped of all debugging info), and started playing again. I let the Playstations perform our automatic testing all night, and the bug didn't happen. Then we played the game for five more hours with the entire team and BAM!, it finally happened again on two consoles!

This time, we had more info, and it turned out that the spot where the game froze was different on both freezing consoles, and both did not contain any calls to Sony's matchmaking servers. In fact, it was in between two logging calls, in a spot where nothing relevant was happening. So we concluded there were only two possible causes: either other threads were hogging the entire CPU (due to how the scheduling system on the Playstation 3 works, high priority threads can do this permanently), or the logging itself was broken.

So we started experimenting around that, and now we quickly found the cause of this 'bug': when the PC that is tracking logs goes into sleep mode, the connected consoles freeze a little while later; once the PC is awoken once more, the consoles continue as well a little later. The PC that was tracking the logs automatically went into sleep mode after not touching it for 30 minutes. This only happened during extensive play-testing, because people normally actually use that PC. So it wasn't even a bug in our code! ARGH! Especially since logging is turned off in the release build...

This may all seem really obvious in hindsight, but in general when we have a bug/freeze/crash it is in our own code, not in one of the tools we use. With such a big codebase, it is easy to not even think about something else. Also, in the chaos of 14 people playing the game on 7 consoles, it is easy to overlook one specific PC going into Sleep mode a little bit before the consoles freeze... Also, I still don't know why not all consoles connected to that PC froze.

So, this hard to find and very persistent bug turned out not to even be a bug! And what do we learn from this? Two things:
1. Really nasty bugs are often in a totally different spot than where you are looking.
2. Sometimes you encounter the lamest and most frustrating stuff while programming games!

QQQ

In his talk at Quakecon this year, legendary programmer John Carmack (Keen, Wolfenstein, Doom, Quake, Rage) talked about code quality, and how even the very best programmers make lots of mistakes. So he says that every programmer should think about ways and methods that make mistakes happen less. This is something that I totally agree with, and this kind of thinking is the basis of the coding standard and methodology that we use at Ronimo Games.

One of the nicer things that have crept into our coding standard over the years is a tiny little trick that keeps a lot of issues from happening. It is pretty obvious, but I know a lot of programmers who don't do something like this, so I figured it would be useful to mention this one today. :)

Often when you are programming something, you quickly make some changes somewhere else to test one specific situation. For example, maybe you are testing the graphics of a weapon against a specific enemy. If that enemy dies all the time, you need to find another one to test on, so a quick change could be to decrease the damage of that weapon against that specific enemy by 90%. Now the enemy stays around a lot longer and you can test more efficiently.

However, once you are done working on that weapon, it is really easy to forget to remove your little hack. Later someone else spends hours searching through code why that enemy dies so slowly. Only to discover that forgotten hack of yours.

Many coders would say now that this is sloppy programming and you should just be a better programmer and remember these things. However, while programming, there are so many things to consider that it is easy to forget one once in a while, no matter how good you are. It is better to design your working method around the idea that you do make mistakes, then to simply say you should be a better programmer.

Now since these little hacks are a great help when programming, everyone I know uses them a lot. So I came up with a really simple trick to keep track of them. Whenever you do a little hack like that, you add a comment with the text "QQQ". So like this:

if (enemy->getName() == "UrQuan") damage *= 0.1f; //QQQ

All coders at Ronimo do this ('voluntarily'). Before we commit our newly written code to the source server, we search for QQQ to see whether we forgot anything. That gives us a list of all the temporary hacks that are still in there. And because we all use QQQ, and not something different for each programmer, we can quickly find QQQs that someone else forgot to remove.

Since QQQ is basically just a marker to remember something, I also use it when I realise I need to remember to rework or check something later on.

So why the letters "QQQ"? This specific term has the benefit that it never occurs in any normal sentence, and is short enough to quickly put in while working. I actually stole it from my brother and sister, who are both researchers and put it in their texts when they need to fill something in later.