The craziest bugs, part 1

Every programmer must have encountered these: weird bugs. Unexplainable bugs that make you want to tear your hair out. Bugs that are just plain funny in their bizarreness. Even bugs that are pants-on-head-retarded. I have written about a very out-of-the-box bug and a painful oversight bug before, and since then I have encountered hundreds (or was it thousands?) of other bugs. Today I would like to give you my favourites.

These bugs are from various categories: from funny, to surprising, to dumb library design. Most we have been able to solve, but for some the exact cause still remains a mystery. The one thing they have in common, is that I remember them fondly. Or frowning. Or while gritting my teeth...

Click here for numbers 3 to 1

7. Std::abs differs between compilers

This is one I encountered recently in Awesomenauts. We thought hardly any Mac users would have gamepads, so we had initially decided not to support those on Mac. After launching Awesomenauts on Mac, this turned out differently, and a lot of Mac users requested proper controller support. We decided to try to patch this in quickly before Christmas, but one particular bugs almost kept Mac joystick support from making it into that patch.

It turned out that the sticks would only work if they had full output. Pressing them to anything but fully right or fully left had no effect in the game whatsoever. Somehow, our joystick class outputted proper floats in the complete [-1, 1] range, but once they got to our gameplay code, they were only 0, -1, or 1. I didn't see any spots where they were turned into ints, so how was this happening?

The reason turned out to be std::abs(). Normally, this function only works on integer types, and fabs() is used for floating point numbers. However, in Visual Studio abs() also works fine on floats, and we had used it in that way on a joystick axis value somewhere. This version of abs() does not exist on Mac! When we tried to use gamepads on Mac, the compiler did not print a warning and instead simply rounded the decimal axis value to an int and applied abs() to that. Since at that point the axis value was already in the range [-1,1], this meant that everything but -1 and 1 were rounded to 0...

Luckily, we found this one in time and were able to patch in Mac gamepad support before Christmas. And our Mac users lived happily ever after... (or so I hope!) However, I still find the combination of Visual Studio having extra functionality and the Mac compiler not printing a warning pretty nasty!

6. Editor framerate extremely low around centre of world

This is a bug that our artists at some point started complaining about. When using our in-house animation editor, the framerate became incredibly low, unless they moved the camera away so that the centre of the world was not in view any more. I didn't have time to look into this right away, but strangely over time the bug seemed to grow worse and worse, until the editor was hardly usable any more.



Quite puzzled, I dove in. I quickly discovered that the renderer was responsible for the framedrop, so I started gathering data on what exactly was being rendered. The cause proved to be quite... interesting.

Awesomenauts in total has over 4000 animations at the moment, and these contain some 5500 particle systems. It turned out that in the editor, all of these particles were always rendered, but with 0 particles each. The renderer did not check for this, and set up shaders, textures and matrices for these particle systems, and then proceeded to feed the videocard a whopping 0 polygons to render. Doing this 5500 times per frame is not a good idea... The solution was twofold: the renderer should check for polygon count before rendering, and the editor should hide these empty particle systems to keep them from reaching the renderer at all.

I tested the results of fixing this on two computers, and on one the framerate went from 28fps to 115fps in the editor, while on the other it went from 36fps to 273fps. Those are the nicer optimisations!

But why did this only happen at the centre of the world, and why did it grow worse over time? This is because most animations are quite small, and all their objects are near the centre of the animation, including the particles. After some more experimentation, it turned out that since the particles are not all exactly at the centre of the world, scrolling around would gradually increase the framerate as more of the area near the centre went out of view. Finally, the reason it grew worse over time, was that our artists were quickly producing more and more animations for the skins we were adding to Awesomenauts, adding more and more particles to destroy the framerate...

5. Disappearing textures

This is an issue that I actually haven't been able to pinpoint and solve, but it is so bizarre, that it deserved a place on this list. A user reported that at some point, Awesomenauts suddenly started looking like this:



He also posted a video of this event. What you are seeing here, is that the main view of Awesomenauts has been replaced by a small portion of the Steam overlay. The letters are names and the lowest line even mentions Steam's standard shortcut: shift+tab. On top of that, the Awesomenauts HUD is visible, as it should.

The reason I have so far not been able to find the cause of this bug, is that it is extremely rare: it has only been reported to us twice. One of my colleagues also had something similar once: a Steam icon had somehow replaced an icon in our own scoreboard. None of these cases ever happened again. Without any way to reproduce or test, I cannot solve this weird bug.

I can guess what is happening, though: somehow a texture from Steam replaced one of my own textures. The reason this replaces the entire screen in the image above, is that apparently the texture being replaced is the rendertexture that is used to apply post effects to the screen. The other report I saw confirmed this: there a different rendertexture was broken, causing only the background to look black (the background is rendered separately to apply depth of field blur, as I previously wrote about in this blogpost).

This makes me suspect that the problem might not even be in our own code: to render their overlay, Steam pretty much hacks into my rendering process every frame. I imagine the problem might for example be that something in Steam's code is problematic with the way I handle threading in my renderer. So this might not even be a bug in my own code... In the meanwhile, since this bug is extremely rare, I have decided to just leave it be.

4. Random programs interfere with my game

I don't know exactly how they manage to do this, but sometimes random programs manage to break Awesomenauts on PC. The worst offender is Adobe Air: this sometimes completely obliterates a player's internet, causing unplayably high ping in Awesomenauts. This doesn't happen for most users, but I have seen a dozen or so reports from players who managed to fix their high ping in Awesomenauts by uninstalling Adobe Air. This isn't just in Awesomenauts: I have also seen reports of the exact same thing in another game by a different developer.

Another example of random other programs messing up our game, was reported by a player. This user had a very laggy, uncontrollable mouse cursor, but only during gameplay. I had no idea what caused this, but at some point the victim reported that the problem had went away. What had he done? He had updated... Java! Java!? What does that have to do with Awesomenauts? Awesomenauts is written in C++ and doesn't use any JAVA components! I still don't know how JAVA is able to break the mouse cursor in Awesomenauts, but it is a nice example of how completely unpredictable PC development can sometimes be. Consoles may be much more complex to develop for, and they may have all kinds of certification requirements, but at least they are always the same! Hurray for that!

That was it for the bugs today! The four bugs above were still somewhat sane, but the rest of this list won't be. Visit back next week for the top 3, where the real insanity happens!

Why Cello Fortress is a twin stick shooter

Cello Fortress could have been any kind of game. The core concept is nothing more than: "a game in which a live cellist controls the game by playing cello, and plays with or against the audience". This idea can be applied to any genre. The cello could control a brawler, a puzzler, a strategy game, a racing game, with some imagination maybe even a point and click adventure. So why did I specifically make a twin stick shooter? A lot of thinking and brainstorming went into this choice, so today I would like to explain that a bit.

Doing something with improvisation on my cello and my computer is a topic I have been thinking about for years, but it wasn't until a year or two ago that it dawned on me that the cello could actually be a game controller. Before that, I was mostly thinking about writing a procedural music generator that could accompany my own cello improvisations. Quite a big step from a game, but it slowly evolved into one from there nevertheless.



Knowing that the game needed to be about a live performance with my cello, with or against the audience, creates a number of requirements for the game design. These requirements fuelled what would become the actual game, and it was quite a challenge to find something that really met them all well enough.

1. Game is fun to play, and also fun to watch for the audience.
2. Playable by as many people in the audience at the same time as possible, while also being playable by only a single player.
3. Such simple controls that no long explanations or tutorials are needed, and that even non-gamers can play.
4. The cellist has room to improvise and play something that sounds good, while still controlling the game in a meaningful way.
5. The influence of the cello is direct enough that players and audience can quickly recognise and understand it.
6. Bonus requirement: the core game itself is original.

Each of these requirements brings a different view on what this game should be. Number 2 for example favours games with a zoomed out view, so that several players have room to move within the same screen. Splitscreen is also an option for this, but didn't seem such a good fit for readability in case of a larger audience where people might be standing further away. Specifically inspiring games for me where Gatling Gears by our Dutch friends at Vanguard, and the WiiWare racer Driift Mania. Both of these games are also excellent fits for requirement number 3, since they require very few sticks and buttons to control, which makes the gameplay easy enough to explain in just a couple of seconds.



Another game genre that came to mind was rhythm games. Dance mats are really nice controllers for festival-like situations, since they are so physical. However, this kind of game seemed at odds with requirement number 1: most rhythm games don't have an on-screen hero that onlookers can follow and root for.

Requirements numbers 4 and 5 are where the real complexity of the game design steps in. How does a cello control a game while still making music? For example, going left by playing high notes and going right by playing low notes is way too boring. I needed something more subtle. At the same time, the influence of the cello should be clear to the audience as quickly as possible, to make sure people don't think it is a 'normal' game with only a live soundtrack. The cello actually controls the game and ideally people would understand this without my explanation.

This makes the racing game a difficult proposition. I had in mind that the cello would generate the track. But to give players a chance at reacting to it, the track needs to build up a bit in front of the furthest player. This has the big downside that it happens just beyond where most people are looking.

I also struggled with how to build the track from the music. I thought about things like making difficult, spiky roads if the cello plays in minor, while generating more smoothly curving roads when playing in major. However, this is way too subtle. Most people probably can't even recognise the difference between major and minor well enough, let alone link it to the gameplay. In my mind I tinkered with lots of other ways in which the cello could control the game, but I didn't come up with anything that worked as well and as naturally as the current twin stick shooter.



I arrived at the twin stick shooter as a good genre for Cello Fortress pretty early in the process, yet for a long time I kept brainstorming and looking further. This was because of requirement number 6. Twin stick shooters are a pretty overused genre in games, and it is difficult to still do something interesting with them in terms of gameplay. So I preferred something more original and kept searching.

However, in the end a quote from my former teacher JP van Seventer reminded me that keeping the core game less original might actually be a good thing. JP is a Wise Person (tm) and is also Ronimo's regular outside advisor. He currently works at the Dutch Game Garden to give advice to young Dutch game start-ups. He once said something along these lines:

"Innovating everything at the same time is not a good idea, because it alienates the audience too much. It is better to innovate on a number of aspects of the game, and keep the rest recognisable. That way players can relate to it much better and understand how it works more quickly."

This has been very influential on my thinking about games. Before this I had dreams of coming up with a game that would be totally unique in every possible way, and this quote made me realise that that might often not be a good idea. This quote definitely applies to Cello Fortress.

With Cello Fortress having been in the media quite a bit in the past week, I see now how difficult it is to explain what Cello Fortress is. Even after the very explanatory trailer that I posted last week, I read lots of confused comments online from people who don't really get how it works. So I am happy that I chose a genre in which the core gameplay itself at least is really easy to explain, so that I can focus my communications on the much more interesting side of the game: the way the cello controls it and the way the game is halfway between a game and a live performance.

Cello Fortress trailer revealed!

A couple of months ago I revealed my new project Cello Fortress. Now it is finally time for a proper trailer! Cello Fortress is a unique combination of a live cello concert and a game, and is intended to be played at events (festivals and such). This trailer shows how the game works, and shows a bit of the interaction between the players and the cellist.

Cello Fortress is pretty weird and unique, so I guess so far only those who actually played it, really understood what Cello Fortress is about. Hopefully this trailer will clear it up for others as well!


(Video footage recorded by Dyzlo Film at Indigo.)

In essence, Cello Fortress is a twin stick shooter. Four players cooperate using Xbox controllers to destroy as many cannons as possible. However, the cannons are not controlled by the computer, but by a live cellist! He improvises live music on his cello, and tries to do that in such a way that the game not only does what he wants, but also that the music actually sounds good. In a sense, this is the ultimate in adaptive music!

The current version of Cello Fortress is still in beta and far from finished. The graphics are just some quick prototyping models thrown together, and all kinds of things still need tweaking and improving. Cello Fortress is already touring, though, and has so far played at several events in the Netherlands.



Cello Fortress is a complex project in several ways: playing cello so that it sounds good and controls the game is a big challenge and requires an experienced cellist and a lot of practice. Analysing what the cello plays is also technically very complex and has, as far as I know, never been done before in a computer game. I expect I can write a couple of interesting blogposts about sound analysis now...

Cello Fortress may be a music game, but it is nothing like existing music games like Guitar Hero. In Cello Fortress the instrument is a real cello, and real music is played. The cellist is also not scored for playing 'right' or 'wrong' notes. Instead, he controls a shooting game by improvising.

Because the music is so central to the game, I wanted to make sure that the trailer would sound like an actual match of Cello Fortress. In real matches, the music is completely improvised (nothing is composed beforehand), so I decided to also improvise the music for this trailer. I ended up recording over twenty improvisations, and I used the one that I liked most. (The timpani were added digitally afterwards to add a little bit of flavour for the trailer.)

I hope that this trailer will not just get attention in the world of games, but also in the world of music. Cello Fortress could be a revolution in music acts! This makes me quite nervous, though: musicians are bound to hear any mistakes in my playing. Cello is an incredibly difficult instrument to play well. Although it has been a hobby of mine for over twenty years, I can still only hope my playing sounds acceptable to trained ears...



Since this music was improvised specifically for this trailer, it is not entirely the 'real thing'. Luckily, someone recently posted some footage of a match with audible sound (all other video recordings I have contain more audience noises than cello music). The video and sound quality are not super, but nevertheless this gives a good idea of what a Cello Fortress match sounds like:


(Filmed live at the Playful Arts Festival.)

Cello Fortress is a really weird and unique game, but for me it makes a lot of sense: playing cello has been a hobby of mine for ages, and I am a professional game developer. I like to make weird, unique things. How could these ingredients not combine into a game? Coming up with the actual concept for Cello Fortress was more difficult though: cello and computer can be combined in many different ways and it took me years to come up with something that is fun for the audience to play and watch, controllable by a cellist, and allows for beautiful music.

Cello Fortress is a project that I do entirely in my spare time. Nevertheless, I am going to steadily keep improving Cello Fortress, and I hope I can play with it at some exciting events and festivals!

What would you like me to write about?

I have been writing articles for this blog for almost 2.5 years now, and I have strived to cover a very diverse set of topics around game development. I don't really know my readers, though, so having such a broad range of themes has made me curious: what subjects on this blog do you enjoy reading about most? What would you like me to write about in the future?

To the right you can see a little poll with the broad topics, please select your favourite! I'd also love to hear about specific topics you would like to read about. Please leave a comment and let me know! Any other feedback on this blog is also welcome.

In the coming year, I'd like to try to cover some of the requests with (hopefully) interesting articles!