The weird cause of Swords & Soldiers network errors

Last week we released a patch on Steam that finally fixed the networking issues some people were having with the PC version of Swords & Soldiers. I'm afraid that because of the work on Awesomenauts, we didn't come to it any earlier. But we finally fixed it now! :)

Now this bug we fixed was a pretty interesting one. Not as interesting as the bug I posted about last December that got my blog a whopping 40,000 viewers in a couple of days, but still interesting enough to share it with you. This one is another nice example of how bug fixing sometimes requires thinking far outside the box.

We were getting two bug reports from users:

• Some users always got a Network Error at the start of a match.
• Some users encountered cheaters who introduced so much lag, that the game became unplayable for the player on the right side of the map.

We initially just assumed the Network Errors were being caused by firewalls or bad router settings. But these users were not having trouble in other games, and it even didn't work when they turned off their firewalls. We also had a user who told us it worked on his laptop and not on his PC, while both were on the same network. That pretty much ruled out router issues as well.

As for the 'cheaters': we immediately had the hunch that this might as well be a network bug in our code somehow, and not really someone cheating.

We had looked at this bug at various moments in the past year without success, and wondered whether it was still a firewall, or something in the Steam networking libraries. We even sent the Steam support team questions about this one. They didn't know any bugs in their system that could cause this, which was of course right, since this turned out to be a bug in our own code...

So, what was up? In a very bright moment, my colleague Maarten, who does a lot of network programming at Ronimo Games, suddenly realised what was happening: to check what the ping is and to keep the connection alive, we regularly send ping messages over the network. Accidentally, we did that every frame. Now sending 60 extra messages per second over the network is a really bad idea, but does not usually kill a connection. However, and this is where we are getting way out of the box: some users had forced vsync to be turned off in their drivers. For those who don't know: vsync makes sure a game runs at the same framerate as your screen, usually 60 frames per second. Forcing vsync off means Swords & Soldiers might be running at hundreds of frames per second on their computers! Sending that many ping messages instantly kills most connections.

This also explains the 'cheaters': if the framerate is not high enough to actually kill the connection, it can still be high enough to make the connection really bad. The player whose computer is server in a match won't notice this, but the other player (whose computer is the client) gets so much lag he can hardly play anymore.



So, the solution was really simple: instead of sending the ping message every frame, we now send a small, fixed number of them every second. Which is actually what we had intended in the first place, had a bug not thwarted our plans!

So, another nice example of how bug fixing requires letting go of all assumptions, and researching with an open mind what is actually happening. Until we found this bug, I never would have connected vsync with networking errors!

I would like to thank the Steam users who helped us with testing to find this issue! I would also like to thank those who offered help that we ended up not using. It is great to see that gamers are so willing to help when we are stuck on a bug! Thank you very much, folks! :D

PS. If anyone encounters any new or further issues, then please don't hesitate to post them at the Steam forum or our own forum, and we will look into them as soon as we can!

Awesomenauts audio dev diary

Today's post is a guest post by Sonic Picnic, the team that did the incredibly awesome music and sound design for both Awesomenauts and Swords & Soldiers. Especially the Awesomenauts theme song is mindblowingly awesome and got our game a lot of love from gamers and the press. Sonic Picnic is a small team of audio designers who create audio for all kinds of media, not just games. For example, they also worked for television shows, documentaries, and the games Rocket Riot and Toki Tori. So, here is their Dev Diary for Awesomenauts. Enjoy!


The intro sequence to Awesomenauts

Music

The development of the music for Awesomenauts started halfway through 2009, at a listening session with the whole Ronimo team, the four members of SonicPicnic, and a few beers. Based on the first sketches and early prototypes of the game, everyone brought along a couple of existing music tracks which they found particularly fitting to the theme and style of the game. Everyone at the table had a different idea about what would be appropriate, so we ended up with a broad spectrum of musical styles and tastes - ranging from metal to hillbilly western tunes.

Luckily, there were some aspects to the music that everyone seemed to agree upon. Most of the tracks were fundamentally electronic or had synths, samplers and drumcomputers playing an important role, whether they were techno, glampop, or electrometal. There was also a strong preference for music that could be heard in games like Streets of Rage, Alien Breed, Project-X, and other games that we used to play on the Sega Megadrive/Genesis and Commodore Amiga. No doubt the graphical style of the game had something to do with that, as it strongly refers to games from that era. Finally, as the game oozed chaos, mayhem and wackiness - the character backstories, the gameplay, the colors, everything – we felt like we had to capture that in our music too, perhaps more than anything else.

Even though we were sure we didn’t want to tie ourselves down on one very specific style, we all agreed that the music should have a strong electronic component. If anything, we wanted it to be wacky, and have some historical reference to the early 90’s.

There was still plenty of room to differentiate within these boundaries, and we really stretched these to their limits when it came to creating the character-specific jingles. Froggy G, for example, has a smooth, somewhat oldschool-hiphop theme, and the jetpack-wearing monkey known as Yuri has a revolutionary Soviet march with a drunk Kozakian Choir.

Apart from their jingles, each of the six characters was also linked to one of the ingame tracks. When the player starts a game, they will be hearing any one of the ingame tracks chosen at random. It kind of feels like a jukebox that has been set to shuffle. Each track lasts for about 3,5 minutes, and the game just continues playing random tracks as it progresses. The bond between the character and its ingame track becomes apparent when the player achieves a ‘killing spree’. At that moment, a beefed-up version of the ingame music starts playing - which is really a mash-up of the recognizable character jingle and a regular ingame track.


Lonestar's Jingle and Killing Spree examples

With this new song pouring out of their speakers, the player gets the feeling of being an unstoppable, unbeatable superhero for a while, until, after a specific amount of time, the killing spree track flows back into the regular ingame music. After a few matches, players will be able to recognize their characters’ tracks from amongst the other tracks, and their ‘moment of fame’ will become greater still.

For the menu, we decided to compose a retro-futuristic synthesizer tune (like the synthesizer music of the seventies and eighties) to really get that space/sci-fi feel. We also composed the ‘Victory’ and ‘Defeat’ jingles that play at the end of each match in the same style, to make sure the melody is nicely interwoven throughout the game.

The title music was a special case: Ronimo wanted to have a hand drawn animation film introducing all the characters and explain the (rather shallow) storyline in a typical 80's cartoon style. The great inspiration for this sequence came from series like Thundercats, M.A.S.K. and Galaxy Rangers. From the start, it was perfectly clear that this animation just had to be supported by a glam-muscular-power-rock theme.


The opening sequence of M.A.S.K.

This was absolutely great fun to compose and write, but we originally limited ourselves to a 30 second intro to fit the length of the animation. But when the theme was released as part of several trailers that Ronimo made for the game, lots of people became excited about the song, and asked us where they could hear the complete version of the song... Which doesn’t exist... yet. So yes, we’re creating a longer version of the intro music and will turn it into a complete hit-length track. Tell your friends!


Recording vocals for the Main Theme

Aside from the electro sounds, the music in Awesomenauts can also be a bit cheesy at times. This reaches its zenith in what finally ended up as the credits music. One of the tracks that we came up with during our initial brainstorm was a song by Demis Roussos (Forever and Ever). We all agreed that there had to be something like that in the game somewhere, because ‘cheesiness’ and ‘wackiness’ are terms both Ronimo and SonicPicnic keep in very high regard. So we came up with a song called “You will ever always be my only Awesomenaut.” We originally intended it for the grand moment of victory in a match, but when it was finished, Ronimo found it to be a little bit (well, quite a little bit) too much when compared to the cheesiness of the rest of the music. So it got banished to the credits. That means it’s still in the game though, so reason enough to check out those credits!

Sound design

At the very first stages of our work on the game, we put a lot of placeholder sounds in the game. We gathered a lot of different sounds from our own library and we also created a bunch of simple ‘soundsketches’, and implemented them into a very early build of the game. This gave us an idea of how the game would work with sound, and acted as a starting point from which we could experiment with different types of sounds.

There are actually a few placeholder sounds that made it in the final build of the game (after some minor editing), for the simple reason that they just worked very well. As with every development process, there were a few features and objects that got axed as development progressed. Some of these had very cool sound effects, so we tried to find a new use for them in the game wherever possible. Sometimes we had to kill our darlings though... but that’s part of the job.

Because Awesomenauts is set in a sci-fi setting, we decided to focus on creating many of the sounds with synthesizers. This fits the graphical style very well, so all the interface sounds and many of the in-game sound effects have been created with a synthesizer. For example, the ‘powerswing’ from the sword of Leon Chameleon, or the charging of the droppod before it's being launched onto the battlefield, and the time bubble skill of Yuri.

Besides the many synth-created sounds, we actually used a lot of real life recordings. The alarm that sounds when a turret is under attack is a good example of this. One day we were working in our studio, and it was 12:00 on the first Monday of the month. That's the time when the air raid sirens are being tested in the Netherlands. We thought it would be nice to use that alarm so one of us quickly put a microphone out of the window of our studio in Utrecht and recorded the alarm. We edited it, put some processing on it like EQ and some reverb, and put it in the game to test it. It was spot-on, and it’s been in the game like that ever since! Another example of sounds we recorded are the footsteps for Froggy G. It is a very simple sound actually, which was created by one of us stomping his hand against his bare leg (not too soft). Yes, sound design is a painful job sometimes...

Voices

We recorded five voice actors for the voices of all the characters. Almost all of these recordings were processed afterwards. For the voice of Clunk, we used a vocoder and we pitched the sound a little to create his distinct robot sound. For the voice of Voltar the Omniscient, we used a lot of signal processing to get his voice to sound like it's coming from a bulb through a little speaker.

For Froggy G we layered two recordings of the same sentence, to give his voice a little multi membrane, froggy sound.

There’s actually lots of fun to be had during the recording of weird and funny character voices.


Recording and tweaking Voltar's voice

Implementation and Mix

We have worked together with Ronimo Games on Swords & Soldiers, their previous game. Compared to that project, we were much more in control of the audio implementing with Awesomenauts, all thanks to the new Ronitech engine. We were able to put sound cues on almost every animation ourselves, instead of asking the programmers for cues. This way we were able to set parameters for timing, loops, volume, whether the sound should always finish, etc. Furthermore, we had control over the distance, pitch and volume variation and instance limiting (time between triggers and quantity of sounds played in a group). It was great to have more control over the audio, and it helped us work more efficiently.

The mix of all the different audio elements is very important, and at first we made a mix in which all sounds were quite loud. This didn’t feel quite right, as a loud explosion has to stand out in order to be impressive. It was quite a challenge to get the mix right, as we wanted to keep a dynamic feel to the sounds (so when you have a higher skill e.g. a bigger bomb you can drop, it would sound louder), but also wanted the music to stand its ground and be constantly strong and bold.


Gameplay example from Awesomenauts

That left the smaller, more subtle sound effects, like footsteps, that enrich the overall experience, in need of a place somewhere in the mix. As always with mixing, it was all about balance. And especially in this game, where a lot is happening at once, the mix was a pretty difficult puzzle.

In the end, when all sound and music had been implemented, combined, and tested, we were quite happy with the awesome result. We hope you will like it as much as we do!

Proun's sales statistics after removing the free version

When I released the complete sales data of Proun last October, I also changed the payment model: I removed the free version and set a minimum price of $1, while keeping the Pay What You Want model. I promised to get back on whether that made a difference, and so I finally do today!



An important issue I see with also having the game available for free, is that it is much easier to just download that than to pay something. No need to get your Creditcard and fill in the numbers, just click the Torrent and download. So I last October I was wondering whether removing the free version would increase the revenue of the game.

This is an interesting question, because the answer is not all that obvious. Of course only a small percentage of those who would download for free, would be willing to pay otherwise. The free version doesn't bring in any money, but it does get the game played by a lot more people. This fuels word of mouth and is incredible marketing. So the free version might actually bring in more paying customers. Also, many gamers who want to try a game will just pirate it if it is not available for free. So, in total, I had no idea whether removing the free version was really a good idea or not.

So, how did it go? First, let's have a look at the graph with the revenue per week before and after the removal of the free version of Proun:



Since the sales peak at launch is enormous and makes everything else too small to analyse, I have also included a zoomed in version for the lower sales periods:



The first thing to notice here, is that Proun is still bringing in a couple hundred euro per month. For a hobby project that I did on the side next to my job at Ronimo Games, that is incredibly nice!

Now, before comparing, a couple of things need to be taken into account. I see two really big sales peaks here, and one smaller one. These are not related to whether there is a free version or not, so it is important to know them before we draw any conclusions.



So, what can we conclude from this? You can see that the average sales per week with and without the free version are approximately the same. The peaks don't really count, since the circumstances were so different there. That leaves only a couple of 'normal' weeks before the free version was removed.

So these numbers are definitely not clear enough to draw an indisputable conclusion here. However, an important thing to keep in mind is that in the past months, I have hardly done any marketing for Proun and there wasn't a whole lot of media attention for the game. Yet sales were still as good as only a couple of months after launch. Normally, I would expect sales to drop more over time, so I think this means that removing the free version added a little bit of revenue. So despite that the evidence is not clear enough to draw a certain conclusion, I think I would advice other developers that want to do Pay What You Want to not include a free version.

Feel free to draw a different conclusion from these graphs, though, and let me know your reasoning by commenting below!

The one thing I can conclude for certain here, is that it is a really good idea to release sales statistics of your game! Not only does it help other developers to make business decisions, it also made me a thousand euro through all the media attention it got. Sharing data that is normally 'secret' turns out to be great marketing! :D

An easy solution to the safe frame problem

One of the things I hated most while developing Swords & Soldiers for the Wii, was the safe frame. CRT televisions (you know, those old, big televisions that some people still have) don't show the entire screen: they simply leave out the screen's edges. I don't know exactly why this is traditionally done, but the amount of screen that is left out varies per television and can be pretty large on some of the worst TVs.



Surprisingly, many modern LCD/LED/Plasma televisions are by default set to also leave out the edges. They just zoom in a bit. However, since there is no technical reason why this is, these televisions come with a setting to turn that irritating behaviour off.

Now the good thing, is that there is a limit to how much of the edges a television actually cuts off. The area that you can be sure will be shown on every television, is called the safe frame or safe area.



Since games need to be playable on any television, Xbox, Playstation and Wii all have requirements that the game should be playable even if everything outside the safe frame is cut off. So crucial information or interface elements can not be positioned outside the safe frame. It is of course a good thing that Microsoft, Sony and Nintendo force developers to make games that are playable for everyone, but this is pretty horrible for us as developers: usually you want the interface to be as far to the sides of the screen as possible to leave lots of playing field, but that is not possible any more.



Luckily for me, our art team had to deal with this and not me. So they moved the interface elements further from the edges of the screen. Now the total screen (as is seen on televisions that don't cut off the edges), looks like this:



Note that it actually takes quite a lot of time and effort to do this correctly: in our office we have no television that has the worst possible safe frame, so instead we made a paper frame to put in front of the screen and 'emulate' the worst possible safe frame. Everything needed to be tested with this piece of paper in front of the screen.

Moving the interface away from the edges works, but it is pretty limiting for interface design, and it is also irritating work. While making Swords & Soldiers, we thought we had to do this, but right after the game was done, we discovered that some games solve this in a much easier way. So easy, that at the time it almost felt like a trick.

So, to keep you, the reader, from spending a lot of time on the safe frame, here is the simple trick that our new game Awesomenauts and some other games use instead:

Allow the user to set the zoom of the screen.



That's it. In OpenGL and DirectX, you can just create smaller viewports and leave the borders of the screen black. So on televisions that cut off a lot of pixels at the borders, you just render the game to a smaller part of the screen. Everything outside that is black. As long as the user sets this correctly (which he is forced to do when he starts the game for the first time), everything will always be shown, so you can put your interface elements wherever you like.



The only real downside this solution has, is that the resolution of the game is now pretty much dynamic. So rendering pixel-precise effects becomes a bit more difficult now. On the other hand, it also has a hidden benefit: since you render to a smaller portion of the screen, people with CRT televisions might get a better framerate...

This really is a simple trick, but for those who didn't know it yet: be glad you do now! ^_^

Follow your heart, not your wallet!

Last November my game Proun won a Dutch Game Award for Best Original Game Design. When I received the award, I grabbed the microphone and did a short speech, which Control Magazine then asked me to write that into a column. Which I did! The Dutch version is currently on their website and in the magazine, and here is the English translation.

The Dutch games industry is doing really well. More developers than ever before! But still, if you look for truly special Dutch games, you will find painfully few. Testament to this is that at the last Dutch Game Awards, two student games and a hobby game were nominated for Best Original Game Design. Where are the companies? So many developers, so few people who create their own concepts. It's as if we have all become entrepeneurs, and left our passion and creativity at the door.

I think this is because many Dutch developers are afraid to follow their heart. With all these talented people, there must be so many good ideas floating around! Yet almost everyone is doing work for hire. Within budget, finished on time, a certain income. Hardly any creative freedom and everyone will have forgotten your game two months later.

Of course, there are exceptions. Just think about Ibb & Obb, Dinner Date, Super Crate Box, Killzone (yes, that one as well!), and of course our own Awesomenauts. And don't get me wrong: there is nothing wrong with doing work for hire. It is an important part of our industry and it is a good thing that we Dutch are as good at it as we are. But it has been taken a step too far: most developers seem to only care about a steady income.

And of course, building your own concepts also means more risk, and will fail once in a while. But I am certain that more truly special games would be born in the Netherlands, and more international success would follow!

Because truly great games are made from the heart, not from the wallet!

The code of Awesomenauts' upgrade system

Last week I explained how our designers can create tons of upgrades for Awesomenauts using our Settings system. However, I didn't say a thing about how we actually made that work. We used some fun template and functor tricks there, so I figured it would be nice to show how it was built to work elegantly, without making our code a big mess of checks for upgrades.

Warning: this blogpost contains hardcore programming awesomeness! If you are an artist or game designer and fear templatised meta-programming functors might disintegrate your brains and/or explode your head, then I advice skipping this specific blogpost and coming back next week!

First lets have a look at how things would work without upgrades. Note that to keep things readable, I only show the essential bits of code, and I have greatly simplified all the code examples here.



The function loadSettings() translates from a variable name to a string, so that we can actually load the variable from a text-file. Doing this for every setting in the game is a bit cumbersome, but necessary because variable names in C++ are not real things and become simple memory addresses when the code is compiled. Some spot is needed where we have the variable name as a string in code, and I chose to put all of those in a single place in the function loadSettings(). I guess it might be possible to write a macro that does the same thing, but I don't know about that myself.

The key thing to notice here is how easy it is to use the settings in gameplay code, as can be seen in the function handleWalking(). We are going to add upgrades to this and out most important goal for that, is to keep the gameplay code this simple.

Since the upgrade system requires each setting to be able to have several upgrades and combine them, we switch the settings from a simple float or bool to a real class.

I want basically the same behaviour from floats, bools, strings and any other types that need to be upgradeable, so I used templates for this. Experienced programmers probably know what templates are, but for those who don't: in the code below the type T is left open and it can be interpreted as whatever we say it is. So we can make an UpgradeableSetting where T is a float, or where T is a bool, or something else. C++ can compile this code with any type we want it to.



The load() function fills baseValue and upgrades by looking for all the occurrences of the setting in the settings file.

The get() function returns the value that this setting has for the Character that is passed to it. The Character is needed here, because we need to check which upgrades he has and return a different value depending on that. Its code is rather simple, but seeing it might help understanding how this works:



Note that this code only handles upgrades that replace the standard value, so upgrades that are marked with @ in the settings file. For simplicity, I have left out upgrades with + (which add to the original value) that I discussed last week.

Now that we have this, we can make some slight modifications to our gameplay code to use the upgraded value:



This all works fine, but I am not all too happy with that function get(). Normally it would be perfect, but there are so many places in the gameplay code where settings are used, that I would rather not add get() to all of those.

Now C++ happens to have a nice solution for this. I rarely see it used outside STL, but it has an absolutely awesome name: "functor". I love that name! Got to be a Decepticon! Anyway, a functor is a construction where you can call an object as if it is a function. With a functor, our little class looks like this:



Now we can do the exact same gameplay code as before, but we can simply leave out the get call. For the rest, this code does the exact same stuff. Because we use the settings so often, this is a nice improvement.



Since we have so many settings in our game, typing the entire type every time is too much work, since that would be something like UpgradeableSetting every time. We simply solved this using typedefs. At Ronimo we normally try to avoid typedefs, because they hide the real type of a value and I would rather know exactly what I am dealing with. However, because there are thousands of settings, I consider this an exception and use typedefs anyway:



Now that we have this new class, let's have a look at the same code we had before, but now with the upgrade system added to it. Note how it has hardly changed, but can do so much more now!



So, that's it. It's just a little bit if code, but this makes the settings system way more powerful and flexible. It is definitely a great improvement to the RoniTech (the engine we created here at Ronimo Games and that we used to build Awesomenauts). In fact, its uses are so diverse that we also use it for many other things than the upgrades that the player can buy in the shop. So let's conclude this blogpost with this nice example of that:

Building hundreds of upgrades for Awesomenauts

In Awesomenauts, players can buy hundreds of different upgrades. There are around 150 unique ones, and most have two or three upgrade levels, so in total there are probably around 300 upgrades in the game. We wanted our designers to be able to create those themselves, without any work from the programming team for specific upgrades. At the same time, we also wanted the upgrade system to be super flexible, so that the upgrades would be very diverse, and not just all be cooldown reductions and damage increases.



When we first started looking for a way to build this, I was really at a loss. How to add a flexible system for upgrades to our settings system? It took some time, but coding intern Daan and I came up with a system that is easy to use and so flexible, that we ended up not just using it for upgrades, but also for temporary boosts and areas with modified mechanics, like low-gravity areas.



The idea is that for every setting, our designers can create modified versions for when a character has certain upgrades. Each setting always has one base value. Then as many modified versions can be added as desired, simply by copying the setting and adding @ and the name of the upgrade.



The @ sign works for many things, but it has the problem that it is problematic to have different upgrades for the same setting: what should happen if you have both upgrades? So we also introduced the + sign, which means that that upgrade's value is added to the original value. This way several upgrades can influence the same value.



This system does not just allow for simple upgrades that upgrade a single value, but can also be used for more complex stuff. Especially since our designers can give a player a temporary upgrade when he presses a button. So let's say we want to create a skill that temporarily makes the player deal twice as much damage, but also make him half as fast and twice as big. A designer could simply use the name of a single upgrade to modify all of these values at the same time.

Of course, the more flexible and complex things get, the easier it is for our designers to mess up. We have had quite a few bugs where certain combinations of upgrades behaved in a different way than our designer intended, simply because they chose the wrong values. The plus side of these kinds of bugs is that our designers can fix them themselves, so as a programmer I am okay with that... ;)

The upgrade system also gives our artists a lot of flexibility. For example, if a bullet does more damage, our artists can upgrade its graphic and its explosion to use a different animation.



The upgrade system described in this blogpost looks pretty obvious to me now, but it definitely was a revelation to us when we came up with it two years ago, and I still think it is the most elegant solution we came up with for any problem at Ronimo Games so far (although I suppose this system is not unique and other RPGs probably use something very similar). It is incredibly flexible and really easy to use, and allows our game designers to make almost any upgrade they want. And the good thing is that it is not specific to any games, so anything we make with the Ronitech (our engine) will have this feature for our designers! ^_^

Next week, I'll explain the technical side of how we made this upgrade work in C++!