Wednesday, 25 November 2020

Arcane Glamour LIVE NOW! Blightbound's biggest update yet!

Today we've launched our biggest update for Blightbound on Steam yet! I'm really proud of what we've achieved with this one, since it not only adds a lot of cool stuff, but also makes big improvements to existing things in the game. Menus have been overhauled, the tutorial has been improved and many minor issues have been fixed. Also, some features that limited players and didn't really achieve their goals have been removed: blight, notoriety and limitations on character select.

Our previous major update is less than two months away and I think it's pretty impressive how much we've cranked out in that time. Now for the most important thing: I hope our players will like the new changes and improvements!

The new character in this update, Roland of Stendhall, also comes with a short story about how he joined the refuge. I had a lot of fun reading how vain Roderick Leeuwenhart depicts him. You can read the story (as well as the other ones) here: I, Roland of Stendhall. I'm especially font of Roderick's invention of the word magesplain: "The geologist saw all sorts of difficulties in this plan and was keen to magesplain them to me."

You can find the full list of changes in the patch notes.


Sunday, 15 November 2020

5 years below minimum wage: the financial history of Ronimo

Starting your own game company is fun and exciting, but it’s also challenging. It takes courage and skill, but above all: patience and perseverance. Some become successful quickly, but in many cases it takes years to achieve financial success and actually make a decent living out of your own game company. It might take long to make your first product or get your first deal, and that first achievement might only be a stepping stone towards a next step that brings financial stability. Today I’d like to show an example of just how long that can take by sharing the financials of the first five years of Ronimo, the company I co-founded with 6 friends nearly 14 years ago.

TLDR: The very short summary is this: it took us 2 years to make any money from Ronimo at all, 4 years to earn (almost) our country’s official minimum wage and 6 years to receive a more decent monthly salary from our own company. During that period we were near bankruptcy twice. Why did it take so long? Read on and you shall know!

A note before I continue: this blogpost is partially about how long it took us to “make a decent living”, but the cost of living differs hugely per country. The Netherlands is a wealthy country so cost of living is relatively high. A quick internet search shows that cost of living is much higher in some countries and only half in others. Since most revenue is worldwide, the same sales might mean financial stability in one place, but not enough to pay the rent in another.

Also, for anyone used to reading US dollars instead of euros: if you just replace the € sign with a $ sign, you’re in the right ballpark (especially given that the exchange rates between dollars and euros have varied a lot over the years).

In our second year of studying at the Utrecht School of the Arts our classmate Fabian Akker brought up the idea of starting a company together, with a group. Around that time we had done a couple of school projects that had failed quite miserably, so my first thought was: “we suck, let’s not.” However, the third year was to bring the first major game project, so we figured that if we could make something awesome there, then maybe we could also start a company making our own games.

The resulting game was De Blob: a huge success! We put it online and got attention from gaming press and even had some publishers contacting us, wondering whether they could buy the rights to De Blob.

(Note that De Blob was not exactly made by Ronimo: of the 9 students who made De Blob, only 5 were part of the 7 founders of Ronimo.)

Convinced by De Blob’s success, we decided to really go through with starting our own company. However, each of us still had to do a 7 months graduation project. We combined them and made starting Ronimo our graduation project. Getting school to approve of that was a bit of a struggle, but once they did, we even got our own office inside school.

At the time, 'indie' as it's known today hardly existed and we had definitely never heard of it. We thought the only way was to make retail games and that required funding from a publisher. So we set out to make a pitchable prototype: Snowball Earth. This was intended to be a Nintendo Wii game and we hoped to find publisher funding once we had graduated. At this point Ronimo didn’t make any money at all, but that was okay since we were still students.

September 2007. We were so focussed on pitching to publishers that we did our graduation stuff on the side and crunched for what came a few weeks later: Games Convention in Leipzig, Germany! There we pitched to at least a dozen publishers. Some were interested and continued conversations with us afterwards. Hopeful, we continued work on the game, looking to improve it and increase our chances of signing a deal.

Our very first presentation was for a then pretty famous person from a big company. He was so excited that… he feel asleep during out presentation. Jet lag. Or disinterest. Or both. When we woke him up, he proceeded to try to sell us his own middleware and hardly looked at our game.

By this time we had graduated and had moved to our own office in Utrecht. A very small office for 7 people, but it was cosy and exciting. What we didn’t have, however, was money. We did some minor work-for-hire jobs, but since we weren’t fully committed to that, we hardly made any money there. Just enough to pay the rent of our office, but definitely not enough to provide ourselves with any income.

This is something I've seen quite a lot: studios who want to make their own games and finance that with work-for-hire rarely succeed at both. Either they hardly make any money from the work-for-hire, or they spend so much time on that that they can hardly focus on their own game. Often the result is that the game takes many years to build and turns out mediocre because of the lack of focus and time. The reason for this is simple: doing work-for-hire well and making it lucrative is hard and it's rare for that to work as an aside, especially for inexperienced recent graduates.

So, we didn’t make any money and we weren’t students anymore. How did we not starve? This varied amongst the founders. First of all, in September 2007 we managed to sell all the rights to De Blob to THQ, a then major publisher that’s now defunct. (Note that THQ Nordic is a different company that later bought the rights to the name and games of THQ, including De Blob.) For this we were each paid a nice amount (can’t disclose it due to NDA unfortunately), enough to pay the rent for quite a while. However, only 5 of the 7 Ronimo founders were part of the De Blob team, so 2 others didn’t have this.

Six of the founders had an additional source of income: the now defunct WWIK government subsidy. This paid recent art graduates around €600 per month. That’s less than half of the official minimum wage in the Netherlands at the time, but enough to not starve. To live cheaply, three of Ronimo’s founders rented an apartment together with one more person.

I personally didn’t get WWIK because I had some savings and thus didn’t qualify, so I went even cheaper: I kept living with my mum until I was 26 years old. I have a lovely mum though so I totally didn’t mind. Thanks, mum!

This is also a good moment to mention how privileged we are to be doing this in the Netherlands. In many places in the world all of this would have been much harder.

So, how did the pitching go? A few publishers were interested and one even flew over to do due diligence: judging whether we would really be able to make the full game. In the end none of them actually offered us a deal because Snowball Earth was too unique and we were too inexperienced to be trusted with that much money. We were asking for €1.5m development budget. Not much for the big game we envisioned, but definitely too much to give to a bunch of students who had so little clue about business and production processes.

Snowball Earth was too big a game to finish without funding, so we ended up cancelling it altogether. Years later we did release our prototypes, which you can still find here together with videos and screenshots.

What next, then? By this time indie was on the rise and we had managed to get Nintendo Wii devkits. We decided to make something small that we could finish and publish ourselves: Swords & Soldiers for WiiWare (the predecessor of the current Nintendo eShop).

We estimated we could make this game in 3 months. One year later, we finished and launched it. I’m still impressed that we managed to make something of that size and quality in just one year, and I’m even more impressed that we were stupid enough to think we could make something like that in just 3 months...

Throughout this year we still didn’t make any money, but we did hire interns. In the Netherlands internships are a standard part of many schools and are not paid like normal jobs. So for only €200 per month we could have a game student work for us full-time. Despite that low compensation, those interns were getting more money from Ronimo than we were! On average, we had 2 or 3 interns at a time helping with development.

In May 2009 Swords & Soldiers launched on Nintendo Wii. It got critical acclaim, reached the #1 selling spot on WiiWare in Europe and #3 in America. In total it sold 30k copies and made €146k during the first year (and very little on WiiWare afterwards). A big success for us at the time, but not that much money in retrospect.

In August 2009, after 2.5 years of working full-time with seven people, we were finally able to pay ourselves a monthly income. A whopping €600 per month! Oh wait, that’s super little… but it certainly felt like a big step forward!

Something we hadn’t realised yet at the time is the importance of porting our games to different platforms. That is, until Sony offered us money to make a PlayStation 3 port of Swords & Soldiers, including multiplayer.

To make this port and continue work on our next game OMG Space! (which would later be renamed to Awesomenauts) we needed more programmers. Up until this point I had been the only programmer at Ronimo (besides interns) and that wasn’t enough to make a port and a new game. We hired two programmers. Unlike us, those coders were paid real wages (though not very high ones). And so, while we the founders finally made more than our interns, we instead now had employees who made way more than us.

In September 2010 Swords & Soldiers released as a downloadable game on PlayStation 3. Unfortunately it didn’t make break even, so the only money we made from this was the initial porting budget we got from Sony.

Now that we realised that porting is a super important source of revenue, we also ported Swords & Soldiers to Steam and released that in December 2010. Making a port is only a fraction of the effort of making a full game, and every new platform is a new roll of the dice: a new chance at success. And indeed, while the PS3 version hardly sold, the Steam version would make us €120k in its first year and €35k in its second year.

Now that we had employees and paid ourselves a little bit, we had significant monthly costs. Too much to carry ourselves, so we were looking for a publisher for Awesomenauts. Near the end of 2010 this was becoming dire: we were only a few months away from being out of money altogether.

We were saved when we signed a publishing deal with DTP (yet another company that doesn’t exist anymore). The total development budget we got from them was €300k. Not much for a game of this size, but it was a lot for us! As is common, we received that money spread out over milestones and not all at once.

Awesomenauts was a very ambitious project, with complex multiplayer and simultaneously launching on two platforms that were new to us (Xbox 360 and PlayStation 3). We needed more programmers to pull that off. Good thing the budget we got from the publisher allowed us to grow a bit more. In the first half of 2011 we hired two more programmers and a producer, bringing the team’s total size to 12 full time employees. On top of that we usually also had 3 or 4 interns working with us.

The funding also allowed us to finally pay ourselves almost minimum wage: €1400 per month. Still less than our employees got, but at least we felt like we were finally making real money.

In March 2012 we managed to secure some additional income: Swords & Soldiers was included in the Humble Android Bundle and this made us €37k. Towards the end of the year it got included as a bonus in another Humble Android Bundle, making us another €10k.

This money was needed desperately, since Awesomenauts had seen numerous delays at this point. I don’t remember the exact original planned release date, but I think in total the console release got delayed by around half a year. The publisher didn’t give us extra budget for that, so we had to make do with the money we had.

In May 2012 Awesomenauts finally launched on Xbox 360 and PlayStation 3. But not before our publisher DTP went insolvent a mere week for launch. This made everything extremely complex and we didn’t know whether we would see any royalties at all. We got lucky: we had some unreleased DLC they wanted so we managed to strike a deal with the trustee for the insolvency so that the DLC would be released and we would still get royalties.

Nevertheless, Awesomenauts initially didn’t sell all that well on consoles and it took long before we got any royalties at all. We were nearly out of money but had one more card to play: a Steam port of Awesomenauts. Finances were so tight that we couldn’t pay ourselves anymore for a short period. We continued to pay our employees though, so only the founders were hit.

Then in August 2012 Awesomenauts launched on Steam and this version turned out to sell way better than the console versions. We were saved! And we had gotten lucky again with our publisher: since DTP was insolvent, they couldn’t pay for development of the Steam port of Awesomenauts, and thus we got the full rights to that version.

Awesomenauts kept doing very well so we supported it for 5 more years with tons of additional content. It also allowed us to finally switch what type of company we were: we switched from being a V.O.F. to a B.V. These are Dutch legal terms so let's not go into the details here. What it comes down to, is that a V.O.F. is strongly tied to the owners’ personal finances. If the company goes bankrupt, so does the owner personally. Being a B.V. is much safer, since now the company can go bankrupt without giving creditors the right to come after your personal belongings as well.

Being a B.V. does come with a requirement here in the Netherlands: unless you have good reason not to, the company needed to pay the active owners at least €2300 per month (after taxes). So in February 2013, six years after we started the company, we finally started to make a wage significantly higher than minimum wage. And even then it wasn’t that much: this excludes some insurances that are standard for employees but not for owners, and for me personally as a programmer: I’m pretty sure I could have made more had I worked elsewhere.

As far as I can tell, most game startups take several years to become financially successful. I think it might have taken us longer than most, but we made it at all and that’s already special. In fact, since the 'indiepocalypse' happened a few years ago most people who start a game company never manage to make a living at all (as I've previously said: the future of indie is amateur). With Ronimo we were lucky that we happened to start our company at a time when indie was hip and happening and it was relatively easy to reach players. Today, competition is much tougher than it was when we started, so chances of success are lower as well.

What’s the moral of this very long story? It’s simple: to start a game company, you don’t need just skill, vision and bravery, but also perseverance and a willingness to make little money for a long while.

Friday, 6 November 2020

Combining 2D and 3D in Blightbound's VFX

An important focus during development of Blightbound is that we we wanted to achieve a 2D look but have 3D movement and a 3D camera. A particular challenge are special effects: we had tons of experience with 2D special effects, but now the special effects also needed to communicate depth. For example, how far does that area-of-effect damage reach exactly? In this blogpost I will show a number of tricks we used to combine 2D and 3D in the Blightbound VFX.

This blogpost is based on an conversation I had with Ronimo VFX artist Kees Klop. Unfortunately Kees won’t be staying at Ronimo: for budget reasons we can’t keep him on after November. So, if you’re looking for a stellar VFX artist, be sure to send him a message on his Art Station or email.

Let’s start by having a look at the Gravity Well skill that some of the mages have. This is a spell that draws in all the enemies near it, making for an excellent combo with area-of-effect attacks by the Mage’s teammates. Like many of the VFX in Blightbound, this effect combines several types of geometry. Here's what it looks like, and a breakdown of some of its elements.


A video of the Gravity Well effect in-game and in our animation editor.

The next effect I would like to discuss is the Deck of Daggers skill that some rogues have. Several knives are thrown, and those that hit get stuck in the enemy for a while, causing damage over time. These knives are rotated out of the enemy's plane to add depth. It's a subtle effect, but having many of such subtle 3D effects in the game in total adds a lot of depth to the 2D drawings.


The Deck of Daggers effect or, as Triss would say it: "I'm a fan of knives!"

Also, our animation tech helps here: since Blightbound plays skeletal animation in real-time (as opposed to rendering it down to spritesheets as we previously did), objects like these knives can really stick to a body part and move along with it. While that's an obvious option to have on 3D characters, it's a lot less common to be able to do this with 2D animation without using a rather stiff animation technique. Our animation workflow however is a topic for a separate, future blogpost.

One of the most screen-filling effects in the game is the victory at the end of a dungeon, after defeating the dungeon's boss. Here we see a combination of many elements, including one that's not used often in our special effects because it's so all-encompassing: colour grading. The colours of the whole scene get changed during this effect.


The victory effect in-game and in our animation editor. Blight (a corrupting mist) is an important theme in Blightbound. Since this effect marks the end of a successful run, it sucks in and dissolves a lot of mist.


Many special effects in Blightbound are made 3D by overlapping flat planes under various angles.


A lot of the visual effects also feature bits of frame-to-frame animation. Many of those were made by Ronimo while many others were taken from the RTFX Generator pack, including this particular one.

The warrior's Warcry ability shows another technique for combining 2D and 3D. Warcry is an area of effect ability that buffs teammates. Since the range of the ability is very important, the starting point here is a circle on the ground. However, that's very flat and becomes less readable when sticking through things like grass or small stones. So to make the effect more 3D, a vertical cylinder with an animating swirl is added.


The Warcry skill in-game and in our editors.


Here's a little teaser: update 0.5 (coming this November) adds the new Tamed Wolf sword to Blightbound. A special perk of this sword is that when the warrior uses his shield, his teammates are also shielded. This effect visually overlaps with the Warcry skill. As you can see here, this shield effect makes the verticallity above the circle even stronger.

On to the assassin's Chakram ability! This is a large projectile that flies, hangs still for a while and then comes back. Unlike most other projectiles it's visible long enough that it's much more than just a flash. Also, it deals damage in an area of a meter or two, so it's quite large.


The Chakram skill in Blightbound.

The Chakram is a horizontally flying circular thingy. With the relatively low camera of Blightbound it was seen under too extreme an angle, making it look too flat. We didn't want to turn it into a 3D model though since we wanted to maintain the 2D, handpainted feel of the graphics. The solution is simple: the Chakram is tilted towards the camera a bit. This is subtle enough that it feels like the Chakram is horizontal, but it adds just enough angle that it looks a lot less flat. This is a technique that we use a lot in Blightbound.


While the main graphic of the Chakram is entirely flat, it's made more spatial by animating the pitch and adding particles and swooshes that move out of the plane.

The special effects shown in this blogpost were all made by Kees Klop, whose work I wanted to celebrate today. However, he is not the only special effects artist who worked on Blightbound: Koen Gabriƫls and Luuk van Leeuwen also made a lot of VFX. Koen did most of the early work of establishing the style for the Blightbound effects. Currently Koen and his intern Ayrthon van de Klippe are working on making VFX for upcoming Blightbound updates.

Finding the right combination of 2D and 3D in the VFX for Blightbound was quite a search during early development of Blightbound. Our artists ended up combining a lot of different techniques, including hand-drawn art, frame-to-frame animations, meshes, particles, intersecting planes, screen distortions and even colour grading. I think the end result works really well in-game: it looks good, fits the style and communicates the gameplay well.

Saturday, 24 October 2020

Bending Blightbound's world to lower the horizon

In 3D games perspective is often treated as a given; a law of nature. But it doesn’t have to be that way: with some shader trickery or clever modelling, perspective can be manipulated to achieve certain compositions that may not be realistic, but look more interesting and are still convincing to the player. One such example is how we kept the horizon on screen in our new game Blightbound by subtly bending the world.

At Ronimo we come from a world of 2D games. In 2D, composition can be whatever you like. That’s why our art director Gijs Hermans may sometimes want to ignore standard perspective rules and instead looks at what he wants to achieve visually. Thus early in development Gijs came to me and said he wanted the camera to look down quite a bit, but still have the horizon in view. In fact, he wanted the horizon to be quite a bit below the top of the screen. His reasoning was that visuals look much better when you don’t see just the floor most of the time. Het position of the horizon is an important tool for shaping a composition.

The origin of this request is a clash that often happens in game development: pretty visuals versus gameplay clarity. Our artists spend a lot of time on achieving both goals simultaneously. A very successful example of this is the way gameplay objects and backgrounds are drawn in a different style in our previous game Swords & Soldiers 2, as I described in this blogpost.

In a game where depth matters, like Blightbound, a low camera is problematic because it makes it difficult to see whether you are standing in front of an enemy or behind them. A high camera solves this, but a high camera removes the horizon from view, making the image a lot more boring.

When Gijs came to me with this request, I thought of two possible solutions: either give the camera a wider field-of-view, or bend the world to move the horizon down. We tried the easiest solution first: wide field-of-view. However, it turned out this needed to be set so wide that the entire perspective looked skewed. Extreme field-of-view often isn’t very pretty and it definitely wasn't in Blightbound.

The alternative I came up with is bending the world down the further it is from the camera. This is an effect that’s used in a bunch of games to create a sense that the world is very small, making the world feel cutesy and funny. However, Blightbound is intended to be a dark fantasy game, definitely not something cute and funny, so we didn’t want anything that extreme. I figured that with some tweaking it might be possible to achieve a more subtle version of this that still keeps the horizon in view but doesn’t have the funny vibe.

My implementation of this effect is quite simple. In the vertex shader I bend down the world depending on the Z-position of the vertex in the world. The nice thing of implementing it this way is that our gameplay code and level design tools can assume a flat world, making them a lot simpler. The bending only exists during rendering, so gameplay logic doesn't need to take it into account.

A minor challenge in implementing this bend is how to handle lighting and shadows. When the camera moves forward and the world bends, we don’t want the lighting on objects to change, since that would make the bending very obvious and would make the player focus on the backgrounds instead of on the gameplay. Also, objects in the background shouldn't be more bright because they are rotated towards the light by the bend. My solution was to calculate all lighting, shadows and fog as if there is no world bend.

Also, a little technical note: since the bend happens on the vertices, objects need to have enough vertices. A big square plane for the ground with no vertices in between can’t be bent. Occasionally this caused bugs where a small object would float above a big object because the big object didn’t have enough vertices to be bent correctly.

The bend effect is quite fun to see in action when set to an extreme value. However, any kind of geometric deformation is quite noticeable when the camera moves, so we chose to fix the bend in the world instead of letting it move with the camera.


A few different settings for the bend, including the final one used in Blightbound.

As you can see in the video, the bend effect is kept quite subtle in Blightbound. We didn’t want that cutesy/funny effect at all, since this is intended to be a dark fantasy game. Our level artist Ralph Rademakers tweaked the effect and the camera a lot until he got it to a point where it felt like there was no bend at all, just a natural camera. However, if you compare with and without bend, you can see that the bend makes a huge difference in what you actually see. And that’s exactly how it was intended: achieve the desired composition but don’t make it look like anything weird is going on.

And then came the fog! The bend effect was implemented when we hadn’t figured out the lore of the world yet. We didn’t know then that we would want to have so much fog. In fact, the working title of the game used to be AwesomeKnights instead of Blightbound! Once we finally decided on the lore we knew that the world of Blightbound is covered in “blight”, a corrupting fog. To match that, Ralph added a lot of fog to all the levels. This creates a great atmosphere, but… hides the horizon!

Does that make the world bend useless? No, definitely not. It's still used in quite a few levels to change the perspective and have a more horizontal view on the background, even if we can’t see as far as before. It’s a more subtle tool than originally intended, but still a very useful tool.

I think the bend effect we used here is a wonderful example of the kind of graphics programming I enjoy most: looking at what’s needed from an artistic standpoint, and then making tech that achieves that. I’m personally not very interested in realistic rendering: 3D is just a tool to make cool art, whatever the shape or type. The bend technique used here makes no sense whatsoever from a physical standpoint, but it adds to making Blightbound a prettier, more compelling game.

Sunday, 18 October 2020

Screen Space Reflections in Blightbound

An important focus during development of our new game Blightbound (currently in Early Access on Steam) is that we want to combine 2D character animation with high quality 3D rendering. Things like lighting, normal maps, depth of field blur, soft particles, fog and real-time shadows are used to make it all gel together and feel different from standard 2D graphics. One such effect I implemented into our engine is SSR: Screen Space Reflections. Today I’d like to explain what SSR is and what fun details I encountered while implementing it into our engine.

A compilation of places with reflections in rain puddles in Blightbound.

Reflections in games can be implemented in many ways. The most obvious way to implement reflections is through raytracing, but until recently GPUs couldn’t do this in any reasonable way, and even now that GPU raytracing exists, too few people have a computer that supports it to make it a feasible technique. This will change in the coming years, especially with the launch of Xbox Series X and Playstation 5, but for Blightbound that’s too late since we want it to look good on currently common GPUs.

So we need something else. The most commonly used techniques for implementing reflections in games are cubemaps, planar reflections and Screen Space Reflections (SSR). We mostly wanted to use reflections for puddles and such, so it’s important to us that characters standing in those puddles are actually reflected in real-time. That means that static cubemaps aren’t an option. A pity, since static cubemaps are by far the cheapest way of doing reflections. The alternative is dynamic reflections through cubemaps or planar reflections, using render-textures. These techniques are great, but require rendering large portions of the scene again to render-textures. I guessed that the additional rendercalls and fillrate would cost too much performance in our case. I decided to go for Screen Space Reflections (SSR) instead.

The basic idea behind SSR is that since you’re already rendering the scene normally, you might as well try to find what’s being reflected by looking it up in the image you already have.


SSR has one huge drawback though: it can only reflect things that are on-screen. So you can’t use a mirror to look around a corner. Nor can you reflect the sky while looking down at the ground. When you don't focus on the reflections this is rarely a problem, but once you look for it you can see some really weird artefacts in reflections due to this. For example, have a look at the reflections of the trees in this video of Far Cry 5.



So we’re looking up our reflections in the image of the scene we already have, but how do we even do that? The idea is that we cast a ray into the world, and then look up points along the ray in the texture. For this we need not just the image, but also the depth buffer. By checking the depth buffer, we can look whether the point on our ray is in front of the scene, or behind it. If one point on the ray is in front of whatever is in the image and the next is behind it, then apparently this is where the ray crossed the scene, so we use the colour at that spot. It’s a bit difficult to explain this in words, so have a look at this scheme instead:


Since SSR can cast rays in any direction, it’s very well suited for reflections on curved surfaces and accurately handles normal maps. We basically get those features for free without any extra effort.

A demo in the Blightbound engine of SSR, with a scrolling normal map for waves.

At its core SSR is a pretty simple technique. However, it’s also full of weird limitations. The challange of implementing SSR comes from working around those. Also, there are some pretty cool ways to extend the possibilities of SSR, so let’s have a look at all the added SSR trickery I implemented for Blightbound.

First off there’s the issue of transparent objects. Since the world of Blightbound is covered in a corrupting mist (the “blight”), our artists put lots of foggy layers and particles into the world to suggest local fog. For the reflections to have the right colours it’s pretty important that these fog layers are included in the reflections. But there are also fog layers in front of the puddles, so we can't render the reflections last either.

The solution I chose is simple: reflections use the previous frame instead of the current frame. This way we always look up what's being reflected in a complete image, including all transparent objects. The downside of this is of course that our reflection isn't completely correct anymore: the camera might have moved since the previous frame, and characters might be in a different pose. However, in practice the difference is so small that this isn't actually noticeable while playing the game.


Transparent objects pose another problem for SSR: they're not in the depth buffer so we can't locate them correctly. Since Blightbound has a lot of 2D animation, lots of objects are partially transparent. However, many objects can use alpha test. For example, the pixels of a character's texture are either fully transparent of not transparent at all. By rendering such objects with alpha test, they can write to the depth buffer without problems.

This doesn't solve the problem for objects with true transparency, like smoke, fog and many other special effects. This is something that I don't think can be reasonably solved with SSR, and indeed we haven't in Blightbound. If you look closely, you can see that some objects aren't reflected at all because of this. However, in most cases this isn't noticeable because if there's an opaque object closely behind it, then we'll see the special effects through that. While quite nonsensical, in practice this works so well that it seems as if most explosions are actually reflected correctly.

Transparent objects like this fire aren't in the depth buffer and thus can't be found for reflections. However, if another object is close behind, like the character on the left, then the transparent object is reflected through that. The result is that it seems as if many special effects are properly reflected.

Having perfectly sharp reflections looks artificial and fake on many surfaces. Reflections in the real world are often blurry, even more so the further the reflected object is from the surface. To get reflections that accurately blur with distance I've applied a simple trick: the rays get a slight random offset applied to their direction. This way objects close remain sharp and objects get blurrier with distance. Artists can tweak the amount of blur per object.

However, this approach produces noise, since we only cast one ray per pixel. We could do more, but that would be really heavy on performance. Instead, to reduce the noise a bit, when far enough away I also sample from a low-resolution blurry version of the scene render-texture. This is a bit redundant but helps reduce the noise. Finally, by default in Blightbound we do supersampling anti-aliasing (SSAA) on the screen as a whole. This results in more than one ray being shot per screen-pixel. Only on older GPUs that can't handle SSAA is this turned off.


Another issue is precision. For really precise reflections, we would need to take a lot of samples along the ray. For performance reasons that's not doable though, so instead we make small jumps. This however produces a weird type of jaggy artefacts. This can be improved upon in many different ways. For example, if we would render the reflections at a lower resolution, we would be able to take a lot more samples per ray at the same performance cost. However, with how I implemented SSR into our rendering pipeline that would have been quite cumbersome, so I went for a different approach which works well for our specific situation:
  • More samples close to the ray's origin, so that close reflections are more precise.
  • Once the intersection has been found, I take a few more samples around it through binary search, to find the exact reflection point more precisely. (The image below is without binary search.)
  • The reflection fades into the fog with distance. This way the ray never needs to go further than a few meters. This fits the world of Blightbound, which is basically always foggy.
The result of these combined is that we get pretty precise reflections. This costs us 37 samples along a distance of at most 3.2 meters (so we never reflect anything that's further away than that from the reflecting surface).

Any remaining imperfects become entirely unnoticeable since blur and normal maps are often used in Blightbound, masking any artefacts even further.



A challenge when implementing SSR is what to do with a ray that passes behind an object. Since our basic hit-check is simply whether the previous sample was in front of an object and the next one is behind an object, a ray that should pass behind an object is instead detected as hitting that object. That's unintended and the result is that objects are smeared out beyond their edges, producing pretty ugly and weird artefacts. To solve this we ignore a hit if the ray dives too far behind an object in one step. This reduces the smearing considerably, but the ray might not hit anything else after, resulting in a ray that doesn't find an object to reflect. In Blightbound we can solve this quite elegantly by simply using the level's fog colour for such failed rays.

By default, SSR produces these kinds of vertical smears. Assuming objects have a limited thickness reduces this problem greatly, but adds other artefacts, like holes in reflections of tilted objects.

This brings us to an important issue in any SSR implementation: what to do with rays that don't hit anything? A ray might fly off the screen without having found an object, or it might even fly towards the camera. Since we can only reflect what we see, SSR can't reflect the backs of objects and definitely can't reflect anything behind the camera. A common solution is to have a static reflection cubemap as a fallback. This cubemap needs to be generated for each area so that the reflection makes sense somewhat, even if it isn't very precise. However, since the world of Blightbound is so foggy I didn't need to implement anything like that and can just fall back to the fog colour of the area, as set by an artist.

The final topic I would like to discuss regarding SSR is graphics quality settings. On PC users expect to be able to tweak graphics quality and since SSR eats quite a lot of performance, it makes sense to offer an option to turn it off. However, what to do with the reflecting objects when there's no SSR? I think the most elegant solution is to switch to static cubemaps when SSR is turned off. Static cubemaps cost very little performance and you at least get some kind of reflection, even if not an accurate and dynamic one.

However, due to a lack of time that's not what I did in Blightbound. It turned out that just leaving out the reflections altogether looks quite okay in all the spots where we use reflections. The puddles simply become dark and that's it.


For reference, here's the shader code of my SSR implementation. This code can't be copied into an engine directly, since it's dependent on the right render-textures and matrices and such from the engine. However, for reference when implementing your own version of SSR I expect this might be useful.

SSR is a fun technique to implement. The basics are pretty easy, and real-time reflections are a very cool effect to see. As I've shown in this post, the real trick in SSR is how to work around all the weird limitations inherent in this technique. I'm really happy with how slick the reflections ended up looking in Blightbound and I really enjoyed implementing SSR into our engine.

Monday, 7 September 2020

The games that inspired me most

A few years ago I was asked by a videogame magazine what games inspired me most. I had a lot of fun compiling a short list for them. Today I've updated that list. These aren't necessarily my favourite games (although they're close): Doom II isn't on this list for example. I'm going to focus not on why I had fun with a game, but why it inspired me, how it taught me something new. They all have some unique aspects that changed how I look at games and game development.

Command & Conquer

As a kid I was so much a fan of Command & Conquer that I even rebuilt some of its units with Lego. I started playing around with 3D computer graphics when I was only 13 years old and C&C was my big inspiration. I wanted to make 3D as awesome as in the cut-scenes for this game. Especially the video where a flame tank drives into a town and torches everything is still vividly in my mind. Many of the game designs that I wrote down on paper at this age were for RTS games inspired by C&C. 



Star Control II

This is easily my favourite game of all time. Great music, crazy humour, fun combat and a rich universe with deep backstories and tons of personality. What is most special though is that this is one of the few games I know where information and exploration are intertwined so strongly. Everything is there at the start of the game, but the universe is way too big to find things without knowing the coordinates of interesting planets. Gathering information and even buying it is a key feature here. In combination with the lack of invisible walls this makes exploration extremely rewarding and quite different from most open world games.


An interesting aside is that in the newly released Star Control: Origins (which I really like, by the way) some players complain about the universe being so empty that grinding through planets is boring. The problem here is that modern players have been taught that it's useful to randomly wander around the world, because most open world games these days make sure there are interesting things everywhere. That's fun of course, but that means that in a game like Fallout information isn't very important: by randomly walking around you'll likely find everything (although that does cost a lot of time). Star Control's exploration is totally different because the universe is so much bigger. I think the problem in Star Control: Origins is purely expectations: it isn't clearly communicated to players that this game is different.

Another aspect of Star Control II that has greatly influenced me is the music. Each alien race you encounter has a theme song that captures the personality of that race. Music in many games (as well as movies) mostly serves to set the mood and influence the viewers emotions. But I like music that takes the foreground and really adds to the personality of the game. Inspired by the alien themes from Star Control II, I pushed for having character theme songs in our game Awesomenauts as well. This resulted in Sonic Picnic making an amazingly diverse soundtrack for Awesomenauts, adding tons of personality.

Bioshock

In most games the story and the gameplay are told intertwined but are not truly related. Not in Bioshock: here the gameplay and the story revolve around a common theme. Both add meaningful insights to the concepts of freedom and choice that are explored. Bioshock is not just a marvellously fun game; it also made me ponder the philosophical themes it explores. Especially the twist halfway the game felt incredibly smart and exciting. I felt like I had personally fallen victim to exactly what had happened to the character in-game. One of the few games with true philosophical depth.


World of Goo

World of Goo is the ultimate example of how much creativity is needed to extend a simple and original concept to a full game. It seems to me like many of the participants of gamejams think that coming up with something original is the highest goal in game design. But much more difficult is to develop that original concept from a gimmick into a full game. World of Goo started out as a gimmicky mini-game and ended up as a glorious game. It adds enormous diversity and depth without adding a lot of features or complexity. This is true mastery of game design.


Deus Ex

The original Deus Ex is a great game in general and the combination of RPG + FPS is pure gold for my taste in games. However, the most unique thing in this game to me is the ending. Usually when games have different endings, they are very black-and-white. There are the good ending and the evil ending. Or maybe there were extra challenges and then there are a good ending and a super good ending. Not in Deus Ex: here the three endings all have up- and downsides. It's not clear which would be the best for humanity and the options are clever enough that I spent considerable time pondering each. Deus Ex has similarly deep philosophical thoughts as Bioshock, but while Bioshock is about the game manipulating the player, Deus Ex is about the player manipulating the world.


Dear Esther

A core aspect of game design to me is that a game should be fun without the graphics and the polish. If a game is already fun with just boxes, then it will be even better when visuals and sound are added. Dear Esther is a fantastic example of a game that proves me wrong. In fact, there's hardly any gameplay. But the storytelling, the music and the visuals all add up to an incredible experience. Dear Esther is also interesting in that it suggests a lot and explains very little. I played this game with my wife and we were constantly discussing what it all meant. The game continuously pushed our thoughts into different directions with new snippets of information.


Railroad Tycoon

I am a big fan of trains and always want to see more of them in games. I fondly remember Railroad Tycoon, Transport Tycoon and Sid Meier's Railroads. As a teen I wrote tons of pages of notes on an RTS that resolved around building railroads to transport resources. Trains could be armed to defend them and gather the resources needed to build an army. I doubt that game will ever happen, but anything with trains makes me all giddy with excitement. In fact, most of the game ideas I come up with today still have room for trains somewhere. I hope that in a few years I will finally have made at least one game that actually features trains!


Ori And The Blind Forest & Rayman Origins


I'm grouping these two games together because they have done a similar thing: they have both raised the bar for 2D art in games. 2D art is sometimes seen as lesser than 3D art, but Ori and Rayman have shown that it's possible for a 2D game to have the appeal of a big AAA production. At Ronimo we have studied the animation, special effects and level art in these two games in detail to see what techniques we could apply to our own games.


Braid


Of all the games in this list, Braid has probably influenced me most, and on several levels. For starters, Braid introduced us to the concept of self-publishing and indie development. Xbox Live Arcade had been around for a little while already but Braid, World of Goo and Castle Crashers really showed us the potential of indie and self publishing. When we started Ronimo we thought we needed to make disk based full price games. This was however not a path we were succeeding at so when we realised that we could do what these games were doing, we adjusted our plans and made a smaller game on our own instead. This became our first commercial release: Swords & Soldiers on Nintendo Wii.

Braid also influenced me on a more personal level. My interpretation of Braid's story is that it's not about an actual princess, but about the danger of looking for perfection. The main character has a girlfriend, but he breaks up with her because she isn't perfect. So he looks for the perfect woman, his 'princess'. But he doesn't find her because no one is perfect. The result is that he's alone and doesn't have his actually quite wonderful girlfriend anymore. In the epilogue the game also mentions other examples of the dangers of obsessing over perfection, like a scientist looking for the perfect science but creating the nuclear bomb as part of the Manhattan Project.


I don't know whether the story was truly meant like this, but this interpretation has greatly influenced me in my view on both creativity and life. Perfection is an impossible goal so striving for it only hampers you. Perfectionism causes you to spend too much time looking for an even better game concept to make, or endlessly iterating your game instead of releasing it and making another, or breaking up with your partner instead of making it work. Braid has truly changed my mindset in what to strive for in life and work.

So, that's it: the games that inspired me most! What games changed your view on games or the world?

Sunday, 9 August 2020

Three nasty netcode bugs we fixed around Blightbound's launch

In the weeks before and after the Early Access launch of our new game Blightbound we've been fixing a lot of bugs, including 3 very stubborn netcode bugs. They weren't any fun while we were looking for them for days, but now that we've found and fixed these they're actually quite intriguing. So today I'd like to share 3 of the more interesting bugs I've seen in the past few years.
While these bugs mostly say something about issues in our code, I think they're also nice examples of how complex low level netcode is, and how easy it is to make a mistake that's overlooked for years, until a specific circumstance makes it come to the surface. All three of these bugs were also in the code of our previous game Awesomenauts, but none of them actually occurred there due to the different circumstances of an Awesomenauts match compared to a Blightbound party.

The one thing all three of these bugs have in common is that they were very hard to reproduce. While they happened often enough to be game-breaking, they rarely happened in testing. This made them very hard to find. For the first two we found a way to reproduce them after a lot of experimenting, while for the final one we never got it ourselves until we had figured out the solution and knew exactly how to trigger it. Finding that one without being able to see it was some solid detective work by my colleague Maarten.

Endless loading screens number 1


This bug occasionally caused clients to get stuck in the loading screen forever. After adding additional logging we saw that the client never received the HostReady message. This is a message that the host sends to the clients to let them know that loading is done and they should get into the game. The client was forever waiting for that HostReady message. Upon further investigation it turned out that the host did actually send the message, but the client received it and then threw it away as irrelevant. Why was that? 

To answer that question, I need to give some background information. An issue with the internet is that it's highly unreliable. Packets can be lost altogether and they can also come in out-of-order or come in very late. One odd situation this can cause, is that a message from a previous level can come in once you're already in the next level, especially if loading times are very low. Applying a message from the previous level to the next level can cause all kinds of weird situations, so we need to recognise such an outdated message and throw it away. We do this with a simple level ID that tells us which level the message belongs to. If we get a message with the wrong level ID, we discard it.

That's simple enough, but we saw that both client and host were in the loading screen for the next level. Why then would the client discard the HostReady message as being from the wrong level? The reason turned out to be in our packet ordering system. Some messages we want to process in the order in which they are sent. So if one message goes missing, we store the ones after it and don't process them until the previous missing one is received and handled.

The bug turned out to be in that specific piece of code: the stored messages didn't store their level ID. Instead, when they were released, we used the level ID for the message that had just come in. This went wrong when a message from the previous level went missing and came in a second or two too late. By that time the HostReady message had already been received, but had been stored since it needed to be processed in order. Since the level ID from the previous room was used for all out-of-order messages, the HostReady message was processed as if it came from the previous room. Thus it was discarded.

I imagine reading this might be mightily confusing, so hopefully this scheme helps explain what went wrong:

Once we had figured this out, the fix was simple enough: store the level ID with each message that is stored for later processing, so that we know the correct level ID  when we get to it.

Endless loading screens number 2

The second loading screen bug we had around that time had a similar cause, but in a different way. When a client is done loading, it tells the host so in a ClientReady message. The host needs to wait for that. Similar to the bug I described above, the ClientReady message was received but discarded because it was from the wrong level. The cause however was a different one.

Sometimes the client starts loading before the host, for whatever reason. Since loading times between levels are very quick in Blightbound, the client can be done loading before the host starts loading. Once the client is done loading, it sends the ClientReady message. However, if the host isn't loading yet and is still in the previous room, then it discards the ClientReady message because it's from the wrong room. A moment later the host starts loading and when it's done loading, it starts waiting for the ClientReady message. Which never comes because it was already received (and discarded).

Here too the solution was pretty straightforward. We first changed the level ID to be incremental (instead of random) so that we can see in the level ID whether an incorrect message is from the next or the previous room. If it's from the previous room, we still discard the message. But if the message is from the next room, we store the message until we have also progressed to that room, and then process it.

The big desync stink


The final netcode bug I'd like to discuss today is rather painful, because unlike the others it still existed at launch. In fact, we didn't even know about it until a day after release. Once we knew it existed it still took us days to find it. In the end this one was fixed in update 0.1.2, six days after launch.

So, what was the bug? What players reported to us was that randomly occasionally the connection would be lost. In itself this is something that can happen: the internet is a highly unreliable place. However, it happened way too often, but never when we tried to reproduce it, not even with simulated extremely bad connections. Also, we have code that recognises connection problems, but this bug didn't trigger that code, so the game thought it was still connected and kept playing, with very odd results.

After a few days a pattern started to emerge: many players reported that this always happened once they had notoriety 3 or 4, when a legendary item was attainable. We didn't think that the bug was actually linked to notoriety or legendary items, because these are quite unrelated to netcode. Instead, we suspected something else: maybe the bug happened after playing together for a certain amount of time.

With this hypothesis in hand we started looking for anything where time matters. Maybe a memory leak that grew over time? Or, more likely: maybe some counter that looped incorrectly? As I described in the above issues, netcode can have many types of counters. I already described the level ID counter and the counter for knowing whether a message is received in-order. There are others as well.

The thing with data that is sent over the netwerk is that it needs to be very efficient. We don't want to send any bytes more than necessary. So whenever we send a counter, we use as few bits as possible. Why send a 32 bit number when 16 bits will do? However, when using smaller numbers they will overflow more quickly. For example, a 16 bit number can store values up to 65,535. Once we go beyond that, it loops back to 0.

Often looping back to 0 isn't a problem: the last time we received the lower packets is so long ago that we won't have to expect any overlap with the last time we used number 0. However, we do need to handle the looping numbers correctly and not blindly throw away number 0 because it's lower than number 65,535.

And this indeed turned out to be the problem. One of the counters we used was a 16 bit number and after about one hour it looped back. In that particular spot we had forgotten to include code to handle that situation. The result is that from there on the game received all messages but discarded them without applying them. Like in the cases above, the solution was simple enough to implement once we knew what the problem was.

One question remains: why wasn't this a problem in Awesomenauts? For starters, Awesomenauts matches are shorter and the connection is reset after a match. So the counter never reaches 65,535. In Blightbound however the connection is kept up in between dungeon runs, so if players keep playing together the number will go out of bounds. But there's more: if the bug would have happened in Awesomenauts, the game would have thought it was a disconnect and would have reconnected, resetting the counter in the process. In Blightbound we determine connection problems in a different way, causing the connection to not be reset in this particular case.


A lesson relearned


Netcode being hard is not new to us and getting weird bugs is inherent to coding such complex systems. However, there is one step we did for Awesomenauts that we didn't do here: auto testing. Since Blightbound uses a lot of netcode from Awesomenauts and Awesomenauts has been tested endlessly, we thought we didn't need to do extensive auto testing for this same code in Blightbound. This was a mistake: the circumstances in Blightbound turn out to be different enough that they cause hidden bugs to surface and wreak havoc.

We're currently running autotests for Blightbound to further stabilise the game. This already resulted in another netcode fix in update 0.1.2 and several obscure crash fixes that will be in the next major update.

To conclude I'd like to share this older but still very cool video of auto testing in Awesomenauts. It shows four computers each randomly bashing buttons and joining and leaving matches. The crazy movement of the Nauts is fun to watch. For more details on how we implemented auto testing in Awesomenauts, have a look at this blogpost.