Here's how I made my computer write music

One of my current hobby projects is creating a procedural music system: a computer program that composes and performs music all by itself. It's still a work-in-progress in its early stages, but some recent improvements have gotten it to produce quite interesting music already, so I figured it's about time I share some of my thoughts around how it works. Let's start with a little video that lets you hear the amount of variation it can currently do:



Note that this video contains a selection of the nicer pieces, so it isn't all this good. Most of the other music it currently produces is a bit more boring, or too similar to these fragments. The program is a work-in-progress and it should improve a lot as I spend more time on it.

I started programming this for a rather odd reason: I wanted to practice improvising on my cello over complex chord schemes and figured a program that automatically produces those would do the trick. As usual with this kind of thing, it got out of hand and I've so far spent my time making it cooler instead of playing along to its tunes on my cello. I hope I can expand it to the point where it can be a game soundtrack, or a live streaming radio channel on Youtube that constantly produces new music.

My approach to procedural music is mostly based on my own ideas about composition. I'm deliberately steering away from a lot of common composition 'rules' since I expect that's been done already. I think I'll get something more interesting and more personal if I start from my own ideas instead. While I don't ignore music theory either, I do try to avoid rules like ending a chord progression with the dominant and then going back to the tonic or anything like that.

Just to be clear: my music generator is not an AI or a self-learning system or anything like that.

The procedural music generator consists of a bunch of elements that together produce the music:

Chords, rhythms and scales
To 'teach' my program basic music theory I've created a bunch of text files that define a lot of common and less common chords, rhythms and scales. You can see which it's using in the video: it might for example choose to play a C major chord in 3/4 rhythm.



Structure generators
The first thing needed to create a piece of music, is a basic structure. This part of the program randomly chooses the chord progression, the rhythm and the scales. It also chooses how individual instruments are allowed to deviate from the rhythm, so that there's some consistency between instruments without requiring them to do the exact same thing.

Instrument layer generators
These generate the notes for specific instruments. Each category of instruments has a role to fulfil and its own specific algorithm for that. For example, the generator for drums is completely different from the generator for bass. Most of these generators are still very basic so there's plenty of room for improvement there. A lot of the variation in the video above comes from having different combinations of instrument layer generators. For example, some have drums, others do not. I expect that having a lot of layer generators is where my procedural music will get most of its variation. Currently I have five, but I'd like to create at least a dozen. The most important missing layer at the moment is one for actual melodies, which is probably also the most challenging one to get right.

VST instruments
Actually performing the music is done using VST instruments. VST is a standard for digital instruments that is used by a lot of programs on Windows. A VST file is basically just a DLL and I've coded a simple music player that can handle them. For each instrument layer I'd like to have several VST instruments that can play that layer, so that I can get different sounds. Currently most of it sounds very digital, so it would be nice to also have VSTs for other types of instruments, like acoustic drums and guitars.

Song structure
This is an element that's currently completely missing: at the moment all music played by my music generator is simply a loop of 4 to 12 bars. I want to program systems for song structures with verse/chorus/bridge, but also for endlessly progressing music and for music that for example slowly builds up to a climax.

Carefully restricted randomness
All the systems in this generator contain a lot of randomness, but the randomness is limited to specific rules I've come up with. If those rules are too free then the result is noise, not music, while if the rules are too strict then the resulting music will always seem same, so carefully balancing the randomness is a big challenge here.



For future improvements I'd like to add more instrument layers and refine the current ones, add song structures and add more VST instruments. Since I've recently become a dad I don't have a whole lot of time to work on this, so I expect progress will be very slow. I hope to include a first basic version of my procedural music generator as a bonus feature in the home edition of Cello Fortress, whenever that's actually finished.

Let's conclude with a question: do you know any communities for open source or free VSTs? There are plenty of VSTs that can be downloaded for free, but in many cases they just specify that they're free to use, not free to distribute with commercial software. I've so far had a hard time finding usable instruments or getting into contact with (hobby) instrument programmers who might be willing to allow me to include their instruments. Leave a comment if you've got suggestions or are a digital instrument programmer yourself.

Opinions needed: which art style for Cello Fortress?

The gameplay of my live performance game Cello Fortress is nearly finished, but the graphics are still basic blocks with some crude textures. Recently I've been working on figuring out what style I want for the game, and I've ended up with two possible styles that I like: one dubbed "Surreal Architecture" and the other "Rough Polygons". Since I have a hard time choosing, I figured I'd just ask here. Which style should I use for Cello Fortress?

I've also added the pros and cons of each style as I see them. Several are just my personal taste so let me know in the comments if you disagree.

"Surreal Architecture" style

The basic idea of this style is that you travel through a world of surreal architecture. The surrealism is of the subtle sort: on first sight it might look like normal classical architecture, but if you stop and think for a second you realise this cannot be: buildings are too big and repetitive, it makes no sense to have a city with nothing but church towers, and why is there an area with giant organ pipes?

This style is inspired by the graphic novel series Les Cités Obscures (also known as Cities Of The Fantastic or De Duistere Steden) by Schuiten and Peters. To get an idea of their style, have a look at these beautiful images. To a lesser degree it is also inspired by the Rork series by Andreas (picture).

The main reason to do this style is that I love classical architecture. I haven't done a lot of 3D modelling recently and this kind of stuff is one of my favourite things to model so I'd love to get back to that. The biggest problems with this style are that the link with Cello Fortress isn't very clear, and that it might distract from what the game is all about: the live cello. So the main reason to do this style is that I'd enjoy making it so much.

As concept art I've made some basic environments in 3dsmax and even rendered a video as if it were the game:

Click for high resolution

Click for high resolution

Previz video of the Surreal Architecture style.

Pros

• More unique and notable
• Lots of fun to make
• Very close to me as a person

Cons

• Lots of work
• Distracts from game's core (the live cello)
• Gameplay less readable for players
• Link between visuals and gameplay less clear
• Needs serious optimisation for good framerate
• Doesn't fit style of existing vehicle and turret art very well
• Not entirely happy with this concept art yet

"Rough Polygons" style

This style is a lot simpler. It is mostly about big, untextured polygons. It has a rough rook and most of the potential beauty of this style comes from the lighting. Using a combination of pre-baked lightmaps and some simple tricks I think I can achieve a global illumination-look as in this image.

A big benefit of this style is that is does not distract from the gameplay. Cello Fortress is such a weird concept that I often need to explain quite a lot to the audience and this style won't distract from the core. It also helps that the gameplay will remain very readable under poor lighting conditions: I often perform with Cello Fortress at festivals and it is not rare to have a beamer projection on a rough wall with too much environment light. Such conditions make the game look a lot less bright and crisp than a normal gaming environment.

Click for high resolution

Pros

• Very readable and understandable for players
• Looks good with crappy beamers on bumpy walls
• Faster to make
• Higher framerate
• Fits style of existing vehicle and turret art

Cons

• Less original / notable
• Not as much fun to make
• Strong shadows on the floor might be too dark but are crucial for this look
  

Why am I not sure which style to choose? The Surreal Architecture style is much more fun to make and would stand out more, but it just doesn't seem like the best choice for the game. It has little to do with the rest of the concept and would distract from what it's all about: the live cello performance.

Which do you think I should make, and why?

Cello Fortress micro vlog #1: Big Chord Attack

I've added a new cello attack to Cello Fortress! Check this little video for a demo and explanation. :)

The downsides of gameplay variety

An important goal in the game design of Swords & Soldiers, Awesomenauts and Swords & Soldiers II is gameplay variety. We try to make all the classes, units and factions play and feel as differently is possible. This makes it more fun to try them all and allows for much more interesting and varied tactics than if everything feels the same. It seems quite obvious that a game gets better if more diversity is added, as long as you have the budget to do so and it does not take away from the core vision. However, there are some serious downsides that are not as obvious. Adding more diversity can hurt many different aspects of a game, as I will explain in this blogpost.

When striving for variety it is important to be aware of the trade-offs. It does not mean that you should avoid diverse mechanics, far from it: it is a key element of the games we make at Ronimo! But consider the consequences before blindly adding variety.

A great example of such trade-offs can be found in the movement mechanics of Awesomenauts. We try to make every character feel different, not only their combat skills, but also their basic movement. Some characters can jump, others can fly, float, make little hops in the air or double jump. There is also variation in walking speed, acceleration and sliding time. Not all of the 21 characters we have so far are completely unique in their movement, but there certainly is a lot of variety. As the interplay between the different classes in Awesomenauts is a key part of the game, this diversity is really important to us.



So where is the trade-off? The problem is that there is such a thing as the ultimate jump. If a jump has the right duration, height and speed, it just feels excellent. Make it a little bit faster or slower and it is instantly less glorious. The jumps in the old Mario games are a great example of this. They strike a perfect balance and it feels awesome just to hop around.

This is where variety bumps its beautiful head. If every jump is to feel different, they cannot all hit that sweet spot. They can feel good in their own way, but it is practically impossible to make them all feel equally super. My colleague Fabian Akker, currently lead game designer on Awesomenauts, recently voiced this very clearly during an internal meeting: "Yes I can make this character faster and yes he will play better then, but I can't make them all be Mario because they will all feel the same." Adding variety will often make the individual parts feel slightly less good than possible, because there is only one "best thing in the game" and not everything can be that thing.

Another example where movement diversity can hurt a game is in graphics. In early prototypes of Awesomenauts we had a skill to walk on ceilings and another to walk on walls. For various reasons both were removed, and one of those reasons was how limiting this would have been to the graphics. To communicate clearly where the player can walk, floors need to be mostly straight and clear. If characters can also walk on ceilings and walls, they would have to be straight and clear too. That would have been a huge limitation to our art style! In this case adding diversity to the gameplay would have removed diversity from the graphics because of the restrictions imposed by the necessities of walkable surfaces.



Diversity is also a problem for balance. The easiest way to make a game perfectly balanced is to have but a single character. Boring, but definitely 'balanced'. As soon as you add more characters, balance quickly becomes infinitely complex and often practically impossible to achieve perfectly. In a previous blogpost I discussed how there are many sides to balance. Even if you manage making all the characters equally strong for pro players, they might still be imbalanced for beginners, in specific team compositions, or in some other way.

A great example of this can be found in map design. It is fun to have a bunch of maps that all play differently. But even the smallest difference can cause problems. If one map is larger than the rest, slow characters will be at a disadvantage. You might try to solve this by giving slow heroes buffs suited to large maps, but that probably introduces other problems. The simple truth: the more diversity, the more impossible it becomes to achieve perfect balance.

And yet balance is so important! How to handle this then? In Awesomenauts we had the problem that one map (AI Station 404) had a single lane at the centre, bunching up everyone. This gave characters with strong area-of-effect attacks (like Raelynn) too much of a benefit. We ended up modifying this map (into AI Station 205) and removing the old version from ranked matchmaking. The downside to this solution is that we lose some diversity in map layouts.



This is just one solution. Another would have been to show the players which map they were getting, so they could avoid certain characters when they feel they are underpowered on a specific map. This would allow us to keep the maps diverse, but at the cost of character diversity, since each map would probably see a more limited set of characters being used. In an ideal world one would find balance tweaks that influence only specific characters on specific maps. However, balance is so complex that such solutions do not always exist, or cannot be found.

Variety can also make a game too complex. I recently finished Advanced Warfare, my very first Call of Duty experience. At the beginning I was immediately overwhelmed by the number of guns, grenades, dashes and options. This is likely due to the series' yearly updates, which force them to add something every time to make it fresh again. At the end of the campaign I still couldn't remember the buttons for half the things I could do. This increase in possibilities adds burdensome complexity, requiring more explanations and tutorials.



We also see this in Awesomenauts: with every character we add it becomes more difficult for beginners to get into the game. In every match they encounter another new Naut they don't know how to fight yet. Having more characters adds longevity and depth, but might make the experience during the first few hours worse for some players.

As a final example I would like to discuss my own hobby project Cello Fortress. Cello Fortress is a hybrid between game and live performance, which means that players only play it for ten minutes each during an event (have a look at this trailer to see how it works). In such a short playsession I can hardly explain anything, so the game needs to be extremely simple. Sometimes players come to me afterwards and ask why I didn't add more diverse weapon. The answer is really simple: because there already is so much I need to explain and communicate during those ten minutes. Variety often adds complexity and there is a limit to how much a player can stomach in such a short session.



I love gameplay variety. My favourite game of 2014 is Far Cry 4, exactly because there is such a big open world where I can do all of these different things. Variety is also super important in our own games Awesomenauts and Swords & Soldiers. But it is important to keep in mind that adding diversity almost always comes with a cost somewhere, so you shouldn't be reckless with it. Add variety, but do it deliberately and take into account the trade-offs it always brings.

Special thanks to Roderick, founder of Leeuwenhart Publishing, for helping me improve my English writing skills. Roderick is the writer of the young adult book Pindakaas & Sushi (in Dutch) and did some freelance writing for Awesomenauts.

Area based depth of field blur

While working on the visual style for my weird live performance game Cello Fortress I came up with a new technique for depth of field blur: area based depth of field blur. As far as I know this is not an existing technique so today I would like to explain how it works.

The visual style for Cello Fortress is far from finished at the moment, but I decided early on that I wanted strong depth of field blur to play a major role. I had already implemented depth of field blur for Proun (which by the way is coming to 3DS, iOS and Android soon!) and I copied that to Cello Fortress. Proun does not have blur on the foreground, so I added that and tried it in-game. The result turned out to not work at all, as you can see in this image (be sure to click the image for a larger version, since this is difficult to see in these small blog-sized images):


click for larger image

The problem is that with standard depth of field blur, only one specific distance to the camera is sharp. In Cello Fortress the camera looks down on the battlefield diagonally, which means that that specific distance looks like a circle. Even weirder is that the part of the screen that is closest to the camera is almost at the centre of the screen: that is the spot the camera hangs exactly above. The result is that both the edges of the screen and the centre are blurred. Of course it is totally undesired that the centre of the gameplay would be blurry.

A simple solution I then tried was to broaden the area that is sharp:


click for larger image


click for larger image

The goal was to have strong depth of field blur as a core part of the visual style for Cello Fortress. So far we either have depth of field blur that interferes with the gameplay, or no depth of field blur at all in the foreground. We need something better:



Once I drew where I wanted blur I realised that this is a simple shape: in world space this is just a simple axis-aligned box. So I implemented a shader that calculates the strength of the depth of field blur based on its distance to that 3D box, instead of distance to the camera. The result is exactly what I was looking for:


Click for larger image. Note that the effect is exaggerated in the smaller screenshot above, the full-res version has the normal, slightly less extreme blur.

A big benefit of this technique is that tweaking it is really straightforward, and that it is independent on the camera. I can easily set the sharp area from code based on the gameplay situation. It also gives me precise control over the height at which the blur starts.

Technically area based blur is quite easy to do. The depth of field blur shader already uses a render-texture that contains the depth of each pixel to the camera. I don't use this depth for anything else, so I can put any value I like there. The pixel shader can easily be adjusted to calculate depth in a different way.



This concept can also be applied to other shapes than boxes. For example, you can also have a sphere within which everything is sharp, while everything outside that sphere is blurred. One can even have several such spheres. How about a visual style where everything is blurred except for the areas around the characters? I think some other interesting visual styles can be made this way, especially in combination with very strong blur.

I have seen some games that use non-distance based depth of field blur, like the beautiful Below, which seems to simply blur the top and bottom of the screen. I am not aware of games that use a 3D area as I use it. Let me know if there are other games that already do this.

The fun of writing your own shaders is that you can bend them to do whatever you want, including weird and unrealistic effects like doing my depth of field blur based on a 3D box area. Feel free to use this idea in your own games, and I'd love to hear from you if try it out!

A simple trick for fast high quality bokeh on lights in Cello Fortress

Bokeh is an effect that pops up in a lot of new games. It is one of those effects that makes a game instantly feel a lot more "next-gen". It is also an effect that usually eats up a lot of performance. For Cello Fortress I came up with a simplified version of bokeh that looks really high quality, is very fast to render and even very easy to implement. My implementation also has some severe limitations, but I think it can work really well for many games.

Bokeh is a part of focal blur, which is also known as depth of field blur or DoF. DoF is the effect that only objects at a certain distance are sharp, while everything closer and further away is blurred. This is an effect that every lens has, and the larger the lens is, the stronger the blur is. Focal blur has been done in many games for quite a while now, but with the normal DoF rendering techniques the bokeh effect is usually lost. Bokeh means that extremely bright spots grow with the blur and appear as clearly recognisable circles (or hexagons or other shapes, depending on the shape of the lens).


Image from HDWallSource.com.

The problem with rendering bokeh in games is that a naive implementation requires high dynamic range (HDR) rendering and an extremely large amount of blur samples. HDR has become quite common in games by now, but taking enough samples to get good sharp bokeh is impractical. For bokeh as large as in the image above, you would probably need many hundreds of samples to get it look smooth. I actually tried that exact thing in Proun two years ago, as a fun little experiment.

Proun already had really high quality DoF (as I described in this blogpost). I added bokeh by simply making some pixels extremely bright. This is a quick dirty trick that was needed because Proun only has limited fake HDR, but the effect looks pretty convincing, as you can see below on the left image. However, if the blur is increased the bokeh becomes extremely noisy, as you can see on the right. It looks noisy despite that it already uses a quite insane 128 samples per pixel! You can find more images of this approach in this blogpost.



This makes this naive approach unusable if strong blur is wanted. We need something smarter. The most common approach in current games seems to be to render actual polygons with a bokeh circle texture on them. To do so we need to find all the bright pixels in the image and then generate a quad for each one.

According to this article The Witcher 2 was ahead of its time by having bokeh all the way back in 2011. The Witcher 2 did this by generating a quad for every pixel and then discarding the ones that are not blurred enough. That's a whole lot of sprites to render! Needless to see this only worked in real-time on the very fastest PC videocards.

Most games that have bokeh these days seem to use a smarter approach, using newer shader models: first they look for all the pixels that are so bright that they would get a clear bokeh circle, and then they generate quads for this. An example of this approach can be found here. Unlike The Witcher 2's technique this creates only the bokeh itself, not the general focal blur, so with this technique the depth of field blur needs to be done separately.

Even with this clever technique the performance hit is still heavy. It requires analysing all the pixels on the screen to find the ones that are much brighter than the ones next to it. It also produces temporal artefacts: if the source of the bokeh is really small it will flicker because it is smaller than a pixel. Normally this is not that much of a problem because it is only a pixel, but if a big bokeh sprite flickers on and off this becomes much more noticeable. These temporal artefacts might already show with slight camera movements. I don't know what technique Star Citizen uses, but the kind of flickering this would result in can clearly be seen in this video in the bright spots in the background.

Now that we roughly know how bokeh is usually rendered, let's look at that simple and fast trick that I use in Cello Fortress. My idea comes from that the most pronounced bokeh is often from actual lights, not just random bright pixels. If you are looking directly at a light, but the light is in the blurred background, then it will generate a very clear bokeh circle. In general a game engine already knows where all the lights are, so this means that we can skip the searching for bright pixels entirely and directly create one screen-space quad for every light. To do so I first render the scene, then apply normal depth of field blur, and finally render the bokeh sprites on top of that.

Not only does this method skip the expensive step of finding the bright pixels that need bokeh, it also fixes all the issues with temporal artefacts. We always know exactly where the lights are so no matter how small they are, the bokeh will never flicker when the camera moves. It just moves along with the light correctly.



To get a good-looking effect I scale the bokeh with how blurred the light should be. This can simply be calculated based on the focal settings of the camera and the distance to the camera. I also fade out the bokeh sprite a bit as it gets larger, since the larger the blur, the less bright the bokeh circle should be (unless the light is infinitely bright). Here is a video that shows the bokeh in action in Cello Fortress. The bokeh is mostly visible at the bottom of the screen.



An important part of bokeh is the actual shape of the bokeh effect. This shape is created by the shape of the lens. I often like hexagonal bokeh best, but I recently discovered that in photography this is generally considered ugly. When reading reviews of a new camera lens I wanted to buy I learned that the more expensive lenses have circular bokeh while cheaper lenses have hexagons. Still, in the end the only thing that matters is what looks good aesthetically in your game. Since most bokeh rendering techniques use those textured sprites the bokeh shape can be modified really simply by using a different texture.



Note that this all does not look as good as it can yet because I have so far spent too little time on the actual visual design of Cello Fortress. I mostly focussed on the gameplay and some cool shaders. Once I start working on proper graphics I should also tweak the brightness, size and colour of the bokeh to make it look better. I should probably also try adding a bit of chromatic aberration to the bokeh texture then.

My bokeh technique does not solve occlusion at the moment. If a light disappears behind an object then it should not still get a bokeh sprite. I didn't solve this yet because it does not occur in the current version of Cello Fortress. However, several solutions can be implemented easily. For example, I already have a depth texture for the depth of field blur, so I can do a look-up in that on the screen position of the light to see whether there is an object in front of it.

The big downside of my method for rendering bokeh is that you need to know where the bokeh will appear beforehand. This means that more subtle bokeh sources like reflections and strong speculars are not practically doable. I can imagine some tricks to get for example bokeh in planar reflections working (if you know where the reflecting planes are), but beyond that it quickly becomes infeasible. The standard technique of searching for the bokeh pixels of the image handles this much better, so if you really need bokeh on speculars, then you will probably need to resort to such techniques.


Image from this article by MJP

I looked up a bunch of articles on bokeh implementations and couldn't find any mention of something similar to what I am doing. This surprises me because the idea is so simple and obvious. If anyone knows of any articles or games that already do this, then please let me know so that I can add a link.

That's it! My bokeh technique is simpler and much faster than most commonly used methods of rendering bokeh and it even fixes problems with temporal artefacts. It is also a limited technique that does not handle all bokeh effects, but in many cases it will probably be good enough. It definitely is for Cello Fortress!

Eliminating ghost notes in Cello Fortress

In my previous blogpost (which is already quite a while ago, sorry for that: I have been really busy on the graphics for Cello Fortress), I explained how my algorithm works for detecting what the cello plays in Cello Fortress. The big missing part there, was that the algorithm I use not only finds the notes that are being played, but also tons of notes that are not being played. Most of those erroneous notes can easily be detected and removed, so let's have a look at how to find and eliminate them.



Octaves

The first problem is octaves. Because of the way my algorithm works, for every note that is found, its octaves are usually also found. Octaves are notes at exactly double the frequency, and double of that, etc. For example, A2 is 110hz, so the octaves that are also found are A3 (220hz), A4 (440hz), A5 (880hz), etc. Finding octaves is inherent to how my algorithm works: I detect notes for which both the own frequency and the overtones (multiples) are strong. The octaves are all exactly at the overtones.

The solution to this is rather simple: if an octave is found, I just throw it away. If I detect A2 and A3, I just throw away A3. This is simple, but unfortunately also a limitation: on the cello it is possible to actually play an octave chord and I cannot detect that this way. However, the octave is a very boring chord, since it is essentially the same note twice, so I have chosen to simply accept that octaves are treated as a single note in my algorithm. Fixing this would require switching to a completely different algorithm altogether, which I do not think is worth it.

Limiting the range

The second improvement is to remove all notes outside the playing range of my cello. For some reason the very low and very high frequencies seem to contain quite a lot of noise, causing notes to be detected where there are none. In my case specifically there is a very low frequency with a strong constant noise in it; I suppose this might be a fan interrupting the computer's internal audio cable magnetically or something like that. The solution is again simple: a cello never goes beneath C2 (65hz), so I simply ignore everything beneath that.

Removing high frequencies is a bit more problematic, since there is no clear limitation to how high a cello can go. However, there is a limitation to me: I can smoothly play up to A4 (440hz), which is one octave above the open A string of the cello. I can go beyond that, but since the quality of my playing quickly decreases there, it is a good idea to not use that for public improvised performances. So despite that I could in theory play higher, I can safely ignore everything above A4. This does pose a problem when other cellists play: I recently met another cellist whom I let play Cello Fortress. It turned out he was much more skilled than me at playing in the higher ranges, so the game limited him there.



So in practice, I only detect notes between 65hz and 440hz. Everything outside that is simply ignored. This is the main reason why Cello Fortress does not work with other instruments at the moment: they have a different range. However, this can easily be remedied by changing some settings. I recently tried controlling Cello Fortress with a violin (roughly G3 to E6, or 196hz to 1319hz), and after tweaking the range it turned out this worked perfectly right away. This is good news for ever doing a "Multi-Instrument Fortress"! (If anyone can come up with a better name for that, please post it in the comments...)

Large chords

Another trick to remove non-existing notes, is to remove notes that are too far above the lowest note being played. Since only strings next to each other can be played on a cello, not all chords are easily possible. In particular, doing really large chords is difficult. I can use this knowledge to remove notes that are too far above the highest detected note. For example, if I play a chord with an open C-string (C2) and also play a very high note on the G-string next to it (anything above F#4), then the high note can safely be thrown away. Again, this does limit certain things that a really good cellist could do, but it fits my own skills nicely and reduces the amount of false notes detected.

Resonating notes

Another category of notes to remove is those caused by resonance. If I play one note really loudly, often some other notes will be audible very softly. This is mostly due to other strings resonating with the main note, or due to scratching because of the loud playing. The pattern here is that such notes are strong enough to be detected, but much softer than the main note. That means I can simply remove notes that are too weak in comparison to the loudest one. This might seem like a limitation, since in practice, playing a chord where one note is very loud and another is very soft is possible. However, this isn't all that doable or useful anyway, so nothing relevant is lost here.

Fifths

The final trick is a lot more subtle than the ones described so far. It is in fifths. Say for example I play an A3 (220hz). This will have overtones at 440hz, 660hz, 880hz, 1100hz, 1320hz, etc. Now the interesting thing if that the fifth to this note, the E4 at 330hz, shares a bunch of these overtones. |The overtones of E4 are 660hz, 990hz, 1320hz, etc. As you can see, two of those overtones are also overtones of A3. In fact, all odd overtones of E4 are also in A3. The result is that if A3 is strong, the total strength of the overtones of E4 is also quite strong! This might seem okay, since there is no base note at 330hz, but it turns out that a little bit of noise or scratching can already cause E4 to be erroneously detected as a note.



I saw this happen quite a lot, and it turns out there is a really simple solution to this problem: if a note has really strong odd overtones, but really weak even overtones, then apparently it is not really being played. So I simply filter based on that to make sure that, in the case of our example, I only detect the E4 when it is actually being played.

The result of combining all of the tricks above is not quite perfect. Some ghost notes are still detected. However, few enough remain that at this point the algorithm is now actually very usable to control a real game. I am still looking into other tricks to remove even more ghost-notes from my detection algorithm, but even if I don't find any further improvements, the results are good enough.

Most of the tricks described in this blogpost have in common that they limit what the cellist can do. However, these limitations are rather subtle and plenty of possibilities remain, so I am quite happy with the balance I have struck here.

Detecting notes from a live cello: the core technology of Cello Fortress

My new game Cello Fortress is controlled by a cello. This is a really weird and unique thing, and comes with some serious challenges. So far I have discussed the game design aspect of this, but at the very core of the game lies a much more technical topic: detecting notes in real-time from a live cello. Cello Fortress really knows what notes I am playing. I developed my own algorithm for that, and although it is not perfect (it quite often shortly detects notes that are not actually played), it works surprisingly well for such a difficult technical problem. So how does it work? Let's have a look!



The big challenge here is that a cello produces a very complex sound pattern. There are all kinds of overtones, scratches and noises in it, and detecting the actual note from that is incredibly difficult. I did some research before I began programming the game, and it turns out that finding notes in a live acoustic instrument is in fact an unsolved problem. There is quite a lot of research in this field, but a perfect solution has not been found yet.

Of course, for a digital instrument like a keyboard this is easy, but as soon as it gets acoustic, it becomes problematic. For guitar there are some devices on the market that can find notes in sound: they can convert live guitar playing to MIDI with very little delay. However, they are supposedly still far from perfect, and don't work with cellos. A cello produces a much more complex tone than a guitar, since it lacks a clear strumming moment: notes just continue and smoothly flow into each other. People also often mention Melodyne to me, but as far as I know, Melodyne can only detect notes after recording, not in real-time.

Seeing that no easy existing solution was around, I decided to start experimenting with this problem myself. I'll have to admit that I didn't read all of the scientific literature on the subject, but it turned out I didn't have to: I managed to come up with an algorithm that detects notes well enough for what I need for Cello Fortress.



So, my basic input here is just a microphone signal. My first step is to grab the last 0.2 seconds of microphone input (9600 samples) and take the Fourier Transform of that. The Fourier Transform is a rather standard mathematical operation that results in a frequency spectrum: a list of how strong each frequencu is within the signal. Since a cello has a very rich sound, a lot of frequencies are in a single note, but the Fourier Transform is still a great starting point for my algorithm.

The math behind Fourier Transforms can look pretty creepy and complex, but the nice thing is that they are usually calculated using the standard Fast Fourier Transform algorithm, which can simply be grabbed from the internet. It is also included in many audio libraries, for example in the awesome FMOD. So to use the Fourier Transform, there is no need to understand the details of how it is calculated.

The spectrum below is from playing an G2 on my cello. Remarkably, there is a clear pattern in the spectrum for this note: there are peaks at evenly spaced frequencies. These are called overtones. I knew about those before I started working on this algorithm, but I had never realized they look so simple. I know that the G I played here is 98hz (that is the open G-string on a cello). An indeed, the left-most peak is at exactly 98hz. Other peaks are at multiples of that: at 196hz, 294hz, 392hz, etc.



This spectrum looks so simple that it seems very easy to write an algorithm that detects notes. However, that turns out to be quite a bit more complex than this. For starters, I would really like to be able to detect chords, because I want to make gameplay with that. Another reason to want to detect chords, is because when I play from one note to the next, the previous note will still be audible due to reverb within the big body of the cello, and due to the previous string still vibrating. Being able to detect several notes at once will greatly help me handle the transition from one note to another. However, chords make the spectrum a whole lot more complex, as you can see in this example:



Regardless of how complex this looks, looking at this, a relatively simple algorithm comes to mind that is indeed the core idea of what I ended up with: let's list all the peaks, and then look for the peaks that would explain all other peaks through their overtones. For example, let's say I have peaks at 45hz, 90hz, 100hz, 135hz, 180hz and 200hz. Those are all multiples of 45hz and 100hz, so those must be the frequencies that are being played. In note names, those would be F#1 and G2.

This simple algorithm quickly turns into a big mess, though. It is difficult to make it handle peaks due to scratches and noise well, so I quickly ended up in a swamp of extra rules and magic numbers to compensate for those.

There is also a more fundamental problem: the spectrum is not very precise. For every block of about 3hz, the Fourier Transform tells me how strong it is. So I know the strength of 100hz, of 103hz, of 106hz, etc. Since a block is 3hz wide, the actual frequency for the bin at 103hz can be anything between 101.5hz and 104.5hz. This lack of precision is not a big problem for the higher notes: the difference between A3 (the open A-string) and the next note (A#3) is 13hz, which is well above 3hz. For low notes this is a bigger problem: the difference between C2 (the lowest note on a cello) and C#2 is only 3.9hz. This is so close to our 3hz precision, that this is bound to give problems.



One solution to this would be to somehow increase the precision of the Fourier Transform. The only way to do this, is to feed it more samples. I currently feed it 0.2 seconds (9600 samples), so I could double the precision if I would feed it 0.4 seconds (19200 samples). However, this has two big downsides. The first is that this increases the delay between playing a note and detecting it. The second downside, and this is much worse, is that many notes are way shorter than that. If I play really fast, I can play 6 notes per second, which means that a period of 0.4s contains several notes. This completely clutters the spectrum and makes it much more difficult to detect individual notes. For these reasons I really don't want to increase the precision of the Fourier Transform by taking a longer sample period.

This lack of precision makes the previous idea of finding the peaks that explain all other peaks not work well. In the above example, I assumed a peak at 45hz. However, because of the size of the bin, the real frequency is anywhere between 43.5hz and 46,5hz. This makes a huge difference for the frequency of the higher overtones. The 10th overtone of 43.5hz is at 435hz, which is five bins away from 450hz (the 10th overtone of 45hz). I tried to modify my peaks for this, but it quickly got stuck in more and more exception-cases.

The solution I came up with is a different approach based on the same idea. I step over all frequencies with steps of 0.5hz. For each frequency, I look up its own strength, and the strengths of the first 20 overtones. So for 43.5hz, I look up the strength at 43.5hz, and the strengths of the overtones at 87hz, 130.5hz, 174hz, etc.

Taking such small steps means I look in the same bin several times, since steps are only 0.5hz and bins are 3hz wide. But in the 20th overtone, those steps of 0.5hz correspond to 10hz, which is a difference of several bins. So doing more steps is actually relevant.

The final part of my algorithm is simple: a frequency must be a note if its own strength is strong enough, and if the added strengths of all its overtones is strong enough. These two minimum strengths are settings that depend mostly on the microphone's output volume. Through experimentation I have found good minimum strengths for both the own frequency and the overtones. The algorithm now looks roughly like this:

for (float frequency = 60; frequency ‹ 450; frequency += 0.5f) { int baseBin = getBinFromFrequency(frequency); float baseStrength = spectrum[baseBin]; float overtonesStrength = 0; for (int multiplier = 2; multiplier ‹= 20; multiplier++) { int bin = getBinFromFrequency(frequency * multiplier); overtonesStrength += spectrum[bin]; } if (baseStrength >= baseStrengthMinimum && overtonesStrength >= overtonesMinimum) { print("Found a note! Frequency:", frequency); } }

Note that this algorithm is bound to find the same tone at several frequencies: if 70hz is well above the thresholds, than 70.5hz and 69.5hz probably also are, even though they are the same note. So when several frequencies are close to each other, I simply take the strongest and throw the others away.

Despite that my bins are 3hz wide, the extra information from the overtones makes it so precise that I can use this to reliably tune the strings of my cello very precisely. Beforehand, I had never expected I would find an algorithm so simple, and yet giving such exact answers!

Does this mean I am done, that I found the golden solution? No, far from it... The result of this algorithm is indeed that with the right sensitivity settings, I can detect practically all notes I play on my cello. However, what we have now is still completely unusable, because it also detects tons of false-positives: notes that are not really being played. Next week, I will explain why this happens, and what tricks I added to solve most of this problem. See you then!

How the cello controls the game in Cello Fortress

The most unique aspect of Cello Fortress is how a cellist does a live performance in front of an audience, while at the same time controlling a game. This is completely different from other music games, in which the musician usually plays on a fake plastic instrument, and even if he plays a real instrument, he does nothing but imitate an existing song. In most such other music games, there is hardly any real gameplay: just points based on how well you played the song.

Cello Fortress is a completely different affair: here the cellist is controlling a real game, with real choice and interaction. Depending on what his opponents do, the cellist plays different notes. The cellist can even do things like baiting the opponents with a certain attack and then switching to another.

So how does that work? What does the cellist need to do to trigger the various attacks? Check this trailer to see (and hear!) how it works:


Live video footage in the trailer shot by Zoomin.tv Games at the Indie Games Concert.

Here is an overview of the attacks as explained in the trailer:

• Slow high notes: long range guns
• Slow chords: homing missiles
• Fast high notes: machine guns*
• Fast chords: double machine guns*
• Dissonant chords: flamethrowers
• Slow low notes: create mines
• Fast low notes: mines move towards the player
• Special melody 1: obliterate left half of screen
• Special melody 2: obliterate right half of screen

*Playing even faster notes increases the speed of the machine guns.

The key thing to realise, is that the first seven of these attacks allow the cellist to play many different styles, melodies and rhythms, and still achieve that attack. The number of possibilities with "slow high notes" is literally infinite. This is a crucial aspect to the game, since it allows the cellist to improvise in many different ways, keeping each match of Cello Fortress fresh and varied. Having so much freedom also allows an experienced cellist to play fluently from one attack to the other.

There is real gameplay and choice in this. For example, something I often do when playing the cello in Cello Fortress, is play something slow to dare players to get close to my cannons. As soon as they do, I switch to fast chords to damage them from short range.

The special melodies are each 8 notes and have been defined beforehand. The fun in these is that the attack is announced when the 4th note is played, but the damage is not actually done until the 8th note is played. Players who pay close attention can hear the attack coming after only two notes, and thus flee before it even happens.

I can play the melody faster or slower to make the attack happen earlier or later. From a gameplay perspective, one would assume I always attack as quickly as possible, but my goal is actually not purely to win: I want to entertain the players and the audience. So I sometimes deliberately let them live to give them a more fun experience. This can be seen around 1:33 in the trailer: I make the final note very long to allow that player to escape. Just like in a film, the best moments are not when the hero dies, but when he narrowly escapes.



These controls were specifically chosen because they combine music and control in a natural way. Achieving this was more difficult than it may seem. In my very first prototype, the cello simply shot one bullet for every note, and the direction of the bullet depended on the pitch of the note. This turned out to play horribly: whenever the players moved from the left to the right, the cellist had to play a scale from low to high. When they moved back, the notes also had to go back from high to low. This made it completely impossible to play anything that sounded like good music.

Another thing I tweaked a lot is the mapping of which pattern triggers which attack. The current controls work quite well on an emotional level: the attack is linked to the feeling of the music. Slow, low notes often sound quite tense and sad on a cello (especially with the specific types of melodies I personally usually play), and alternating between slow and fast notes creates an awesomely menacing atmosphere. This can be seen in the trailer from around 1:00. Creating tension this way works incredibly well: I performed with Cello Fortress in front of an audience of several hundred people at the Indie Games Concert, and the noises from the audience made it clear that they experienced the tension very strongly.

A note I should make on this trailer, is that in the real game, there is a slight delay between the music the cello plays and the moment the guns react to it. This is because analysing music in real-time takes a bit of time. To make the trailer more understandable, I have moved the sound a bit to make the music fit the gameplay exactly.

While I am already performing with it, I am also still working on Cello Fortress to improve it. So what is next? My focus for the coming period is first creating real graphics, and after that I want to add a couple more attacks for the cellist. In the meanwhile, I hope more events, venues and exhibits will contact me to perform with Cello Fortress! Check www.cellofortress.com for tour dates and contact info!

Style exploration for Cello Fortress

So far I have focussed all my efforts for Cello Fortress on the gameplay. Controlling a game with a live cello is a big challenge, both technically and in terms of game design. Now that that is turning out well, the next step is to get rid of the prototype art and give the game real graphics. I have been thinking a lot about the exact visual style and the goal is of course to make Cello Fortress look really good. While the current version of Cello Fortress does not contain any of these visual ideas yet, today I'd like to show my inspirations, and some visual experiments that I did.

The base inspirational image for Cello Fortress' visuals comes from the special edition booklet to Radiohead's album Amnesiac. It is a really noisy and small image, and the scan makes it even worse, but the core of it is this: extremely low-polygon mountains with extremely low-res textures, so that there are big square pixels on the flat triangles.



Taking this idea, I did a quick experiment in 3D Studio MAX. I just piled some pyramid-like shapes in the scene and put on some textures, without any intend on making a real composition or anything finished. I didn't want it to actually be as rough as the Radiohead image, so I added lighting and depth of field blur to it.



The other big influence comes from an indie game called Teleglitch. In Teleglitch, whenever you shoot the colours on the screen are ripped apart. This gives a very raw and powerful effect, so I wanted something similar. Skip to 58s in this video to see the effect in action:


Gameplay footage from Teleglitch, check the effect after 58s

Of course, I wouldn't want to just copy that, so I came up with something different that was inspired by this effect: bullets and explosions deform the landscape in 3D with a rough noise. I added this effect to my little 3D Studio MAX testscene and made this little previz animation that shows the effect in action, much to my satisfaction:


Cello Fortress pre-visualisation

At this point I actually thought I had copied the effect from another Radiohead video, but I couldn't find it anymore. In my memory, Yorke's head deformed in exactly the same way whenever he held his finger close to his head. After a lot of searching I discovered the video was called "Go To Sleep", but to my surprise the effect is very different here. Instead of deforming, they animate the number of polygons in his head, which looks quite different, as you can see here. (Note that in 3D Studio MAX, this can be done very easily by animating the Optimise modifier.)

So my inspiration turned out to be completely different from what I had remembered, but my own result looks good, so I am happy. ^_^ This kind of process is something I like a lot: if I add my own interpretation to something that inspired me, it becomes something new. This also happened with Proun: it was strongly inspired by Kandinsky and Rietveld, but by adding my own ideas, it became instantly recognisable as something else, something of my own.

At this point, I concluded rawness was my goal. I found another great example of the vibe I wanted in a trailer for Luftrausers, a game by our friends and neighbours from Vlambeer. I really like the extreme vibe they managed to achieve:


Luftrausers by Vlambeer, coming soon to PS3, Vita and PC!

From here, it seems quite simple to conclude what Cello Fortress' visual style should be: raw, pixels, big polygons, extreme effects, noise, screen shakes, grey. However, more recently I discovered some other images that I find extremely inspiring, but that contradict that. They also work with the big-polygons-big-pixels style, but are much less raw. Just have a look at these beauties:







These images from the fantastic Geo A Day and JR Schmidt combine a lot more colour and smoothness with the sharpness of the polygons, and I'd love to make something like that. However, they are quite at odds with the roughness I was previously aiming for.

The final element that has been inspiring me for Cello Fortress, and that I think will combine very well with the images above, is the tilt shift effect that is seen in the new Sim City. It is also found in tons of random videos on Youtube, like in "The Sandpit", which, if you have never seen it before, you need to watch right now. Tilt shift is basically just an extreme depth of field blur that makes everything look small. I also think it looks absolutely gorgeous.


Tilt shift in the new Sim City (which, by the way, is a great game, and much better than the 6.5 Metacritic currently has it at because of those DRM shenanigans...)

Having all these elements, I am not sure yet how I shall mix them. I expect some kind of combination of all of these ingredients, but I am really curious how raw Cello Fortress will end up exactly. Knowing myself, there is a good chance it will turn out a lot more smooth, like my Solid Motion series.

Regardless of what it ends up being exactly, I hope it ends up both unique and awesome! Next week I will release a new gameplay trailer for Cello Fortress, still with the old prototype graphics, and after that I'll get cracking on these graphics!

Why Cello Fortress is a twin stick shooter

Cello Fortress could have been any kind of game. The core concept is nothing more than: "a game in which a live cellist controls the game by playing cello, and plays with or against the audience". This idea can be applied to any genre. The cello could control a brawler, a puzzler, a strategy game, a racing game, with some imagination maybe even a point and click adventure. So why did I specifically make a twin stick shooter? A lot of thinking and brainstorming went into this choice, so today I would like to explain that a bit.

Doing something with improvisation on my cello and my computer is a topic I have been thinking about for years, but it wasn't until a year or two ago that it dawned on me that the cello could actually be a game controller. Before that, I was mostly thinking about writing a procedural music generator that could accompany my own cello improvisations. Quite a big step from a game, but it slowly evolved into one from there nevertheless.



Knowing that the game needed to be about a live performance with my cello, with or against the audience, creates a number of requirements for the game design. These requirements fuelled what would become the actual game, and it was quite a challenge to find something that really met them all well enough.

1. Game is fun to play, and also fun to watch for the audience.
2. Playable by as many people in the audience at the same time as possible, while also being playable by only a single player.
3. Such simple controls that no long explanations or tutorials are needed, and that even non-gamers can play.
4. The cellist has room to improvise and play something that sounds good, while still controlling the game in a meaningful way.
5. The influence of the cello is direct enough that players and audience can quickly recognise and understand it.
6. Bonus requirement: the core game itself is original.

Each of these requirements brings a different view on what this game should be. Number 2 for example favours games with a zoomed out view, so that several players have room to move within the same screen. Splitscreen is also an option for this, but didn't seem such a good fit for readability in case of a larger audience where people might be standing further away. Specifically inspiring games for me where Gatling Gears by our Dutch friends at Vanguard, and the WiiWare racer Driift Mania. Both of these games are also excellent fits for requirement number 3, since they require very few sticks and buttons to control, which makes the gameplay easy enough to explain in just a couple of seconds.



Another game genre that came to mind was rhythm games. Dance mats are really nice controllers for festival-like situations, since they are so physical. However, this kind of game seemed at odds with requirement number 1: most rhythm games don't have an on-screen hero that onlookers can follow and root for.

Requirements numbers 4 and 5 are where the real complexity of the game design steps in. How does a cello control a game while still making music? For example, going left by playing high notes and going right by playing low notes is way too boring. I needed something more subtle. At the same time, the influence of the cello should be clear to the audience as quickly as possible, to make sure people don't think it is a 'normal' game with only a live soundtrack. The cello actually controls the game and ideally people would understand this without my explanation.

This makes the racing game a difficult proposition. I had in mind that the cello would generate the track. But to give players a chance at reacting to it, the track needs to build up a bit in front of the furthest player. This has the big downside that it happens just beyond where most people are looking.

I also struggled with how to build the track from the music. I thought about things like making difficult, spiky roads if the cello plays in minor, while generating more smoothly curving roads when playing in major. However, this is way too subtle. Most people probably can't even recognise the difference between major and minor well enough, let alone link it to the gameplay. In my mind I tinkered with lots of other ways in which the cello could control the game, but I didn't come up with anything that worked as well and as naturally as the current twin stick shooter.



I arrived at the twin stick shooter as a good genre for Cello Fortress pretty early in the process, yet for a long time I kept brainstorming and looking further. This was because of requirement number 6. Twin stick shooters are a pretty overused genre in games, and it is difficult to still do something interesting with them in terms of gameplay. So I preferred something more original and kept searching.

However, in the end a quote from my former teacher JP van Seventer reminded me that keeping the core game less original might actually be a good thing. JP is a Wise Person (tm) and is also Ronimo's regular outside advisor. He currently works at the Dutch Game Garden to give advice to young Dutch game start-ups. He once said something along these lines:

"Innovating everything at the same time is not a good idea, because it alienates the audience too much. It is better to innovate on a number of aspects of the game, and keep the rest recognisable. That way players can relate to it much better and understand how it works more quickly."

This has been very influential on my thinking about games. Before this I had dreams of coming up with a game that would be totally unique in every possible way, and this quote made me realise that that might often not be a good idea. This quote definitely applies to Cello Fortress.

With Cello Fortress having been in the media quite a bit in the past week, I see now how difficult it is to explain what Cello Fortress is. Even after the very explanatory trailer that I posted last week, I read lots of confused comments online from people who don't really get how it works. So I am happy that I chose a genre in which the core gameplay itself at least is really easy to explain, so that I can focus my communications on the much more interesting side of the game: the way the cello controls it and the way the game is halfway between a game and a live performance.

Cello Fortress trailer revealed!

A couple of months ago I revealed my new project Cello Fortress. Now it is finally time for a proper trailer! Cello Fortress is a unique combination of a live cello concert and a game, and is intended to be played at events (festivals and such). This trailer shows how the game works, and shows a bit of the interaction between the players and the cellist.

Cello Fortress is pretty weird and unique, so I guess so far only those who actually played it, really understood what Cello Fortress is about. Hopefully this trailer will clear it up for others as well!


(Video footage recorded by Dyzlo Film at Indigo.)

In essence, Cello Fortress is a twin stick shooter. Four players cooperate using Xbox controllers to destroy as many cannons as possible. However, the cannons are not controlled by the computer, but by a live cellist! He improvises live music on his cello, and tries to do that in such a way that the game not only does what he wants, but also that the music actually sounds good. In a sense, this is the ultimate in adaptive music!

The current version of Cello Fortress is still in beta and far from finished. The graphics are just some quick prototyping models thrown together, and all kinds of things still need tweaking and improving. Cello Fortress is already touring, though, and has so far played at several events in the Netherlands.



Cello Fortress is a complex project in several ways: playing cello so that it sounds good and controls the game is a big challenge and requires an experienced cellist and a lot of practice. Analysing what the cello plays is also technically very complex and has, as far as I know, never been done before in a computer game. I expect I can write a couple of interesting blogposts about sound analysis now...

Cello Fortress may be a music game, but it is nothing like existing music games like Guitar Hero. In Cello Fortress the instrument is a real cello, and real music is played. The cellist is also not scored for playing 'right' or 'wrong' notes. Instead, he controls a shooting game by improvising.

Because the music is so central to the game, I wanted to make sure that the trailer would sound like an actual match of Cello Fortress. In real matches, the music is completely improvised (nothing is composed beforehand), so I decided to also improvise the music for this trailer. I ended up recording over twenty improvisations, and I used the one that I liked most. (The timpani were added digitally afterwards to add a little bit of flavour for the trailer.)

I hope that this trailer will not just get attention in the world of games, but also in the world of music. Cello Fortress could be a revolution in music acts! This makes me quite nervous, though: musicians are bound to hear any mistakes in my playing. Cello is an incredibly difficult instrument to play well. Although it has been a hobby of mine for over twenty years, I can still only hope my playing sounds acceptable to trained ears...



Since this music was improvised specifically for this trailer, it is not entirely the 'real thing'. Luckily, someone recently posted some footage of a match with audible sound (all other video recordings I have contain more audience noises than cello music). The video and sound quality are not super, but nevertheless this gives a good idea of what a Cello Fortress match sounds like:


(Filmed live at the Playful Arts Festival.)

Cello Fortress is a really weird and unique game, but for me it makes a lot of sense: playing cello has been a hobby of mine for ages, and I am a professional game developer. I like to make weird, unique things. How could these ingredients not combine into a game? Coming up with the actual concept for Cello Fortress was more difficult though: cello and computer can be combined in many different ways and it took me years to come up with something that is fun for the audience to play and watch, controllable by a cellist, and allows for beautiful music.

Cello Fortress is a project that I do entirely in my spare time. Nevertheless, I am going to steadily keep improving Cello Fortress, and I hope I can play with it at some exciting events and festivals!

Announcing Cello Fortress: a unique mix of live concert and game

Today I am announcing my new project: Cello Fortress! Cello Fortress is a really weird game concept, very experimental. I am really happy to finally have started working on it, after having it floating around in my head for years. ^_^

Cello Fortress can only be played live at events, and it will debut at the Indigo exhibition later this month!



Cello Fortress combines a live cello concert with a twin-stick-shooter, in which the cellist plays against the audience. It brings a unique experience in which I control the game by improvising on my cello, simultaneously fending off attacks and making music. The audience takes up controllers and tries to beat the fortress. The result is an exciting interplay between cellist, players and audience.



Up to four players use controllers to navigate their tank, using one stick to move and the other to shoot. They dodge bullets and attack the turrets. At the same time, the game analyses the notes played by the cellist, as picked up by a microphone. Aggressive notes activate the burst-cannons, dissonant chords turn on the flame-throwers, and an ominous melody charges a bombardment.



Cello Fortress is only playable at live concerts given by... me! So I am not only the creator of the game, but also the cellist. The debut of Cello Fortress will be at the Indigo 2012 exhibition on 28 and 29 September in Utrecht, Netherlands. The version played at Indigo will be a fully playable, early prototype, to experiment with this weird concept.

I find this project an incredibly interesting experiment. I have played cello for years, including in a band and currently in the Kunstorkest amateur baroque orchestra. I do lots of improvisation at jam sessions and such. But how does it feel to improvise with a gameplay goal? If I need to suddenly play low notes to fire a certain cannon, then I will need to come up with a melody that brings me to the low notes quickly, without sounding like random notes. After all, I am making music here! Besides the players, there is also an audience listening and viewing, so they need to hear good music as well as see interesting gameplay. Quite a challenge, and I look forward to trying it in front of an audience.

This game contains so many weird elements that it is interesting to list some of the rare and unique aspects in Cello Fortress:

• Using a real musical instrument as a game-controller (like Rocksmith, but this time using a cello and to control normal gameplay)
• Combining concert and game
• Dynamic difficulty adjustment directly by the designer during gameplay
• Completely adaptive soundtrack
• The game is a show in front of an audience

Since the game is still in the early stages, the version playable at Indigo will probably not look very polished yet. However, I am already playing around with the visual style of the game, so here is a piece of concept art for it:



So to play Cello Fortress, visit Indigo! Indigo is a yearly exhibition by Dutch Game Garden that shows the best new work by Dutch game developers. This year it is held in Utrecht's town hall, at Korte Minrebroederstraat 2. On September 28 it is open mainly for press and business people, while on September 29 it is open to the general public. Entrance to Indigo is free, and Cello Fortress will be playable regularly throughout the day. Hope to see you there!

A name and a logo