Saturday, 6 September 2014

The importance of packet count

When learning about online multiplayer programming I always read that it is important to keep the bandwidth usage low. There are tons of articles that discuss bandwidth optimisations and limitations. However, while developing Awesomenauts we learned that packet count can in some cases be equally important. Somehow I have rarely seen this mentioned in articles or books, so I figured it was about time to write a blogpost to tell the world: packet count is also important!

When we started development of Awesomenauts I though that packet count was only relevant because of the size of the packet headers. Every UDP packet has a UDP header (8 bytes) and an IP header (at least 20 bytes). This means that no matter how little data you send per packet, it always gets an added 28 bytes. This makes packet count relevent for bandwidth: if you send 200 packets per second, then you are sending 5600 bytes per second only in headers.

I thought this is where the importance of packet count ends. If I can somehow optimise my game to send only 30 bytes per packet, then it is okay to send 200 packets per second because the total bandwidth will still only be 200 * (20+8+30) = 11600 bytes per second, which is fine for a modern game.

It turns out that this is not true on the real internet. During development of Awesomenauts we found out that high packet count by itself is a serious problem. Quite a few internet connections that would happily send as much as 40 packets per second of 1200 bytes each (totalling 48kB/s) become problematic when they need to send 200 packets per second of 50 bytes each (totalling only 10kB/s).



In our experience sending more than 100 packets per second is problematic for some connections, resulting in packet loss, lag spikes and ultimately losing the connection altogether. We recently decreased the average send rate in a full Awesomenauts match from 150 packets per second to 75 and this seems to have decreased the number of connection errors by around 25%. In that same patch we also decreased the average bandwidth by 50%, so I cannot say for sure whether decreasing just the packet count would have had the same effect. However, based on earlier experiments with this I think the packet count decrease was more important than the bandwidth decrease. Our impression is that packet counts above 100 per second are a problem while below 100 packets per second it is not very relevant to optimise further

The internet in general is a weird topic because it behaves so unpredictably on some connections. For example, we have seen that some routers always bunch our packets: they constantly arrive two at a time even though there is 33ms between sending.

Note that there is more to packet headers and overhead than the UDP and IP headers I mentioned above. For example, I recently learned that when on a DSL line an extra DSL header is added. This is all hidden from the game code, but it means that having lots of packets can on some connections also mean using more bandwidth than you realise.

So there you have it: be mindful of your packet counts! Sending lots of small packets is not a good idea and should be avoided if possible. This is especially relevant for peer-to-peer games, since everyone in the game talks to everyone else and packet counts thus rise quickly.