Authored by Bilby

Compression and Caching

What they are, why they are important, and how we use them

Compression and caching are both very different methods of speeding something to do with data up, they often go hand in hand and are used literally all the time! Seriously, both of these methods would have been used several times for us to send this page to you, and for you to be able to see it on your screen right now!

A brief description

You can skip this if you know what they both are, we’ll have a meatier conversation in a second!


There are two types of compression, lossy and lossless. These get thrown around a lot, especially in the media industry since it affects image and audio file quality. We are using compression on raw data for the purposes of this article, so I won’t touch on lossy.

Lossless compression is taking data and modifying it so that it retains it’s original information, but uses less space. Compression algorithms such as Zip look for patters in the data, and swap them out for a flag that tells the uncompressing tool what it needs to do to recreate that pattern.
This is a very simplified example, but think of a data set like this:

{ black, red, green, black, red, green, black, red, green, black, red, green, black, red, green, black, red, green }

you could “compress” this like so:

{ (black, red, green) x 6 }

And viola, the file size has shrunk without loosing any information!


Caching is a method of saying “I already have this, why would I need it again?”, your browser caches images and other page elements more than you might think, ever wondered why a page seems to load faster on the second refresh? It’s because the browser has taken a bunch of content and stored it, so that when you refresh, you won’t need to collect all of that data again.

How do we use these at Knoob?

Darude and I can be very particular when it comes to code efficiency, we don’t like things that will run “ok”, we like things to be as fast as possible within reason. This is why we do everything in our power to ensure you guys get higher fps (not in this article, but it’s something we do optimise as much as possible), and faster load times!

When it comes to caching, you would most likely see it in use on our FastDL server when you need to download a map or addon from us. The compression FastDL uses (bzip2) can more than halve file downloads! That means you download a ma twice as fast, and we only need to send half as much! It’s a win win!
If you are ever thinking about using FastDL for a SRCDS server, bzipping things is an absolute must, and if your using Linux to host your web service, I even have a tool for you to use!

We use caching for things like steam data on our servers, the way this works is our server has a database, and every time we find a user we haven’t seen before, we log their steam ID and a few other things that can be attained from a steam API call to make sure that when you load something like the stats page, we aren’t making a huge number of requests to Steams servers.

To keep this data updated is a challenge, how do we know when to refresh it? We can’t simply see if a user updated their avatar or profile name on Steam without using the expensive call to get that information in the first place, so what do we do?
In this case, I am using the server loadscreens to grab that info every time you log into a Knoob server, that means that the active players get their profiles updated, whereas we don’t waste time on the guys that only play once.

This is an ideal situation, since the loadscreen on any server that shows you your name or avatar will have to make this API call anyway, us included, so we also chuck a copy of it in our database while we’re at it!

This is just a quick update on some of the stuff I’ve been doing behind the scenes, I hope that you gain something from it!
Darude has been doing alot of work on the Gmod side of things too, so we should be able to keep bringing out the cool stuff for you guys to enjoy.

– Bilby