How much swap space we're using across our servers (in October 2022)
One of the reactions I've seen to us moving away from swap partitions (to swap files) on our Linux servers is that some people run without any swap space at all. Through a chain of thought I wound up wondering how much swap space we're actually using (as opposed to how much we have configured). Fortunately, our metrics system makes it relatively straightforward to answer questions like that, or at least to come up with some numbers.
Because our servers have different amounts of swap configured, I'm going to look at both how much swap space has been left free and how much swap space has been used. The simpler number is the amount of remaining (free) swap space. Over the past 30 days, three of our compute servers used all of their swap space, our primary login server ran down to only 13.5 MBytes free, and our test virtualization server got as low as 214 Mbytes free. Everything else always had at least 512 Mbytes free. A potentially more interesting number is the average amount of free swap space over the last 30 days, which will factor out short term spikes in swap space usage. Here, nothing had less than 550 Mbytes of swap free, even the compute servers. Looking at the standard deviation of free swap over time suggests that many of our servers don't vary much in their swap usage.
(Our servers only rarely get rebooted, so these numbers are pretty much the long term state for many of these machines.)
Looking at the maximum amount of swap space used over the past 30 days, only four servers used 1 GB or more of swap space. Only eight used more than 512 MB, and only eleven used 128 MB or more. Since we typically configure at least 1 GB of swap space, this suggests that we're seriously overconfiguring most machines. In terms of average swap space used, only one server averaged over 512 MB and only five averaged over 128 MB (and one of those was a compute server).
These aren't really the results that I expected when I started looking. Before I looked, I would have expected more swap space usage from highly unused pages of data being slowly aged out in favour of more productive uses of RAM, such as cached files. But apparently modern Linux kernels mostly don't do this.
(Despite what I've found out here, we probably won't reduce our default swap size below 1 GB. 1 GB of disk space isn't much these days and it's a cheap insurance policy.)
PS: The servers in question are mostly Ubuntu 22.04, with a moderate number of 20.04 and 18.04 machines and a few CentOS 7 ones.