2022-11-10
The problem of (Unix) swapfiles and server backups
I recently wrote about how we're moving away from swap partitions to swap files on our Linux servers, since swap files work fine (and are what Ubuntu defaults to) and swap partitions are slightly more work to set up. However, a potential drawback of swap files recently occurred to me. Suppose that you're using swap files, and also that you're making backups of the root filesystem of some of your servers (because they're 'pets' instead of cattle, and this way you automatically capture local logs and local state). Then, since your swap file is a file in the root filesystem, your backups include your swap file, all however many GBytes it is.
There are two pieces of good news for us in our particular environment (with Ubuntu 22.04 servers). The first is that in general our swap files are small; an extra GB for the moderate number of servers that we back up won't really break the bank, not when set against the amount of data from our fileservers. The second is that Linux doesn't update the modification time for swap files when they're used, so to a backup system these look like files that never change. Since they never change, they'll only be included in full ('level 0') backups, not in the much more common incremental backups that our backup system does.
(This is in general the saving grace of backing up the entire root filesystem for servers; almost all of it is completely static and so only appears in full backups. Only logs and a few other things show up in incremental backups.)
Still, this isn't ideal. If someday we give a NVMe based server a big swap file (because NVMe is so fast, it might not be as dangerous), we might need (or at least want) to care about this. I suspect that I should investigate how to make Amanda automatically exclude these files.
(This also makes me thankful that we corrected the default Ubuntu 20.04 and 22.04 swap file size of 4 GB down to 1 GB in our installer customizations. We had other reasons, but saving 3 GB on full backups is a nice bonus.)