Recently we had a little incident with our Amanda backup system, where the Amanda daemon on both the backup server and one particular backup client got into a state where they kept running and, more importantly, kept writing out plaintive debugging log messages. We discovered this when first the Amanda server and then an important Amanda client entirely filled up their root filesystems with what was, by that time, a several hundred gigabyte debug log file, which they each wrote into their /var/log/amanda directory tree. Afterward, we wanted to limit the size of Amanda debugging logs so that they couldn't fill up the root filesystem any more, especially on Amanda clients (which our normal servers, especially our fileservers).

All of our root filesystems are ext4, which supports quotas for users, groups, and "projects", as sort of covered in the ext4 manual page. In theory we could have added a size limit on /var/log/amanda with project quotas. In practice this would have required updating the root filesystem's mount options in order to get it to take effect (and that means editing /etc/fstab too), plus we have no experience with ext4 quotas in general and especially with project quotas. Instead we realized that there was a simpler solution.

(We can't use user quotas on the user that Amanda runs as because Amanda also has to write and update various things outside of /var/log/amanda. We don't want those to be damaged if /var/log/amanda gets too big.)

The easiest way to get a small, size limited filesystem on Linux is with a tmpfs mount. Of course the contents of a tmpfs mount are ephemeral, but we almost never look at Amanda's debug logs and so we decided that it was okay to lose the past few days of them on a reboot or other event (Amanda defaults to only keeping four days of them). Better yet, with systemd you can add a tmpfs mount with a systemd unit and some systemd commands, without having to modify /etc/fstab in any way. Some quick checking showed that our /var/log/amanda directories were all normally quite small, with the largest ones being 25 Mbytes or so, so the extra memory needed for a tmpfs for them is fine.

Without comments, the resulting systemd var-log-amanda.mount file is:

[Unit]
Description=Temporary Amanda directory /var/log/amanda
# I am not 100% sure about this. It's copied from other
# tmpfs mount units.
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target
After=swap.target

[Mount]
What=tmpfs
Where=/var/log/amanda
Type=tmpfs
Options=size=128m,nr_inodes=100000,\
    nosuid,nodev,strictatime,\
    mode=0770,uid=34,gid=34

[Install]
RequiredBy=local-fs.target

(The UID and GID are those of the standard fixed Ubuntu 'backup' user and group. Possibly we can specify these by name instead; I haven't experimented to see if that's supported by mount and tmpfs. The real Options= line isn't split across multiple lines this way; I did it here to not break web page layout.)

In theory it would be better to use zram for this, since Amanda's debug logs are all text and should compress nicely. In practice, setting up a zram device and a filesystem on it and getting it all mounted has more moving parts than a tmpfs mount, which can be done as a single .mount systemd unit.

If we wanted persistence, another option could be a loopback device that used an appropriately sized file on the root filesystem as its backing store. I suspect that the actual mounting can be set up in a single systemd mount unit with appropriate options (since mount has options for setting up the loop device for you given the backing file).