Why I hate having /tmp as a tmpfs

April 5, 2012

There is a recent move in Linux to turn /tmp into a tmpfs. As a sysadmin, I am afraid that I have a visceral dislike of this (and always have).

The core problem with a RAM-backed /tmp is that it creates a new easy way to accidentally DOS your machine. When /tmp is only a disk, it's pretty clear how much space you have left and filling up /tmp is only a minor to moderate inconvenience. When /tmp is backed by RAM, filling up /tmp means driving your machine out of memory (something that Linux generally has an explosive reaction to). Worse, how much /tmp space you really have is unpredictable because it depends on how much RAM you need for other things. In theory this might be predictable, but in practice RAM demands are subject to abrupt and rapid swings as programs start and stop and change what they're doing.

(Even without a bad reaction from the Linux kernel to an OOM, an OOM situation is both worse and more wide-ranging than having /tmp or even the root filesystem run out of space. Being out of memory affects pretty much everything on the machine, and that's assuming you don't have enough swap space to cause your machine to melt down.)

This is bad enough on a single-user machine, where at least you are only blowing your own foot off when you DOS the machine through an accidental OOM because you (through old habits) or your program (through not being revised to the latest nominal standards) innocently put something sufficiently large in /tmp. On shared multi-user machines, it's pretty close to intolerable; the damage done is much larger and so is the chances of it happening, since all you need is one person to have one accident.

(By the way, this is not theoretical. We have had people put multi-gigabyte temporary files in /tmp, especially on our compute servers. Sometimes they manage to fill /tmp up, even though it has many gigabytes of disk space.)

Ultimately, what making /tmp into a tmpfs does in practice is to make the machine more fragile. How much more fragile depends on what happens on the machine, but it's undeniably more fragile. I don't like things that make my machines more fragile, so I don't like this.

By the way I'm aware that other systems (such as Solaris) did this years ago. I didn't like this transition on them either, for exactly this reason. I consider it a really good thing that only staff can log on to our Solaris machines, because a RAM-backed /tmp makes them too fragile for me to be happy with general access to Solaris.

(See also SysAdmin1138.)

Sidebar: the right way to do this transition

It's really simple: make a new /tmpfs mount point that is, well a tmpfs. The latest new standards make it clear that any number of programs need revising anyways to put their data in the right place; while you are revising those programs, you can perfectly well make them use /tmpfs when appropriate. And the result does not blow people's feet off when they continue following decades of user behavior and program defaults. If you want and it makes you feel better, you can then make /tmp into a symlink to /var/tmp.

(As usual, this is certain Linux people not solving the real problem.)


Comments on this page:

From 87.118.135.130 at 2012-04-05 03:05:35:

Exhausting machine's memory can be easily prevented with the "size" mount switch. Here is a sample from /etc/fstab

none /tmp tmpfs size=400m 0 0

//vaLentin

From 131.58.64.193 at 2012-04-05 06:42:04:

Same with Solaris: "swap - /tmp tmpfs - yes size=16384m". If there wasn't a limit, any user or run away program could DOS the server.

I would be happier if there was some sort of Hierarchical File System where "extra" RAM could be used for /tmp speed benefits up to a limit, after which it went to disk.

From 70.31.29.25 at 2012-04-05 07:25:02:

With regards to the OOM, I wish that the default would be not to overcommit memory. I believe this is the default in both Solaris and FreeBSD, and it seems to work just fine there, so I'm not why the Linux folks decided to allow it.

I've taken to adding the following on many of our production Linux machines:

 # Only allow allocating of virtual memory that we actually have (i.e, RAM + swap)
 vm.overcommit_memory = 2
 vm.overcommit_ratio = 100

I used to run some Perforce servers where this was a problem for a while (before we could upgrade the hardware to have more RAM).

By cks at 2012-04-05 10:49:42:

The problem with defaulting to strict no-overcommit is that it is overly conservative in real life in many situations, which means that it results in artificially denying programs memory when there is and will be lots of RAM free. I wrote more about our experience with this in NonStrictOvercommitDefault (and see also this on the general issues).

The problem with quota settings for tmpfs is that either they must be very small or they're only probabilistic protection; they will probably or maybe or perhaps keep you safe if people use all of /tmp, but you can't be sure and so your machine may still explode every so often. The more of your memory you allow /tmp to use, the less safe you are.

(The other problem with quota settings is whether they will be the default in stock configured systems. I suspect that they will not be.)

By trs80 at 2012-04-05 23:44:42:

Debian is also trying this, but the maintainer is being shouted down on debian-user. Hopefully sanity will prevail.

From 94.216.51.44 at 2012-04-06 08:19:21:

I am using tmpfs only on machines with more than 4 GB of memory. IIRC the default tmpfs restricts itself to half the physical memory if nothing is specified, that should be a safe default in these cases.

By cks at 2012-04-06 21:13:41:

Half of memory is not a safe default for 'large' memory machines unless most people don't ever use most of their RAM. That is certainly not the case here.

From 94.173.191.19 at 2012-04-07 00:48:21:

I can certainly agree with that: the first time I even realized that Arch uses a tmpfs now was when my netbook ground to a halt because of an AUR package build - both of the 2 most popular AUR-supporting pacman wrappers (yaourt/packer) do their builds under /tmp. On a netbook which is already running something else, just one big package can bring the whole thing crashing down. Especially because the final stage of a makepkg (the core tool underlying Arch's package system) run involves compressing the package. Obviously it has to do this before the files in /tmp have been removed, so you get three instances of the package's data in RAM; the raw data in a tmpfs, the data mid-compression in RAM proper, and the resulting compressed data being written out, also to the tmpfs. And so if the netbook is, for example, also running a browser with a moderate number of tabs, it doesn't take an especially large package to fill up enough RAM to be a problem.

I'm sure there are plenty of good reasons for a tmpfs on /tmp (such as those outlined in the first post you link), but until tools catch up with the new intended usage (even a good number built specifically for known-tmpfs systems don't currently do it right), it's extremely problematic; as you might expect of an Arch user, I'm not averse to digging into the system to figure out why things are breaking, but if such a problem happened in a newbie-focused distro such as Ubuntu, would the average target user have the faintest idea what to do?

From 94.173.191.19 at 2012-04-07 00:49:57:

Correction: Third link, not first. Seems my brain discarded any knowledge of the first two shortly after reading them.

Written on 05 April 2012.
« More on equality in Python (well, mostly Python 2.7)
Why we haven't taken to DTrace »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Apr 5 01:09:48 2012
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.