Today, I had the distinct pleasure of discovering that /var/run must exist on the root filesystem, even if you have a separate /var filesystem. If your root filesystem does not have such a hidden /var/run, you experience mysterious failures of various boot stuff, including an inability to bring up the network; for bonus points, nothing gives you any meaningful error messages.

(For bonus points, the default Ubuntu server startup sequence wipes out the console scroll buffer, so you can't scroll back to see many boot time messages anyways. And nothing captures them elsewhere.)

I find this incredibly obnoxious, because it means that if you move your root filesystem around, you must move it with something that peeks under mount points (effectively only dump or an equivalent will do) and you must not, on any account, move it to a place with a replacement /var already mounted in place. (Guess what we did, not knowing any better.)

If you started out without a separate /var filesystem and now want to move to one, apparently your life just sucks.

There are some comments in /etc/init.d/mountvirtfs that suggest that it should be recreating the root filesystem's /var/run if it doesn't exist. However, there are two problems:

• there is no actual code in mountvirtfs to do this, just comments saying that it should be done.
• trying to do it wouldn't help anyways, because the root filesystem is mounted read-only at this point.

(While /var/lock also exists on the root filesystem and is necessary, don't worry about it; LVM will helpfully create it for you in early startup as a side effect of making its /var/lock/lvm directory. So you only have to reboot twice to have everything working right with that.)