== Another obnoxious discovery about Ubuntu's _/var/run_ stuff Today, I had the distinct pleasure of discovering that _/var/run_ must exist *on the root filesystem*, ~~even if you have a separate _/var_ filesystem~~. If your root filesystem does not have such a hidden _/var/run_, you experience mysterious failures of various boot stuff, including an inability to bring up the network; for bonus points, nothing gives you any meaningful error messages. (For bonus points, the default Ubuntu server startup sequence wipes out the console scroll buffer, so you can't scroll back to see many boot time messages anyways. And nothing captures them elsewhere.) I find this ~~incredibly~~ obnoxious, because it means that if you move your root filesystem around, you *must* move it with something that peeks under mount points (effectively only _dump_ or an equivalent will do) and you must not, on any account, move it to a place with a replacement _/var_ already mounted in place. (Guess what we did, not knowing any better.) If you started out without a separate _/var_ filesystem and now want to move to one, apparently your life just sucks. There are some comments in _/etc/init.d/mountvirtfs_ that suggest that it should be recreating the root filesystem's _/var/run_ if it doesn't exist. However, there are two problems: * there is no actual *code* in _mountvirtfs_ to do this, just comments saying that it should be done. * trying to do it wouldn't help anyways, because the root filesystem is mounted read-only at this point. (While _/var/lock_ also exists on the root filesystem and is necessary, don't worry about it; LVM will helpfully create it for you in early startup as a side effect of making its _/var/lock/lvm_ directory. So you only have to reboot twice to have everything working right with that.)