Wandering Thoughts: Recent Entries For 2011/04/23

The Upstart dependency problem

We just ran into another issue with how Upstart handles startup scripts. The simple way to put it is that Upstart glues together the startup script itself and ordering dependency information on when it needs to be run. This is a problem because the startup script is 'owned' by the package but dependencies can be system dependent, which means that local sysadmins need to change them.

(This is similar to the previous Upstart coupling problem.)

Now, the old /etc/init.d system didn't exactly have dependencies, but it at least did have ordering and there was a strong convention that packages left the ordering alone when they installed updates. This let sysadmins manipulate the startup ordering so that the local special dependencies worked out right.

With Upstart, the only way to modify or add ordering dependencies is to modify the actual /etc/init/ script. Even if the package management system then leaves it alone on package updates, you have to hope that the update didn't make any (important) changes to the startup script, changes that you will have to notice and propagate into your own local version. There is no way to change just the dependency information while having the package system manage the rest of things.

You know, I had thought that both the sysadmin world and the Linux world had learned a lesson about shoving unrelated information into the same file. In fact the Linux world has spent years carefully splitting monolithic files into separate, much easier to manage things; this is what gave us such things as /etc/cron.d (and these things are a great idea; it is much easier to manage files in a directory than sections of a file). It's sad to see Upstart taking a step back into the past.

(It is especially annoying because almost all of the /etc/init.d scripts converted to Upstart scripts have no greater dependency than 'start me in runlevel X, Y, and Z'.)

Sidebar: an example of why ordering dependencies are system-dependent

We have a chain of dependencies for a machine getting its NFS mounts up. For reasons beyond the scope of this entry, mounting any NFS filesystem requires that the SSH daemon be running on the client, then our automounter replacement requires a single NFS filesystem be mounted first. So far, so good; we can do all of this with our own custom init scripts with their own ordering without needing to change system ones.

Then we add our system of user-run webservers. The simple way to start user-run webservers on boot is for users to have crontabs with @reboot entries. Cron runs @reboot entries immediately on startup. This implies that on our web server (and in fact on any user-accessible server), cron startup depends on our NFS filesystems being mounted; if cron starts before then, user @reboot actions will fail because the files they're trying to use are on filesystems that haven't been mounted yet. We can't handle this dependency with new init scripts; we have to change cron's own dependency list.