The Upstart dependency problem

April 23, 2011

We just ran into another issue with how Upstart handles startup scripts. The simple way to put it is that Upstart glues together the startup script itself and ordering dependency information on when it needs to be run. This is a problem because the startup script is 'owned' by the package but dependencies can be system dependent, which means that local sysadmins need to change them.

(This is similar to the previous Upstart coupling problem.)

Now, the old /etc/init.d system didn't exactly have dependencies, but it at least did have ordering and there was a strong convention that packages left the ordering alone when they installed updates. This let sysadmins manipulate the startup ordering so that the local special dependencies worked out right.

With Upstart, the only way to modify or add ordering dependencies is to modify the actual /etc/init/ script. Even if the package management system then leaves it alone on package updates, you have to hope that the update didn't make any (important) changes to the startup script, changes that you will have to notice and propagate into your own local version. There is no way to change just the dependency information while having the package system manage the rest of things.

You know, I had thought that both the sysadmin world and the Linux world had learned a lesson about shoving unrelated information into the same file. In fact the Linux world has spent years carefully splitting monolithic files into separate, much easier to manage things; this is what gave us such things as /etc/cron.d (and these things are a great idea; it is much easier to manage files in a directory than sections of a file). It's sad to see Upstart taking a step back into the past.

(It is especially annoying because almost all of the /etc/init.d scripts converted to Upstart scripts have no greater dependency than 'start me in runlevel X, Y, and Z'.)

Sidebar: an example of why ordering dependencies are system-dependent

We have a chain of dependencies for a machine getting its NFS mounts up. For reasons beyond the scope of this entry, mounting any NFS filesystem requires that the SSH daemon be running on the client, then our automounter replacement requires a single NFS filesystem be mounted first. So far, so good; we can do all of this with our own custom init scripts with their own ordering without needing to change system ones.

Then we add our system of user-run webservers. The simple way to start user-run webservers on boot is for users to have crontabs with @reboot entries. Cron runs @reboot entries immediately on startup. This implies that on our web server (and in fact on any user-accessible server), cron startup depends on our NFS filesystems being mounted; if cron starts before then, user @reboot actions will fail because the files they're trying to use are on filesystems that haven't been mounted yet. We can't handle this dependency with new init scripts; we have to change cron's own dependency list.


Comments on this page:

From 71.57.69.135 at 2011-04-23 10:33:00:

This is just Ubuntu's poor design. Fedora/RedHat implement the traditional init system using Upstart, so that you can continue to use the normal chkconfig tool to manage what services start at which runlevel, regardless of Upstarts native support for such things.

Elijah

From 70.30.90.144 at 2011-04-23 12:21:56:

You may want to check out the "rcNG" system that a lot of the BSDs use. It is also referred to as the "rc.d" system:

http://www.netbsd.org/docs/guide/en/chap-rc.html
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-rcd.html
http://www.mewburn.net/luke/papers/rc.d.pdf (USENIX 2001 paper)

It was originally created on NetBSD, but FreeBSD and Dragonfly have converted to using it as well:

http://www.freebsd.org/cgi/man.cgi?query=rc
http://www.freebsd.org/cgi/man.cgi?query=rcorder
http://www.freebsd.org/cgi/man.cgi?query=rc.conf

From 70.95.102.101 at 2011-04-23 14:29:56:

Elijah, I don't think from Chris's blog that its the "What run level" problem, it's that everything is now controlled within the specific init script for the service. Under the old init.d approach you could specifically set an order for services to load within a runlevel, within /etc/init.d/rc.(1|2|3|4|5) (location varies based on the distribution, annoyingly, might find them under /etc/rc.d), by altering the number on the symbolic link. Services inside that directory get started in alphanumeric order. Even if the init script got changed you didn't have to worry, the symbolic links were what set the start order.

In Ubuntu there is a command line tool, update-rc.d, which allows you to assign services to levels, so your assertion that you can't use the normal tools for the job aren't true. Debian has been using update-rc.d from nearly the beginning of its existence (and Ubuntu has from day 1 as a consequence). As someone who uses both regularly, I can assure you it's really not all that different from chkconfig.

Under Ubuntu's implementation of upstart the files Chris is talking about are in /etc/init. Taken from the start of mysql.conf file in there on an Ubuntu 10.10 vps:

# MySQL Service
description     "MySQL Server"
author          "Mario Limonciello <superm1@ubuntu.com>"

start on (net-device-up
         and local-filesystems
         and runlevel [2345])
stop on runlevel [016]
....

it then goes on to define the startup process itself. What Chris isn't happy about is that the dependencies, that 'start on' section, are all set in the file inside that directory. The risk is that if the package manager hasn't been told these are protected config files it could just up and replace them without warning, removing any custom dependencies you've got. Other packages like Apache have their .d directories so that 1) the main config file can be updated or replaced by the package manager without disrupting any custom changes, and 2) it's easier to manage by having nicely segmented configuration.

From 205.208.123.236 at 2011-04-23 19:01:06:

"Under the old init.d approach you could specifically set an order for services to load within a runlevel", and you can still do this in RHEL6, because that traditional system is implemented in Upstart. That is, RHEL6 uses Upstart as a direct replacement for init, instead of the hybrid approach Ubuntu uses (where there are upstart-style and deb-init-style scripts). There may be downsides to RHEL's approach, but I prefer it precisely because of Chris' complaint.

Elijah

By gsauthof at 2011-04-26 05:13:44:

Chris, did you have a look at the new systemd project? (it replaces upstart in FC15) I am wondering if you would have similar dependency problems with systemd.

By cks at 2011-04-26 13:01:19:

I did some looking into systemd. While the question is hard to answer from the available systemd manual pages (I looked here), it appears likely that you can glue something together that doesn't require modifying supplied service configuration files.

(A general systemd index and reference is here.)

From 93.34.59.252 at 2011-07-23 05:59:07:

So, how do I start an app server only after mysql has been started? That was easy with the rc.d system.

From 65.110.252.109 at 2012-12-10 14:44:38:

I suspect there are better examples, but here, it looks like the problem isn't that Cron's dependency changed, but that those per-user crontabs are really additional jobs (the webservers) that depend on NFS mounts. Worse, it's not even cron itself, it's the fact that you're hacking Cron to serve as an additional init system.

It looks like you can add per-user Upstart scripts, which makes a lot more sense. (I assume you can do something similar under systemd.)

I think you might be onto something, but I'm not sure this is a good example.

-- Dave
By cks at 2012-12-11 13:27:01:

I don't think that per-user Upstart scripts will help us. There are several things wrong with them in an environment like ours. Off the top of my head:

  • according to the documentation, a user has to run initctl first before their $HOME/.init is read. So how does that happen automatically after the machine is rebooted?

  • $HOME/.init is global, not per-machine. We don't want webserver jobs starting on any machine that the user happens to run initctl on (or vice-versa for boot time jobs on other machines). This gets worse if $HOME/.init is processed automatically.

  • if $HOME/.init was processed automatically and assuming that we could somehow delay processing until after NFS mounts were up, scanning for it is something that very much does not scale. We have over 1500 accounts with NFS-mounted home directories; I do not want any system looking for $HOME/.init in all of them.

Apart from the start ordering, crontab is exactly what we need; it is automatic, per-machine, and requires registration (avoiding mass scans).

Written on 23 April 2011.
« Nailing down new-style classes and types in Python
Notes to myself on the priorities of Linux routing policy rules »

Page tools: View Source, View Normal.
Search:
Login: Password:

Last modified: Sat Apr 23 00:41:38 2011
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.