Wandering Thoughts archives

The good and bad of the System V init system

The good of System V init is that it gave us several big improvements over what came before in V7 and BSD Unix. First and largest, it modularized the boot process; instead of a monolithic shell script (or two, if you counted /etc/rc.local) you had a collection of little ones, one for each separate service. This alone is a massive win and enabled all sorts of things that we take for granted today (for example, casually stopping or starting a service).

The other big change is that System V init turned the entire work of init from a collection of hacks into a systematic and generalized thing. It formally defined runlevels and runlevel transitions and created in /etc/inittab a general mechanism for specifying all of the work init did, from booting to running gettys on serial lines (or running anything) to how to reboot the system. System V init removed the magic and hardcoding in favour of transparency. Things like reboot stopped killing processes and making special system calls and turned into 'tell init to go into runlevel ...', and then /etc/inittab and runlevel transitions said what to do so that this actually rebooted the machine. In the process it added a way to specify how services shut down.

(Simply defining runlevels formally meant that other systems could now tell what state the system was in and behave differently between eg single user mode and multiuser mode.)

The very general and high level view of the bad of the System V init system is that fundamentally all it does is blindly run shell scripts (and that only when the runlevel changes). This creates all sorts of lower-level consequences:

• SysV init doesn't know what services are even theoretically running right now, much less which ones of them might have failed since they were started.

• It doesn't know what processes are associated with what services. Even individual init scripts don't know this reliably, especially for modern multi-process services.

• Even init scripts themselves can't be certain what the state of their service is. They must resort to ad hoc approaches like PID files, flag files for 'did someone run <script> start at some time this boot', checking process listings, and so on. These can misfire.

• Services are restarted in a different environment than how they are started on boot. Often contamination leaks in to a restarted service (in the form of stray environment variables and other things).

• Output from services being started is not logged or captured in any systematic way. Many init scripts simply throw it away and there's certainly no official proper place to put it.

• The ordering of service starts is entirely linear, by explicit specification and guarantee. System V init explicitly says 'I start things in the following order'. There is no parallelism.

• Services are only started and stopped when the runlevel changes. There is no support for starting services on demand, on events, or when their prerequisites become ready (or stopping them when a prerequisite is being shut down).

• System V init has no idea of dependencies and thus no way for services to declare 'if X is restarted I need to be restarted too' or 'don't start me until X declares itself ready'.

• There is no provision for restarting services on failure. Technically you can give your service a direct /etc/inittab entry (if it doesn't background itself) but then you move it outside of what people consider 'the init system' and lose everything associated with a regular init script.

• Since init scripts are shell scripts, they're essentially impossible for programs to analyse to determine various things about them.

• It's both hard and system-dependent to write a completely correct init script (and many init scripts are mostly boilerplate). As a result it's common for init scripts to not be completely correct.

• Init scripts are not lightweight things in general, either in reading them to understand them or in executing them to do things.

In theory you can try to fix many of these issues by adding workarounds in your standard init script functionality. Your 'standard' init script utilities would capture all daemon output in a documented place and way, start everything in cgroups (on Linux) or containers to track processes reliably, have support for restarting services on failure, carefully scrub every last bit of the environment on restarts, monitor things even after start, et cetera et cetera, and then you would insist that absolutely every init script use your utilities and only your utilities. In practice nothing like this has ever worked in practice (people always show up with init scripts that have bugs, take shortcuts, or do not even try to use your complex 'standard' init utilities) and the result would not particularly be a 'System V init system' except in a fairly loose sense.

(It would also make each init script do even more work and run even more slowly than they do now.)

Init's (historical) roles

Historically (by which I mean since at least V6 Unix), init aka PID 1 has had three and then four roles:

1. It inherits orphan processes, ie processes that have had their regular parent exit. Doing this almost certainly simplified a bunch of V7 kernel code because it meant that every process has a parent process.

2. Starting up the user level of Unix on boot. Originally this was done by running a monolithic shell script, as can still be sort of seen in OpenBSD. System V init modularized and generalized it into the multi-file form.

3. Starting, managing, and restarting the getty processes for the console and (other) serial lines. System V init generalized this so that init started and restarted whatever you told it to via entries in /etc/inittab.

4. Shutting down the user level of the system and rebooting. This role first appeared in System V init, using the modularity that it had introduced for booting. Modern BSDs also give init responsibility for rebooting (and it will run a shell script as part of this), but as late as 4.4 BSD reboot(8) did almost all of the work itself and there was no concept of running a shell script to bring services down in any orderly way; reboot(8) just killed everything in sight.

(Really. You can read the 4.4 BSD reboot(8) source if you want, it's not long. The violence starts at the 'kill(-1, SIGTERM)'.)

Putting the three (and then four) roles on the shoulders of a single process is likely due to both conservation of resources in early Unixes (given that they ran in very limited environments they likely didn't want to take up memory with extra programs) and simple least complexity and effort. Once you had init as the inheritor of orphan processes you might as well make it do all the other roles since it was already there. Why throw in additional programs without a good need? It probably helped that even in V7 the other two roles were pretty simple and minimal, per eg the V7 /etc/rc.

As a historical note, it was BSD Unix that decided that init was so crucial that the system should be rebooted if it ever exited. V7 Unix will probably get into an odd state if init ever exits but as far as I can tell from the kernel source PID 1 is not treated specially as far as exiting goes; as a practical matter V7 Unix just assumes it will never happen. Even what happens if /etc/init can't be executed on boot is not strictly a kernel thing in V7.

(In the initial environment of BSD, this decision was probably doubly correct. Even if you never have to deal with any orphaned processes or the kernel cleaned them up itself (let's wave our hands aggressively here), losing init means that getty processes will not be restarted on serial lines when people log out, which over time makes it impossible for anyone to log in. Of course in the modern era of networked machines this is no longer such an issue and you probably care a lot more about sshd than about gettys.)

Some modern init systems have split some or most of these roles out from PID 1. Solaris, for example, moved everything except the first role to separate processes (the SMF stuff runs in svc.startd et al and getty processes are handled through ttymon and sac).