== Why your Ubuntu server stalls a while on boot if networking has problems

Yesterday I wrote on [[how to shoot yourself in the foot by making
a mistake in _/etc/network/interfaces_ UbuntuInterfaceBootProblem]].
I kept digging into this today, and so now I can tell you why this
happens and what you can do about it. The simple answer is that it
comes from _/etc/init/failsafe.conf_.

What _failsafe.conf_ is trying to do is kind of hard to explain
without a background in Upstart (Ubuntu's 'traditional' init system).
A real System V init system is always in a 'runlevel', and this
drives what it does (eg it determines which _/etc/rcN.d_ directory
to process). Upstart sort of half abandons runlevels; they are not
built into Upstart itself and some _/etc/init_ jobs don't use them,
but [[there's a standard Upstart event to set the runlevel
http://manpages.ubuntu.com/manpages/wily/man7/runlevel.7.html]] and
many _/etc/init_ jobs are started and stopped based on this runlevel
event.
Let's simplify that: Upstart's runlevel stuff is a way of avoiding
specifying real dependencies for _/etc/init_ jobs and handling them
for _/etc/rcN.d_ scripts. Instead jobs can just say '_start on
runlevel [2345]_' and get started once the system has finished its
basic boot processing, whatever that is and whatever it takes.

Since the Upstart runlevel is not built in, something must generate
an appropriate 'runlevel N' event during boot at an appropriate
time.  That thing is _/etc/init/rc-sysinit.conf_, which in turn
must be careful to run only at some appropriate point in Upstart's
boot process, once this basic boot processing is done. When is basic
boot processing done? Well, the _rc-sysinit.conf_ answer is 'when
filesystems are there and static networking is up', by in Upstart
terms means when [[the _filesystem(7)_
http://manpages.ubuntu.com/manpages/wily/man7/filesystem.7.html]]
and _static-network-up_ [[upstart events
http://manpages.ubuntu.com/manpages/wily/man7/upstart-events.7.html]]
are emitted by something.

So what happens if networking doesn't come fully up, for instance
if your _/etc/network/interfaces_ has a mistake in it? If Upstart
left things as they were, your system would just hang in early boot;
_rc-sysinit.conf_ would be left waiting for an Upstart event that
would never happen. This is what _failsafe.conf_ is there for. It
waits a while for networking to come up, and if that doesn't happen
it emits a special Upstart event that tells _rc-sysinit.conf_ to
go on anyways.

In the abstract this is a sensible idea. In the concrete, _failsafe.conf_
has a number of problems:

* the timeout is hardcoded, which means that it's guaranteed to
  be too long for some people and probably not long enough for
  others.

* it doesn't produce any useful messages when it has to delay,
  and if you're not using [[Plymouth https://wiki.ubuntu.com/Plymouth]]
  it's totally silent. Servers typically don't run Plymouth.

* Upstart as a whole has a very inflexible view of what 'static
  networking is up' means. It apparently requires that every 'auto'
  interface listed in _/etc/network/interfaces_ both exist and have
  link signal (have a cable plugged in and be connected to something);
  see eg 
  [[this bug
  https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/1094593]]
  and [[this bug
  https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/964775]].
  You don't get to say 'proceed even without link signal' or 'this
  interface is optional' or the like.

For Ubuntu versions that use Upstart, you can fix this by changing
_/etc/init/failsafe.conf_ to shorten the timeouts and print out
actual messages (anything you output with eg _echo_ will wind up
on the console). We're in the process of doing this locally; I
opted to print out a rather verbose message for [[my usual reasons
../sysadmin/ContextInErrorMessages]].

Of course, all of this is going to be inapplicable in the upcoming
Ubuntu 16.04, since Ubuntu switched from Upstart to systemd as of
15.04 ([[cf https://wiki.ubuntu.com/SystemdForUpstartUsers]]).
However Ubuntu has put [[something similar to _failsafe.conf_
https://launchpad.net/ubuntu/+source/ifupdown/0.7.48.1ubuntu6]]
into their systemd setup and thus I expect that we'll wind up making
similar modifications to it in some way.

(A true native systemd setup has a [[completely different
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/]] and
generally more granular way of handling failures to bring up
networking, but I don't expect Ubuntu to make that big of a change
any time soon.)