How we install Ubuntu machines here

December 7, 2014

We have a standard install system for our Ubuntu machines (which are the majority of machines that we build). I wouldn't call it an automated install system (in the past I've used the term 'scripted'), but it is mostly automated with only a relatively modest amount of human intervention. The choice of partially automated installs may seem odd to people, but in our case it meets our needs and is easy to work with.

Our install process runs in three stages. First we have a customized Ubuntu server install image that is set up with a preseed file and a few other things on the CD image. The preseed file pre-selects a basic package set, answers a bunch of the installer questions for static things like time zones, sets a standard initial root password, and drops some files in /root on the installed system, most importantly a postinstall script.

After the system is installed and reboots, we log into it (generally over the network) and run the pre-dropped postinstall script. This grinds through a bunch of standard setup things (including making sure that the machine's time is synchronized to our NTP servers) but its most important job is bootstrapping the system far enough that it can do NFS mounts in our NFS mount authentication environment. Among other things this requires setting up the machine's canonical SSH host keys, which involves a manual authentication step to fetch them and thus demands a human there to type the access password. After getting the system to a state where it can do NFS mounts, it mounts our central administrative filesystem.

The third step is a general postinstall script that lives on this central administrative filesystem. This script asks a few questions about how the machine will be set up and what sort of package set it should have, then grinds through all the work of installing many packages (some of them from non-default repositories), setting up various local configuration files from the master versions, and applying various other customizations. After this process finishes, a standard login or compute server is basically done and in general the machine is fully enrolled in various automated management systems for things like password propagate and NFS mount management.

(Machines that aren't standard login or compute servers generally then need some additional steps following our documented build procedures.)

In general our approach here has been to standardize and script everything that is either easy to do, that's very tricky to do by hand, or that's something we do a lot. We haven't tried to go all the way to almost fully automated installs, partly because it seems too much work for the reward given the modest amount of (re)installs we do and partly because there's some steps in this process that intrinsically require human involvement. Our current system works very well; we can spin up standard new systems roughly as fast as the disks can unpack packages and with minimal human involvement, and the whole system is easy to develop and manage.

Also, let me be blunt about one reason I prefer the human in the loop approach here: unfortunately Debian and Ubuntu packages have a really annoying habit of pausing to give you quizzes every so often. These quizzes basically function as land mines if you're trying to do a fully automated install, because you can never be sure if you've pre-answered all of them and if force-ignoring one is going to blow up in your face. Having a human in the loop to say 'augh no I need to pick THIS answer' is a lot safer.

(I can't say that humans are better at picking up on problems if something comes up, because the Ubuntu package install process spews out so much text that it's basically impossible to read. In practice we all tune out while it flows by.)


Comments on this page:

From 62.195.238.236 at 2014-12-07 02:59:40:

that is why lots of people use FAI (http://fai-project.org/) instead of preseeding. Debconf is a pain.

By Albert at 2014-12-07 08:36:16:

Do you use some sort of automated configuration systems like puppet, ansible and friends?

By cks at 2014-12-08 10:16:02:

The short answer is that we don't use much automated configuration management, for various reasons. If we did, such a system would presumably be able to do a lot of this setup process for us as part of synchronizing a (new) system to the state it should be in.

Written on 07 December 2014.
« Browser addons can effectively create a new browser
Why I don't believe in generic TLS terminator programs »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Dec 7 01:10:43 2014
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.