Why I don't want fully automated installs

January 12, 2014

One of the modern system administration practices is fully automated OS installs; practically everyone supports it and some OSes even consider it their primary deployment method. I am a lonely holdout, at least for moderate sized environments such as ours where machine (re)installs are not an everyday activity. I have two reasons for this.

The first reason is a pragmatic one. As far as I can tell, it's still generally a fair amount of work to put together a reliable, fully functional auto-install environment that supports a fairly wide variety of machines and software setups. We may be an extreme case; we have at least five different ways to set up Ubuntu 12.04 servers (not counting hardware variations, including whether the system disk is mirrored) and we install machines across multiple networks. With our relatively low rate of (re)installs I can't convince myself that the time savings from having fully automated installs will actually pay off here.

(Note that we already have partially automated installs and even that takes a fair amount of work to assemble for a given LTS release.)

But the bigger reason is a philosophical one. I don't want fully automated installs because I think that they're too dangerous. I consider it a feature that you have to add boot media to a machine and then interact with it several times before the installer will destroy all data on the machine (or at least a disk or two of it) and set it up on our network with some IP address and some software configuration. The idea of 'if this machine PXE boots it will get (re)installed' is, for me, a nightmare. I very much want the chance to double-check what the install process is doing and to interact with it.

So what I really want is not fully automated installs but more smarts in partially automated installs. For example, it would be nice if we could have some high-level options that we could accept or reject, like adding some local standard partitioning schemes. Some of our machines will deviate from them (and I certainly want to see the disks that the install is about to scribble over), but a lot of the time I'd take the 'standard CSLab partitions' option.

(Our current Ubuntu install process basically asks us only for the machine's IP address information and its partitioning, and it's hard to get away from asking for the IP address. No, we don't want to run DHCP on our server networks.)

PS: I want to say that I also want better ways to build, maintain, and update partially automated installs, but I haven't looked at the state of the art in that for a while.

(To be clear: what I'm talking about here is OS installs, where you go from empty disks on a new server to an installed and perhaps somewhat customized OS on the network. There are lots of fairly good solutions for post-install customization and localization.)


Comments on this page:

Two ideas:

1. I've heard it recommended that a non-default VLAN be set up for imaging, and that's the only place PXE booting is possible. This seemed like a lot of changes to get to the final goal, so I didn't pursue it.

2. Fancier PXE chained boot loaders like iPXE can be set to have non-imaging defaults. My normal setup is a 10 second delay at the menu, then attempt to boot of primary hard disk, then if that fails drop into a memory test.

I can intervene in that 10 seconds to start OS installers or diagnostic tools. No more slinging around old, unpatched physical media anymore.

Also, you can script iPXE - I haven't done this, but custom config scripts could be generated/loaded depending on the booted machine's MAC address or other distinguishing features by feeding that information to a script on a http server.

I can't speak to Ubuntu, but I would avoid almost any manual (interactive) RHEL install in favour of some automation. I've even generated a dummy Kickstart in Cobbler then edited it by hand to strip out all the build server dependencies (for a non-connected build), rather than undertake another tedious, arbitrary trip through the Anaconda GUI. The benefits of having a consistent, standardised, recorded install that runs without intervention across everything you manage considerably outweigh the vagaries of manual installs IMHO, even if it costs extra effort, even if that's per server. (I realise YMMV and that's effectively what you're saying here - I'm just suggesting you don't knock it until you've tried it.)

As for unintended reinstalls, we build everything on a separate build VLAN and then move it to its production network. If we really have to avoid network boots, we can generate a custom ISO image in Cobbler instead. (You can also disable PXE boot for any system defined in Cobbler after use, although I agree this is easy to forget.)

By Bashir at 2014-01-13 14:27:50:
The idea of 'if this machine PXE boots it will get (re)installed' is, for me, a nightmare.

You should check out the Foreman: http://theforeman.org/

Specifically, the 'Build' feature of Foreman, which should let you very easily see which machines have PXE config enabled and which do not.

It also lets you easily enable PXE if needed via the web interface cmd line/script.

As far as I can tell, it's still generally a fair amount of work to put together a reliable, fully functional auto-install environment that supports a fairly wide variety of machines and software setups.

Foreman makes this too easy. There are many users deploying large numbers of Linux and Solaris systems using Foreman. I believe there is some Windows support also.

Written on 12 January 2014.
« Why I am not enthused about Red Hat Enterprise 6
Sadly, we're moving away from Red Hat Enterprise Linux »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Jan 12 03:05:17 2014
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.