Why we're not currently interested in PXE-based Linux installs

July 9, 2017

In theory, burning Ubuntu install DVDs (or writing USB sticks) and then booting servers from them in order to do installs is an old-fashioned and unnecessary thing. One perfectly functional modern way is to PXE-boot your basic installer image, go through whatever questions your Ubuntu install process needs to ask, and then likely have the installer get more or less everything over the network from regular Ubuntu package repositories (or perhaps a local mirror). Assuming that it works, you might as well enable the Ubuntu update repositories as well as the basic ones, so that you get the latest versions of packages right from the start (which would deal with my wish for easily updated Ubuntu ISO images).

We don't do any sort of PXE or network installs, though, and we probably never will. There are a number of reasons for this. To start with, PXE network booting probably requires a certain amount of irritating extra setup work for each such machine to be installed, for example to add its Ethernet address to a DHCP server (which requires actually getting said Ethernet address). Ways around this are not particularly appealing, because they either require running an open DHCP server on our primary production network (where most of our servers go) or contriving an entire second 'install network' sandbox and assuming that most machines to be installed will have a second network port. It also requires us to run a TFTP server somewhere to maintain and serve up PXE images.

(This might be a bit different if we used DHCP for our servers, but we don't; all of our servers have static IPs.)

Next, I consider it a feature that you can do the initial install of a machine without needing to do much network traffic, because it means that we can install a bunch of machines in parallel at more or less full speed. All you need is a bunch of prepared media (and enough DVD readers, if we're using DVDs). As a purely pragmatic thing this also vastly speeds up my virtual machine installs, since my 'DVD' is actually an ISO image on relatively fast local disks. Even a local Ubuntu mirror doesn't fully help here unless we give it a 10G network connection and a beefy, fast disk system (and we're not going to do that).

(We actually have a local Ubuntu mirror that we do package upgrades and extra package installs from in the postinstall phase of our normal install process. I've seen some signs that it may be a chokepoint when several machines are going through their postinstall process at once, although I'd need to take measurements to be sure.)

Finally, I also consider it a feature that a server won't boot into the installer unless there is physical media plugged into it. Even with an installer that does nothing until you interact with it (and we definitely don't believe in fully automated netboot installs), there are plenty of ways for this to go wrong. All you need is for the machine to decide to prioritize PXE-booting higher than its local drives one day and whoops, your server is sitting dead in the installer until you can come by in person to fix that. On the flipside, having a dedicated 'install network' sandbox does deal with this problem; a machine can't PXE boot unless it's physically connected to that network, and you'd obviously disconnect machines after the install has finished.

(I'm going to assume that the Ubuntu network install process can deal with PXE-booting from one network port but configuring your real IP address on another one and then not configuring the PXE boot port at all in the installed system. This may be overly generous.)

The ultimate reason probably comes down to how often we install machines. If we were (re)installing lots of servers reasonably often, it might be worth dealing with all of these issues so that we didn't have to wrangle media (and DVD readers) all the time and we'd get a faster install overall under at least some circumstances. Our work in learning all about PXE booting, over the network Ubuntu installs, and so on, and building and maintaining the necessary infrastructure would have a real payoff. But on average we don't install machines all that often. Our server population is mostly static, with new arrivals being relatively rare and reinstalls of existing servers being uncommon. This raises the cost and hassles of a PXE netboot environment and very much reduces the payoff from setting one up.

(I was recently installing a bunch of machines, but that's a relatively rare occurrence.)


Comments on this page:

By Ewen McNeill at 2017-07-09 02:01:21:

I get why you don't want to set up a full PXE boot automated install environment. But it's less clear to me why you don't install with the Minimal Installer, which will drag in the packages to install from the network, but boot from CD. Of your issues, I think the only one left is the package installs are constrained by network bandwidth -- but generally I find that is approximately as fast as optical media, if not faster (especially with a local apt or even HTTP cache -- and having one of those around us useful in general).

Ewen

To start with, PXE network booting probably requires a certain amount of irritating extra setup work for each such machine to be installed, for example to add its Ethernet address to a DHCP server

not usually, just match the PXEClient vendor name. afaik both isc dhcp and dnsmasq can be configured to respond only to PXEClient requests and ignore all other DHCP packets.

All you need is for the machine to decide to prioritize PXE-booting higher than its local drives one day and whoops, your server is sitting dead in the installer until you can come by in person to fix that.

search --label FSROOT --set localroot
if [ -n $localroot ]; then
    root="$localroot"
    chainloader +1
    boot
else
    chainloader pxelinux.0
fi

on average we don't install machines all that often

especially since USB flash drives cost basically nothing in low quantities.

In terms of fully automated, I entirely agree - that's only useful in a limited set of highly controlled environments.

There's a middle path - using an intermediate bootloader that can load via PXE, such as iPXE, which can default to booting off local disk, and if interrupted presents a menu of diagnostic or installer images. This lets you run PXE on your primary subnet without the danger of accidental destructive install, but still requires all the dhcp/tftp setup.

Not having to fiddle with install media, or being able to easily drop into a memory or disk diagnostic is hugely helpful.

By Random at 2017-07-09 19:07:48:

My reasons for avoiding PXE are different, although maybe not 100% unrelated, but anyway, avoid it I do.

First, and most basic, I don't control the DHCP server for campus. That's kind of big. I could run my own, sure, and race for it. I have not found that to be effective. Alternatively, I could have an isolated network that I use to set up my servers first, but I can't imagine why I'd want that bother.

Also, what you said about fiddling around in BIOS.

But really, there's just not much point to me; between iDRAC Enterprise (yes, Dells) and Kickstart (yes, Fedora/CentOS) I'm good. And tack ansible onto the end of that, and I'm double plus good. I use iDRAC to attach the image as a virtual drive to the server, reboot it, remember one private network URL to use with "inst.ks=", let it finish, run ansible at it, and move on to the next. It's pretty satisfying.

Full disclosure: There's actual steps 3 and 4: icinga2 (I have a script, but there's still some human intervention to copy and paste a generated key and confirm a bunch of things) and MATLAB (again, mostly cookie cutter except running the quick and sadly X-based activate script, as their automated stuff has always been a huge hassle and I just stopped bothering with it.) But still. It's not bad.

Written on 09 July 2017.
« I wish you could easily update the packages on Ubuntu ISO images
Ubuntu's 'Daily Build' images aren't for us »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Jul 9 01:11:30 2017
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.