Wandering Thoughts archives

LILO vs GRUB: why LILO still wins for servers

IBM Developerworks recently carried an article comparing the LILO and GRUB Linux bootloaders that wound up concluding there was nothing LILO did better than GRUB.

I beg to disagree, and strongly. LILO has two important features for server operation:

First, a one-shot mode, where LILO tries booting a particular kernel only once; after that, it goes back to its normal default. This is a reassuring way of trying new kernels that you are not entirely sure will work. If you set the kernel up to reboot on panics, you have a good chance that the system will automatically recover into your old and known to work kernel. (It's especially handy on servers without remotely accessible serial consoles, which would otherwise require an in-person visit to recover.)

(We are not rich enough to buy remotely controllable power switches and 'managed servers' that dump everything from the BIOS on up out the serial port. If you are, maybe you don't have to care about this.)

Second, it's easy to use both a serial line and the normal console. LILO can be set up to simultaneously use both the normal PClone video console and a serial line. Serial consoles are an important part of hands-off management of servers, but at the same time you do want to be able to do in-person hands-on visits.

GRUB has some support for serial consoles, but from the documentation it appears that GRUB wants to pick one 'console' and speak only to it. LILO will echo output on both, which is very helpful when examining or watching things. (I welcome corrections, and I consider this the less important difference for server usage.)

Plus, frankly, I do not want complicated near-graphical menus in a server environment. A simple command line is very appealing to me. (As far as I know, GRUB's command line is not useful for invoking menu entries by name, just for entering raw GRUB commands. And if you use menu entries, you have to get a menu of some sort.)

Sidebar: what about LILO's hardcoded block numbers?

One of the often-repeated downsides of LILO is that it doesn't read the filesystem but instead records the block numbers of where your kernels are on the disk, and therefor goes to pieces if they change (and you don't rerun lilo).

Although it's not widely advertised, GRUB uses hardcoded block numbers too (read the documentation of the 'install' command carefully). It just uses them for less, so you get hit by it less often.

So with GRUB you get affected less, but with LILO everyone knows about the issue. (In my experience, the early-stage GRUB diagnostics are not as helpful as LILO's.)

How not to package software in RPMs

I spent today wrestling various pieces of commercial software into our Fedora Core 4 auto-install environment. In theory this should have been fast and dirt simple, because all of the vendors had things packed up in RPMs.

In practice, most vendors produce RPMs that are glowing examples of how not to package your software in RPM form. Instead of creating RPMs that are fully installed software, they generally create RPMs that are little more than lightly packaged up files and then add huge amounts of post-install scripts to shuffle everything around.

Pride of place has to go to the Sun JDK RPM, which:

• does not even ship as an RPM; instead it ships as a shell script which extracts the RPM and tries to immediately install it.
• has almost a thousand lines of post-install and pre-uninstall scripts (!!).
• uses those hundreds of lines of post-install scripts to unpack files, add files, and remove archive files during the installation.

(Newsflash to Sun: you should do all that before you build the RPM, and then package the final result as the RPM. That's what RPMs are for to start with, the fully installed and unpacked binary version of your software.)

The Flash RPM I found refused to activate itself without an X display to talk to. So much for using that in RPM form at all. (Fortunately Flash is just two files in /usr/lib/mozilla/plugins, which is easy enough to do in a script.)

Others of the RPMs that I had to look at today 'merely' feel free to do things like delete RPM-managed files belonging to other packages and replace them with symlinks.

Things like this matter because almost everyone running Fedora Core 4 is managing a great deal of their system through RPM. When a package mangles the system behind RPM's back, the entire environment becomes fragile. (All of the stuff I covered in CPANProblem applies.)

The irritating thing about these system manglings is that by and large they're unnecessary, especially things like the JDK ugliness. These companies are not running into novel problems in building RPM packages, and other software manages to do the right thing. The companies were just lazy. (And because of the license terms of their software, outsiders can't fix the problem, build proper packages, and distribute the results.)

(What'd I do? I shrugged, checked to make sure the post-install scripts didn't do anything too heinous, held my nose, and just used them as is. But I had to spend the time to read over the scripts, check out the results, and so on.)

Completely using an alternate yum configuration

As part of automatic installs of Fedora Core, our customization step installs of the current Red Hat updates from a local mirror. We use yum for this, because it both handles dependencies and uses compact metadata files instead of having to read all of the update and base OS RPMs themselves. (We have 100 megabit networks, but it still adds up and slows down installs.)

We do this by feeding yum a custom configuration file; basically we take the standard /etc/yum.conf file and add some bits to point to our on-disk repositories. (In turn, this means I get to explore the many charms and wonders of building yum metadata. Which deserves an entry itself.)

This mostly works, but with Fedora Core 4 this step started bitching about 'ignoring duplicate entry for <repository type>'. Since it worked, I ignored it. As part of diagnosing the install failure yesterday, this escalated to a fatal problem: yum died with that ever-helpful message:

Cannot find a valid baseurl for repo: extras

This was mysterious, because our yum configuration doesn't have an 'extras' repository, only 'base' (base Fedora Core 4 RPMs) and 'updates-released' (the updates themselves).

The cause is a new feature in yum, where it will look in a list of directories for files that describe additional repositories. (In Fedora Core 4, Red Hat sets up all of the repositories this way for reasons having to with how RPM operates.)

This new features, reposdir, does not have to be explicitly specified in your yum configuration file; yum uses a default directory search path. Fedora Core 4 uses both the feature and one of the default directories, so no reposdir entry is in /etc/yum.conf.

Since our one started as a copy of that, we hadn't set a reposdir either, so yum was seeing our explicitly specified repository locations and then also reading the normal system ones. This resulted in the duplicates of 'base' and 'updates-released' that yum had been warning about, and when the install-time nameservice was broken a complete inability to get information on the contents of 'extras', a fatal error.

So: to completely use an alternate yum configuration file, you need to set an explicit reposdir. Yum is happy with the directory not existing (assuming you specify the necessary repositories explicitly in your configuration file), but you have to override the default with something.

Disk setup in our Fedora Core 4 Kickstart environment

The problem with working on automatic installers is that testing them is such a slow and tedious process. These days, it takes about 30 minutes to reinstall Fedora Core 4, with patches and local changes, on my test machine. (Then when something looks like it works, I have to do three tests instead of just one: upgrade from Fedora Core 2 (with an extra reinstall to get back to FC2), reinstall Fedora Core 4, and install from a USB memory stick.)

Because of this I got to drum my fingers a lot today as I was nailing down the changes necessary (I think, I haven't been able to test them yet) to make our Kickstart disk setup stuff work on SATA machines as well as IDE ones.

When we Kickstart-install a machine, we want it to have nothing left on its disks except our partitions, created during the install. Essentially, we want to start with a clear partition table: either initialized from scratch or with all the existing partitions deleted. Confusing things, Kickstart has two directives to control this:

• clearpart will let you delete some or all partitions.
• the alarmingly named zerombr seems to just cause Kickstart to ignore invalid Master Boot Records on disks and initialize them from scratch.

We need zerombr to deal with new disks straight from the vendor that don't have a valid MBR. We need clearpart to deal with things like new systems that ship pre-partitioned and with Windows installed (as well as reinstalling systems). Thus, our Fedora Core 2 Kickstart config file used both.

Unfortunately, the Fedora Core 4 Anaconda bugs mean that the clearpart half of this is unusable. Our workaround destroys the MBR before Kickstart starts looking at the disk partitions, so the sequence we now have is:

1. a %pre section that zaps out sector 0 on the first IDE and/or SATA disk. (Using tools mentioned in ModernDiskStoragePains to make sure that we're dealing with a SATA disk instead of a SCSI or a USB one).
2. zerombr to tell Kickstart that it's OK to touch the disk with the invalid MBR.
3. regular Kickstart partitioning.

Zeroing out sector 0 is done with a simple

dd if=/dev/zero of=<device> bs=512 count=1

I could avoid specifying zerombr at all if step #1 instead set up a valid but empty partition table, but that would take something more complicated than dd. Since Anaconda already has that code, I figure I'll let it do the work.

And that was my day (apart from the usual of being nibbled to death by moths, and some work on the USB memory stick version of the installer).

(Note: yes, I am sort of using Kickstart and Anaconda interchangeably in this entry. Yes, I know the difference.)

Sidebar: a heuristic for detecting SATA disks

It turns out that the kernel seems to name all of its SATA drivers things starting with sata_. So my disk clearing code works by getting the driver responsible for /dev/sda, if there is one, and clearing sector 0 only if the driver name starts with sata_. Fortunately this is dirt simple in Bourne shell code.

If we find a SATA chipset with a different driver name, I'll have to adjust the heuristic.

Updated: I had to add ata_piix to the list of SATA driver names, since it breaks the pattern of sata_ at the start. Whoops, as they say.

The pains of modern disk storage

I spent today building some of the infrastructure that will hopefully let us safely Kickstart-install Fedora Core 4 on the new generation of SATA-based PC workstations. (Safely, because destroying a server by accidentally starting a Kickstart install on it is one of my quiet nightmares.)

What makes this difficult is a a simple decision, by itself sane and rational: the Linux kernel people decided that rather than making up a new set of block devices for things like USB memory sticks or SATA disks, they would just reuse the SCSI system. (This is helped along by the fact that pretty much every new disk interface uses some subset of the SCSI command set.)

This means that if you have a /dev/sda on a modern Linux system, you don't know very much about what it is. It might be a real SCSI disk on a server; it might be a SATA disk on a workstation; it might even be the USB memory stick you just booted a new computer from. (USB memory sticks look like hard drives to the PC, which has other interesting consequences.)

Enter Fedora Core 4's buggy Anaconda (still not fixed, not that I'm bitter or anything). The 'cannot repartition drives' bug means that we need to manually destroy the partition table on the install disk. Which means that we need to work out what the install disk is.

Our old workstations were pure IDE systems, so the answer was simple: /dev/hda, if it existed. If it didn't exist, we didn't want to go on anyways. On a SATA-based system, the main drive will be /dev/sda when the system finishes installing, but might not be during an install from a USB memory stick. And our servers have /dev/sda drives that we definitely don't want to wipe out.

Today I built a program that works out what kernel module is responsible for which 'SCSI' disk, if any of them exist (using Linux's sysfs filesystem, which exports a bunch of kernel information). The next step is to make our Kickstart system not touch a /dev/sda that is owned by aic7xxx (real SCSI on servers) or usb-storage (USB memory drives), but be willing to zap others.

(Unfortunately, as far as I know there is no way to tell that a particular device has the generic property 'SATA device'. If there was, it would simplify life.)