Wandering Thoughts: Recent Entries For 2012/07/20

Ubuntu 12.04 can't reliably boot with software RAID (and why)

Recently one of my co-workers discovered, diagnosed, and worked around a significant issue with software RAID on Ubuntu 12.04. I'm writing it up here partly to get it all straight in my head and partly so we can help out anyone else with the same problem. The quick summary of the situation comes from my tweet:

Ubuntu 12.04 will not reliably boot a system with software RAID arrays due to races in the initramfs scripts.

(As you might guess, I am not happy.)

If you set up Ubuntu 12.04 with one or more software RAID arrays for things other than the root filesystem, you will almost certainly find that some of the time when you reboot your system it will come up with one or more software RAID arrays in a degraded state with one or more component devices not added to the array. If you have set bootdegraded=true as one of your boot options (eg on the kernel command line), your system will boot fully (and you can hot-add omitted device back to the array); if you haven't, the initramfs will pause briefly to ask you if you want to continue booting anyways, time out on the question, and drop you into an initramfs shell.

This can happen whether or not your root filesystem is on a software RAID array (although it doesn't happen to the root array itself, only to other arrays) and even if you do not have the software RAID arrays configured or used in your system in any way (not listed in /etc/mdadm/mdadm.conf, not used in /etc/fstab and so on); simply having software RAID arrays on a disk attached to your system at boot time is enough to trigger the problem. It doesn't require disks that are slow to respond to the kernel (to the extent that we've reproduced this in VMWare, where the disks aren't even physical and respond to kernel probes basically instantly).

Now let's talk about how this happens.

Like other modern systems Ubuntu 12.04 handles device discovery with udev, even during early boot in the initramfs. Part of udev's device discovery is the assembly of RAID arrays from components. What this means is that software RAID assembly is asynchronous; the initramfs starts the udev daemon, the daemon ends up with a list of events to process, and as it works through them the software RAID arrays start to appear. In the mean time the rest of the initramfs boot process continues on and in short order sets itself up to mount the root filesystem. As part of preparing to mount the root filesystem, the initramfs code then checks to see if all visible arrays are fully assembled and healthy without waiting for udev to have processed all pending events. You know, the events that can include incrementally assembling those arrays.

This is a race. If udev wins the race and fully assembles all visible software RAID arrays before the rest of the initramfs checks them, you win and your system boots. If udev loses the race, you lose; the check for degraded software RAID arrays will see some partially assembled arrays and throw up its hands.

Our brute force solution is to modify the check for degraded software RAID arrays to explicitly wait for the udev event queue to drain by running 'udevadm settle'. This appears to work so far but we haven't extensively tested it; it's possible that there's still a race present but it's now small enough that we haven't managed to hit it yet.

This is unquestionably an Ubuntu bug and I hope that it will be fixed in some future update.

Sidebar: our fix in specific

(For the benefit of anyone with this problem who's doing Internet searches.)

Change /usr/share/initramfs-tools/scripts/mdadm-functions as follows:

 degraded_arrays()
 {
+	udevadm settle
 	mdadm --misc --scan --detail --test >/dev/null 2>&1
 	return $((! $?))
 }

Then rebuild your current initramfs by running 'update-initramfs -u'.

Since I suspect that mdadm-functions is not considered a configuration file, you may want to put a dpkg hold on the Ubuntu mdadm package so that an automatic upgrade doesn't wipe out your change.

(This may not be the best and most Ubuntu-correct solution. It's just what we've done and tested right now.)

Sidebar: where the bits of this are on 12.04

• /lib/udev/rules.d/85-mdadm.rules: the udev rule to incrementally assemble software RAID arrays as components become available.

Various parts of the initramfs boot process are found (on a running system) in /usr/share/initramfs-tools/scripts:

• init-top/udev: the scriptlet that starts udev.

• local-premount/mdadm: the scriptlet that checks for all arrays being good; however, it just runs some functions from the next bit. (All of local-premount is run by the local scriptlet, which is run by the initramfs /init if the system is booting from a local disk.)

• mdadm-functions: the code that does all the work of checking and 'handling' incomplete software RAID arrays.

Looking at this, I suspect that a better solution is to stick our own script in local-premount, arranged to run before the mdadm script, and have it run the 'udevadm settle'. That would avoid changing any package-supplied scripts.

(Testing has shown that creating a local-top/mdadm-settle scriptlet isn't good enough. It gets run, but too early. This probably means that modifying the degraded_arrays function is the most reliable solution since it happens the closest to the actual check, and we just get to live with modifying a package-supplied file and so on.)

The temptation of selective sender address verification

You could say that we have at least three mail problems. We have a requirement to accept all email by default, we have people who either forward all their mail to places with more strict spam policies or have autoreplies, and such people sometimes get email from known domains that can't be sent email (sometimes in annoying ways). The result is a moderate amount of email camped out on our system that I know will never be delivered, and a sender address verification temptation.

There all sorts of reasons why general callback-based sender verification is a bad idea; people much more articulate than I am have written lots about this. I would never do general sender verification. However, the temptation is that I could make our mailer do selective sender verification, verifying only addresses at certain domains. Say, the known domains that send us stuff that then causes bounces or autoreplies that said domains never accept back. This wouldn't make callback verification any less of a bad thing but on the other hand I would only be applying it to domains that are already behaving antisocially towards us (as far as we're concerned). In theory , this would have the net effect of blocking email from those 'bad' domains but (mostly) only to the extent that they themselves are being bad SMTP citizens; if they became willing to accept their email addresses back, we'd become willing to accept their email.

(In theory this is what happens with all uses of callback sender verification.)

Sadly, this is a bad idea and illustrates a certain dangerous mindset that I can get into when I think about spam. Spam fundamentally irritates me; it just gets under my skin and acts as one of my hot buttons. It's uncomfortably easy for me to slip into a mindset where I start losing balance and perspective, one where the most important thing is blocking spam and other concerns are decidedly secondary.

In theory I could argue that we have a defensible technical reason for insisting on callback verification for these domains; after all, our email system effectively chokes on these undeliverable addresses. In practice I know full well that the mail system is not being particularly affected by the current or likely future level of delivery attempts to these domains and that the real reason I want callback sender verification for these domains is that it'll cause us to stop accepting their email. We have no mandate to stop accepting email from domains I don't like (rather the contrary), so I'd be trying to sneak it in through the back door in a vaguely deniable fashion.

(Having written this, I am hopefully now somewhat better prepared to resist this particular temptation. But I have to say that it's very attractive to the bit of me that likes poetic justice and hates spam.)