Don't use dd as a quick version of disk mirroring

October 16, 2014

Suppose, not entirely hypothetically, that you initially set up a server with one system disk but have come to wish that it had a mirrored pair of them. The server is in production and in-place migration to software RAID requires a downtime or two, so as a cheap 'in case of emergency' measure you stick in a second disk and then clone your current system disk to it with dd (remember to fsck the root filesystem afterwards).

(This has a number of problems if you ever actually need to boot from the second disk, but let's set them aside for now.)

Unfortunately, on a modern Linux machine you have just armed a time bomb that is aimed at your foot. It may never go off, or it may go off more than a year and a half later (when you've forgotten all about this), or it may go off the next time you reboot the machine. The problem is that modern Linux systems identify their root filesystem by its UUID, not its disk location, and because you cloned the disk with dd you now have two different filesystems with the same UUID.

(Unless you do something to manually change the UUID on the cloned copy, which you can. But you have to remember that step. On extN filesystems, it's done with tune2fs's -U argument; you probably want '-U random'.)

Most of the time, the kernel and initramfs will probably see your first disk first and inventory the UUID on its root partition first and so on, and thus boot from the right filesystem on the first disk. But this is not guaranteed. Someday the kernel may get around to looking at sdb1 before it looks at sda1, find the UUID it's looking for, and mount your cloned copy as the root filesystem instead of the real thing. If you're lucky, the cloned copy is so out of date that things fail explosively and you notice immediately (although figuring out what's going on may take a bit of time and in the mean time life can be quite exciting). If you're unlucky, the cloned copy is close enough to the real root filesystem that things mostly work and you might only have a few little anomalies, like missing log files or mysteriously reverted package versions or the like. You might not even really notice.

(This is the background behind my recent tweet.)


Comments on this page:

By Colin at 2014-10-16 07:47:50:

OK, overall this is why we should be using lvm and creating volumes and not just trusting rub to boot to ext2. Hence you point grub at the filesystem on the Logical volume not a filesystem on a disk?

By liam at unc edu at 2014-10-16 09:30:39:

If you are taking a downtime to add a disk, then why not do the mirroring? If you are hot adding the disk, and cloning a boot disk then just leaving your system running with no reboot test, well, that's a load of potential future problems regardless of what command cloned the disk!

Cheers, Liam

By Dan Astoorian (Dan.Astoorian) at 2014-10-16 10:19:13:

You may also want to use tune2fs -L (or e2label) to change the volume label of the cloned disk, since /etc/fstab can refer to filesystems by label rather than by UUID. (Technically, so can GRUB, although I don't know anyone who would go to the trouble to make it do so.)

By cks at 2014-10-17 01:04:20:

Colin: Unfortunately bringing up LVM volumes and so on involves at least as much magic as finding extN filesystems by UUID, and that magic is if anything more likely to go wrong when presented with two block devices with metadata that claims they are the same thing. I would rather be in this situation with plain extN, because at least it's relatively trivial to override disk metadata based detection for mounting them.

liam: We hot added the second disk (all of our servers support this) and intended it only as a cheap just in case precaution instead of anything more solid. We've actually rebooted the machine any number of times since then without problems, including as recently as last week.

(As a precaution it's probably not a fully baked idea and we may well wind up taking the disk out again now that we're actively thinking about this.)

Written on 16 October 2014.
« Why system administrators hate security researchers every so often
My experience doing relatively low level X stuff in Go »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Oct 16 02:13:02 2014
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.