2005-10-25
A smooth disk swap
It's nice when everything works like it's supposed to; the most excitement we had today was swapping in a replacement drive in this server. (I like Mondays for this sort of thing, since I'll be in the office for the next four days in case problems come up.)
The old IDE disk needed replacement because it was starting to remind us that it was consumer electronics by flaking out on seeks under heavy IO load every so often (hopefully it was the drive and not the cabling). Fortunately we are smart enough to use IDE disks in RAID-1 pairs so swapping drives is actually pretty easy, apart from the physical bits.
This server isn't ritzy enough to have pluggable drive bays; it's in an ordinary 4U rackmount case, although one that has been nicely engineered. As part of the good engineering, it has all of the drives in a 'drive cage' assembly; undo some screws, pull the cables off, and you can lift the entire thing out.
So we opened the case up, pulled the drive cage out, took it off to a desk (much more convenient than the machine room), swapped the old and the new drive (important safety tip: always label the old drive), and dropped everything back together. Elapsed time: about 25 minutes, mostly due to lots of screws.
Because the two disks are identical, and partitioned identically, putting the new disk into action was more or less like this:
# sfdisk -d /dev/hda | sfdisk /dev/hdc # mkswap /dev/hdc2; swapon -a # mkfs /dev/hdc1; mount /boot2 # cd /boot2 # dump -0f - /boot | restore -xf - # mdadm -a /dev/md0 /dev/hdc5 # mdadm -a /dev/md1 /dev/hdc6 [... you can see how this goes ...]
Linux people may not have seen
sfdisk before. I
quite recommend it; it's a very convenient, rather Unixy command for
fiddling around with partition tables. (The incantation used above
copies /dev/hda's partitioning to /dev/hdc, which is exactly what
we wanted.)
Tomorrow will be more thrilling, since backups of one partition of one server are now definitely failing consistently. Mysteriously.