== Fixing your system after hitting the RAID growth gotcha

The easiest way to dig yourself out of the hole created by the [[RAID
growth gotcha RaidGrowthGotcha]] is probably to use a live/rescue CD.
But let us suppose that you don't have one handy, which was the case
for me yesterday. Further let us suppose that you have _/boot_ as a
[[separate filesystem SeparateBootAdvantage]], not as part of the root
filesystem (if this is not the case, you absolutely need a rescue CD;
sorry).

The basic goal is to rebuild a version of your current initrd that has
an updated _/etc/mdadm.conf_ that specifies that your root mirror has
the right number of devices. Since we can't boot the system normally,
we can't just bring it up, edit the real _mdadm.conf_, and regenerate
the initrd with the normal tools; instead, you need to boot the system
in a minimal mode and unpack, fix, and rebuild the initrd by hand.

First, you need some setup:

* you need to [[boot the system without the root mirror
  BootingWithoutMirror]]. ~~Heed the cautions~~.

  Once booted, you'll probably want to mount _/usr_ (read-only),
  if only so that you can read manpages.

* you need writeable scratch space to rebuild the initd; I mounted
  a tmpfs _/tmp_.

* you need _/boot_ mounted read-write; if your _/boot_ is [[mirrored
  GoingFullyMirrored]], you'll have to assemble the software RAID
  first with appropriate _mdadm_ invocations.

After that, it is relatively simple:

* make a scratch directory in _/tmp_ (or wherever) and unpack your
  current initrd to it. Initrds are compressed cpio images, so this
  is something like:
	_cd /tmp/t; zcat </boot/initrd | cpio -di_

* edit the now-unpacked _etc/mdadm.conf_ to have the right
  _num-devices_ value for the software RAID with your root
  filesystem. You don't need to update the numbers for the
  other software RAID devices; they aren't started in the
  initrd.

* reassemble the initrd. On Fedora, the 100% authentic way
  to do this, exactly duplicating what _mkinitrd_ does, is:
	_echo nash-find . | /sbin/nash --force --quiet |
	cpio -H newc --quiet -o | gzip -9 >/tmp/initrd_

* rename your current initrd to something else as a backup,
  and then copy your newly generated initrd into _/boot_
  with the right name.

At this point you can reboot and your system should come up as far
as mounting the root filesystem. If you have other software RAID
mirrors it will then throw a fit because none of them will successfully
assemble either, and drop you into rescue mode. To fix this, edit
_/etc/mdadm.conf_ to specify the right _num-devices_ number for all
software RAID mirrors (*including* the root mirror). After you've
done this, reboot and things should work.

(You may need to make the root filesystem read-write first with
'_fsck /_' followed by '_mount -o remount,rw /_'.)

If you have no other software RAID mirrors, you still need to update
_/etc/mdadm.conf_ once the system has booted or you will get to go
through this all over again after the next kernel update.

(You can see why I suspect a rescue CD would be easier. With a rescue CD
you should be able to assemble and mount the root filesystem, chroot to
it and set any other necessary filesystems up, edit _/etc/mdadm.conf_,
and then run _mkinitrd_ or the like.)

For safety, you probably want to rebuild your current initrd using
the official tools after doing the full _mdadm.conf_ update.