Getting the names of your Linux software RAID devices to stick

July 12, 2022

Suppose that you added some disks to a server and then used them to create a software RAID mirror called 'md2', like so:

mdadm -C /dev/md2 -n 2 -l raid1 -b internal /dev/sde1 /dev/sdf1

(Having an internal bitmap may be the default these days, I haven't checked.)

Everything goes great, the software RAID finishes resyncing (eventually), you reboot, and suddenly your system has a '/dev/md127' and no '/dev/md2'. What?

(At least, this is what happens on Ubuntu and on CentOS 7. Other Linux distributions may work differently, but most probably won't because a lot of this is behavior from udev and systemd.)

Once upon a time, perhaps back in the days when the Linux kernel assembled software RAID devices itself during boot, the name of a software RAID device was effectively persisted in its on-disk data. Once you created a RAID array called 'md2', you could be pretty confident that it would stay being /dev/md2 for the next long while, no matter what you did or didn't do. There were good things about those days but also problems (such as what was the system supposed to do if it saw two separate RAID arrays that both wanted to be md2), so modern Linux systems don't work that way.

In practice, what matters is what is in mdadm.conf, the mdadm configuration file. If your software RAID array is listed in there (with a /dev/mdN name), it will get assigned that name when it's brought up. If it's not listed in mdadm.conf, it will get assigned a high number (often starting with /dev/md127 and then apparently counting down) and that number may change from boot to boot if you have multiple unlisted software RAID arrays.

Thus, the minimal thing that you want to do is to add your new software RAID array to your mdadm.conf (which may be /etc/mdadm.conf or /etc/mdadm/mdadm.conf). Fortunately mdadm has a command option that will print out the information you need to add:

# mdadm --detail --brief /dev/md2
ARRAY /dev/md2 metadata=1.2 UUID=[...]

(You can also use 'mdadm --detail --scan'. I don't know why mdadm accepts both and treats them the same, but I think that using --brief is more accurate to what you want.)

The complication in this is that sometimes, you need the array to not just be listed in mdadm.conf, but also to be in the version of mdadm.conf that's in your initramfs. Which means that after you add it to the normal mdadm.conf, you probably want to run 'update-initramfs -u'.

All of this is easy to overlook if you mostly create your software RAID arrays when you install your systems (as we do). I believe that all competent modern Linux installers (Ubuntu, CentOS, whatever) take care of all of this for you, so that your install time software RAID arrays are listed in all of the right places and have nice static names and so on.

(See also how non-system software RAID arrays are started at boot under systemd and a much older entry on fixing a RAID growth gotcha that was also about having to update mdadm.conf. Hopefully the old RAID growth gotcha is long gone by now, but you never know. There's a more recent problem if you remove devices from a software RAID mirror, where you probably want to immediately zero the RAID superblocks on the removed devices.)

PS: If you refer to your software RAID arrays in /etc/fstab and so on through either the filesystem UUID or the RAID UUID, you aren't really affected by the /dev/md* name bouncing around. But that name does show up in various places, including host metrics and kernel logs, so it's my view that keeping it consistent and straight is good. And having a fixed name and an entry in mdadm.conf makes it clear that the array is supposed to be there, as opposed to being a stray array on reused disks or the like.


Comments on this page:

By Walex at 2022-07-13 15:32:42:

«but also to be in the version of mdadm.conf that's in your initramfs. Which means that after you add it to the normal mdadm.conf, you probably want to run 'update-initramfs -u'.»

That is a very good reminder, and applies unfortunately to several other system aspects. In general an "initramfs" image is both unnecessary (except for rootless systems) and a terrible idea, but for Linux it is now unavoidable because of the silly merger between / and /usr.

As to MD devices, it is till happens that on the "newer" version 1 superblocks "the name of a software RAID device was effectively persisted in its on-disk data": at creation it can be set with mdadm -C -N ... and it can be changed with mdadm -U name ... which uses the update-and-assemble option. There are also "homehost" and "home-cluster" fields used to prevent assembly of an array on the "wrong" system.

I also use an udevd based technique to create symbolic links that name specific devices or block devices or filesystems, to reduce the possibility of mistakes (filesystem, partition or disk names/UUIDs/IDs are a mess) and to make automounting a lot easier and clearer, as described in:

http://www.sabi.co.uk/blog/21-one.html?210503#210503

Written on 12 July 2022.
« It feels surprisingly good to block Bingbot from my blog front page
How Unix didn't used to support '#!', a brief history »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue Jul 12 21:34:23 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.