Wandering Thoughts archives

Why booting Linux from a ZFS root filesystem with GRUB can be hard

When I talked about the current weak areas of ZFS on Linux, one of them was using ZFS as your root filesystem (generally with Grub). Today I want to talk a bit more about why this is a non-trivial thing and by extension why booting from any new filesystem type takes more than you might think.

But first, let's be technical here because there can be two filesystems involved: your actual root filesystem and the boot filesystem. The boot filesystem is the filesystem with your kernel and initramfs; the root filesystem is /. These days it's common to have both in the same filesystem and for now I'll assume that's the setup we're dealing with.

In order to boot from ZFS we need several things to work. First, the GRUB bootloader code itself needs to understand enough about ZFS to be able to read things from a ZFS filesystem, most notably the GRUB menu file, the kernel, and the initramfs. Code for doing this is available (Solaris and Illumos have booted from ZFS using GRUB for some time) and based on the current Fedora source RPM for GRUB 2, it appears to be integrated into the main GRUB source. I believe that this also means that GRUB 2 will normally autodetect that /boot is on a ZFS filesystem and build support for this into the GRUB boot image that's written to your disk by grub-install (aka grub2-install on some machines).

(However the GRUB ZFS code may not support the latest ZFS pool and filesystem features and I'm not certain what sort of pool vdevs it supports. Also, I hope it goes without saying that your ZFS root pool needs to be entirely on devices that the BIOS will see at boot time, because those are the devices that GRUB 2 can talk to.)

But that's only the start, because GRUB is out of the picture after the kernel and initramfs have been loaded and started. Once the kernel is running the initramfs needs to include ZFS and know how to find the ZFS root pool and root filesystem. This needs both kernel ZFS modules and code in the initramfs scripts (and a way of specifying what the root pool and root filesystem are). Very few distributions will natively include this support because very few are packaging or officially supporting ZFS on Linux today.

If you have both of the above you can boot Linux with a ZFS root filesystem. The final step is to be able to build or rebuild your initramfs with the ZFS support (still) there. This needs whatever code in your distribution creates this stuff to be ZFS aware; it needs to recognize that your root is a ZFS filesystem, work out the right ZFS pool and filesystem name, and embed all of this (plus general ZFS support) into the initramfs, the GRUB menu, and so on. Again many distributions won't have this natively and will need replacement packages or patching or the like.

(All of this ignores the distribution's installer. If you want to be ZFS from the start, you need an installer that supports ZFS and includes the ZFS modules and tools and so on. My impression is that this is rare.)

What changes if you separate out the boot filesystem from the root filesystem and make the boot filesystem not a ZFS filesystem is that your version of GRUB doesn't need to understand ZFS any more. Since ZFS support seems to be common in any recent version of GRUB, this may not get you very much.

(In a sense this is good news, since it's much easier to fiddle around with the contents of your initramfs than it is to add support for another filesystem to GRUB. Once you have ZFS support for your kernel all you need to do is work out how to get it into the initramfs too.)

PS: You can check if your version of GRUB (well, GRUB 2) has ZFS support by checking to see if you have a zfs.mod GRUB module hanging around somewhere in /boot/grub2 or /boot/grub or wherever your distribution puts it.