The unfortunate configuration choice Grub2 makes in UEFI configurations

April 15, 2018

When I talked about my new home machine, I mentioned that I wasn't even trying to use UEFI on it after my experiences with my work machine, fundamentally because Grub2's UEFI setup makes an unfortunate configuration choice. Today I'm going to talk about what that choice is and why it's an unfortunate one, at least for people like me (and for people with servers).

Put simply, it's that the UEFI Grub2 requires you to put grub.cfg in the EFI system partition. On the one hand, this is a reasonable choice, and probably one that simplifies Grub's life. In a UEFI environment, the EFI system partition exists and is easily accessible through (U)EFI services, so you have a natural place to put everything else that Grub2 needs, including both its additional modules and grub.cfg. In a traditional non-UEFI environment, grub2 needs to do some magic to be able to load your grub.cfg from whatever sort of filesystem and RAID setup and so on your /boot is in; the actual mechanisms of that are relatively impressive and more than a bit complex (cf). UEFI makes life much simpler.

On the other hand, grub.cfg is the one piece of your Grub2 configuration that changes frequently, because it gets updated every time you add a kernel, remove a kernel, or modify kernel command line arguments. This is an issue for me and for people with servers, because the EFI system partition can't be part of a RAID mirror. If you want to be able to boot from a second disk if your primary disk fails, you need a duplicate copy of your EFI system partition, and because grub.cfg changes frequently, you need to keep this up to date on a frequent basis. Otherwise, not only will you perhaps be booting from an older (but functional) version of Grub2 that you didn't update, you'll probably be trying to boot kernels that don't even exist any more (perhaps with wrong or missing kernel arguments). You'll probably be able to boot your system in the end, but it's not likely to be easy or automatic.

Life would be a lot easier and better here if you could configure Grub2 to load your real grub.cfg from your (non-EFI) /boot. You could use software RAID, LVM, and any filesystem normally supported by Grub2. People with mirrored system disks would still have all of the good stuff that you get with them in a non-EFI configuration (although every so often you'd need to remember to update the stuff in your EFI system partition on the second disk).

My guess is that the easiest way to add this to Grub2 would be to give Grub2 some way of including additional files in grub.cfg. With this, you'd still have a stub grub.cfg in the EFI system partition (which Grub2 could load with UEFI services just like today), and this stub would specify everything else. It would know the UUID of the filesystem with your /boot in it and also what additional RAID or LVM UUIDs it needed to look for and start in order to find it, just as a non-EFI Grub2 knows those details today (cf), but these wouldn't change very often so your EFI system partition grub.cfg would stay mostly unchanging.

Of course this Grub2 configuration choice isn't important unless you have mirrored system disks. If your system disk is unmirrored, an unmirrored EFI system partition creates no additional problems and the current UEFI Grub2 design is fine. Since this probably describes most systems using UEFI today, I don't expect UEFI Grub2 to change any time soon. Probably it will only start to change when servers become UEFI-only and people running them discover that their mirrored, redundant system disks actually aren't any more because of this.

Comments on this page:

Ubuntu build this stub cfg into grub.efi.

In the absence of RAID, apparently there are arguments to put the real grub.cfg on EFI, other than implementation convenience.

grub-install has often been used on multiple disks (c.f. the Debian package install prompts)... maybe the distro update-grub scripts need to learn about multiple EFI partitions.

From at 2018-04-15 05:28:44:

My guess is that the easiest way to add this to Grub2 would be to give Grub2 some way of including additional files in grub.cfg. With this, you'd still have a stub grub.cfg in the EFI system partition (which Grub2 could load with UEFI services just like today), and this stub would specify everything else.

At least on Debian, that's already the case. The grubx64.efi that's on the EFI partition has everything embedded (the drivers, the initial stub config which says "the real grub.cfg is at (,gpt2)/boot/grub"...).

(While it's done automatically at grub-install time, I think it could be built manually by using grub-mkimage --config= as well.)

I have a UEFI-only Supermicro system with mirrored disks where I installed Fedora 27 using the default installer (selecting mirrored BTRFS setup).

The result is: /boot/efi is also created on a Linux RAID-1 device (superblock format 1.0) while the underlying GPT EFI system partitions actually have the type 'Linux RAID' instead of 'EFI system'.

This works pretty good in practice, i.e. the Supermicro UEFI firmware doesn't care about the GPT partition type - it happily uses the first vfat filesystem with the right files at the right locations as EFI system partition. Also, the RAID-1 superblock is at the end of the partition, thus the firmware doesn't have to be aware of it.

I tested a similar setup on other non-Supermicro systems with UEFI boot and it works there, as well.

By cks at 2018-04-15 16:32:35:

Apparently some UEFI BIOSes don't like the superblock format 1.0 trick, and in general you're in trouble if anything ever writes to the EFI system partition at the UEFI level. The whole set of uncertainties persuaded me to not try it on my current workstations; traditional BIOS boot just seems simpler and I know it works. But in the future I may have no choice if vendors really do start shipping UEFI only motherboards.

Thanks for the information about the Ubuntu and Debian approaches, especially since it taught me about Grub2's configfile directive, which does just what I want as far as including an additional file. I'm probably not going to try to configure my Fedora setups this way because I'd have to figure out a way to get Fedora to update my real /boot/grub2/grub.cfg config file when it installs new kernels instead of the stub one in the EFI system partition, but at least I have the option with the standard Grub2.

By Samuel Hodgkins at 2018-04-18 02:00:43:

I use a Gentoo box as a desktop, and I'll doublecheck later but that box's /boot is stored as a ZFS dataset on a single-disk rpool. Last I looked all the tiny ESP contained was the .efi GRUB uses as stage1, and unlike with Debian/Ubuntu it's not a standalone image, the .mod files are loaded from /boot as is grub.cfg. Perhaps your distro just mounts the ESP at /boot rather than somewhere else? (See also: --efi-directory when running grub-install, example at -- Samuel Hodgkins if the 'who are you' field doesn't show up

By cks at 2018-04-18 14:51:40:

You got me to look at my Fedora laptop, which uses UEFI and has been an all-UEFI install from the start. It mounts the EFI system partition as /boot/efi, but just as with Gentoo the Grub(2) .mod files are not in there; all it has is a number of .efi files (plus grub.cfg, grubenv, and a Unicode font). None of the .efi files appear to be dynamically generated at install time, as rpm -V claims they're unchanged from the RPM package version. Based on what I can see of the Grub2 $root and $prefix variables, the Fedora Grub .efi files must have the relevant Grub modules bundled in (or built in) somehow, because it doesn't seem to have anything defined that would let it load them from their Fedora location in /boot/grub2/x86_64-efi, which is not in the EFI system partition and so not directly available at boot.

By Samuel Hodgkins at 2018-04-18 16:35:51:

Apparently this setup is a Fedora thing rather than being caused by GRUB: 'grub2-install shouldn't be used on EFI systems. The grub2-efi package installs a prebaked grubx64.efi on the EFI System partition, which looks for grub.cfg on the ESP in /EFI/fedora/ whereas the grub2-install command creates a custom grubx64.efi, deletes the original installed one, and looks for grub.cfg in /boot/grub2/.'

Written on 15 April 2018.
« For the first time, my home PC has no expansion cards
Some notes and issues from trying out urxvt as an xterm replacement »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Apr 15 03:07:46 2018
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.