A gotcha with Fedora 30's switch of Grub to BootLoaderSpec based configuration

August 16, 2019

I upgraded my office workstation from Fedora 29 to Fedora 30 yesterday. In the past, such upgrades been problem free, but this time around things went fairly badly, with the first and largest problem being that after the upgrade, booting any kernel gave me a brief burst of kernel messages, then a blank screen and after a few minutes a return to the BIOS and Grub main menu. To get my desktop to boot at all, I had to add 'nomodeset' to the kernel command line; among other consequences, this made my desktop a single display machine instead of a dual display one.

(It was remarkably disorienting to have my screen mirrored across both displays. I kept trying to change to the 'other' display and having things not work.)

The short version of the root cause is that my grub.cfg was rebuilt using outdated kernel command line arguments that came from /etc/default/grub, instead of the current command line arguments that had previously been used in my original grub.cfg. Because of how the Fedora 30 grub.cfg is implemented, these wrong command line arguments were then remarkably sticky and it wasn't clear how to change them.

In Fedora 29 and earlier, your grub.cfg is probably being maintained through grubby, Fedora's program for this. When grubby adds a menu entry for a new kernel, it more or less copies the kernel command line arguments from your current one. While there is a GRUB_CMDLINE_LINUX setting in /etc/default/grub, its contents are ignored until and unless you rebuild your grub.cfg from scratch, and there's nothing that tries to update it from what your current kernels in your current grub.cfg are actually using. This means that your /etc/default/grub version can wind up being very different from what you're currently using and actually need to make your kernels work.

One of the things that usually happens by default when you upgrade to Fedora 30 is that Fedora switches how grub.cfg is created and updated from the old way of doing it itself via grubby to using a Boot Loader Specification (BLS) based scheme; you can read about this switch in the Fedora wiki. This switch regenerates your grub.cfg using a shell script called (in Fedora) grub2-switch-to-blscfg, and this shell script of course uses /etc/default/grub's GRUB_CMDLINE_LINUX as the source of the kernel arguments.

(This is controlled by whether GRUB_ENABLE_BLSCFG is set to true or false in your /etc/default/grub. If it's not set at all, grub2-switch-to-blscfg adds a 'GRUB_ENABLE_BLSCFG=true' setting to /etc/default/grub for you, and of course goes on to regenerate your grub.cfg. grub2-switch-to-blscfg itself is run from the Fedora 30 grub2-tools RPM posttrans scriptlet if GRUB_ENABLE_BLSCFG is not already set to something in your /etc/default/grub.)

A regenerated grub.cfg has a default_kernelopts setting, and that looks like it should be what you want to change. However, it is not. The real kernel command line for normal BLS entries is actually in the Grub2 $kernelopts environment variable, which is loaded from the grubenv file, normally /boot/grub2/grubenv (which may be a symlink to /boot/efi/EFI/fedora/grubenv, even if you're not actually using EFI boot). The best way to change this is to use 'grub2-editenv - list' and 'grub2-editenv - set kernelopts="..."'. I assume that default_kernelopts is magically used by the blscfg Grub2 module if $kernelopts is unset, and possibly gets written back to grubenv by Grub2 in that case.

(You can check that your kernels are using $kernelopts by inspecting an entry in /boot/loader/entries and seeing that it has 'options $kernelopts' instead of anything else. You can manually change that for a specific entry if you want to.)

This is going to make it more interesting (by which I mean annoying) if and when I need to change my standard kernel options. I think I'm going to have to change all of /etc/default/grub, the kernelopts in grubenv, and the default_kernelopts in grub.cfg, just to be sure. If I was happy with the auto-generated grub.cfg, I could just change /etc/default/grub and force a regeneration, but I'm not and I have not yet worked out how to make its handling of the video modes and the menus agree with what I want (which is a basic text experience).

(While I was initially tempted to leave my system as a non-BLS system, I changed my mind because of long term issues. Fedora will probably drop support for grubby based setups sooner or later, so I might as well get on the BLS train now.)

To give credit where it's due, one (lucky) reason that I was able to eventually work out all of this is that I'd already heard about problems with the BLS transition in Fedora 30 in things like Fedora 30: When grub2-mkconfig Doesn’t Work, and My experiences upgrading to Fedora 30. Without that initial awareness of the existence of the BLS transition in Fedora 30 (and the problems it caused people), I might have been flailing around for even longer than I was.

PS: As a result of all of this, I've discovered that you no longer need to specify the root device in the kernel command line arguments. I assume the necessary information for that is in the dracut-built initramfs. As far as the blank screen and kernel panics go, I suspect that the cause is either or both of 'amdgpu.dpm=0' and 'logo.nologo', which were still present in the /etc/default/grub arguments but which I'd long since removed from my actual kernel command lines.

(I could conduct more experiments to try to find out which kernel argument is the fatal one, but my interest in more reboots is rather low.)

Update, August 21st: I needed to reboot my machine to apply a Fedora kernel update, so I did some experiments and the fatal kernel command line argument is amdgpu.dpm=0, which I needed when the machine was new but had turned off since then.

Written on 16 August 2019.
« Systemd and waiting until network interfaces or addresses are configured
A situation where Python has undefined values »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Aug 16 20:58:09 2019
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.