2024-04-10
It's far from clear how grub package updates work on Ubuntu
Recently I ran across (and eventually reported) an issue on pre-beta Ubuntu 24.04 where a grub package update would fail for systems with software RAID root disks and BIOS MBR booting. The specific error was that grub-install could not install the new version of GRUB's boot-time code on '/dev/md0', the (nominal) device of the root filesystem, reporting an error to the effect of:
grub-install: warning: File system `ext2' doesn't support embedding. grub-install: warning: Embedding is not possible. GRUB can only be installed in this setup by using blocklists. However, blocklists are UNRELIABLE and their use is discouraged.. grub-install: error: diskfilter writes are not supported. grub-install failure for /dev/md0
(You can work around this by reconfiguring the grub package to use
the underlying disk devices, either by doing 'dpkg-reconfigure
grub-pc' or by installing package updates in a manner where dpkg
is allowed to ask you questions. Also, this is another case of
grub-install having unclear error messages.)
One of the puzzling things about this entire situation is that the exact same configuration works on Ubuntu 22.04 and there are no obvious differences between 22.04 and 24.04 here. For instance, there are debconf keys for what the root filesystem device is and they are exactly the same between 22.04 and 24.04:
; debconf-show grub-pc [...] * grub-pc/install_devices: /dev/disk/by-id/md-name-ubuntu-server:0
At this point you might guess (as I did) that 'grub-install /dev/md0' works on Ubuntu 22.04. However, it does not; it fails with the same error as in 24.04. So presumably how grub-install is invoked during package updates is different between 22.04 and 24.04.
As far as I can tell, the 'grub-pc' package runs grub-install from its 'postinst' script, which you can find in /var/lib/dpkg/info/grub-pc.postinst. If you take a look at this script, you can see that it's a rather complex script that is quite embedded into the general Debian package update and debconf framework. If there are ways to run it as a standalone script so that you can understand what it's doing, those ways aren't at all obvious. It's also not obvious how the script is making or not making decisions, and the 22.04 and 24.04 versions seem pretty similar. Nor does scanning and searching either version of the script provide any smoking guns in the form of, for example, mentions of 'md-'.
(You have to know a reasonable amount about dpkg to even find /var/lib/dpkg/info and know that the 'grub-pc.postinst' file is what you're looking for. The dpkg manual page does mention that packages can have various scripts associated with them.)
All of this adds up to something that's almost impossible for ordinary people to troubleshoot or debug. All we can readily determine is that this worked in Ubuntu 20.04 LTS and 22.04 LTS, and doesn't work in the pre-beta 24.04 (and probably not in the beta 24.04, and most likely not in the released 24.04). The mechanisms of it working and not working are opaque, buried inside several layers of black boxes.
Part of this opacity is that it's not even clear what Ubuntu's grub package does or is supposed to do on package update. If you run a UEFI system with mirrored system disks, for example, you may be a little bit surprised to find out that Ubuntu's grub is probably quietly updating all your EFI system partitions when it does package updates.
PS: after much delving into things using various tools and the fact that I have various scratch virtual machines available, I now believe that the answer is that Ubuntu 20.04 and 22.04 don't run grub-install at all when the grub package (for MBR booting) is updated. This fact is casually semi-disguised in the 20.04 and 22.04 grub-pc postinst script. Presumably the 20.04 and 22.04 server installer should have set 'grub-pc/install_devices' to a different value, but that problem was being covered up by grub-install normally not running and using that value.