Wandering Thoughts archives

2020-04-29

The problem of Ubuntu 20.04, Snaps, and where your home directory is

I tweeted:

Due to Chromium becoming a Snap in Ubuntu 20.04 and our user home directories not being under /home, it looks like our users will have to live without Chromium. Thanks, Ubuntu.

This may sound a little bit hard to believe, so let me show you the stages of trying to use chromium-browser on 20.04 (installed by doing 'apt-get install chromium-browser'):

; chromium-browser
2020/04/30 00:44:29.065359 cmd_run.go:560: WARNING: XAUTHORITY environment value is not a clean path: "/h/281/cks/.Xauthority"
 cannot open path of the current working directory: Permission denied
; cp -a $HOME/.Xauthority /tmp/cks-xauth; export XAUTHORITY=/tmp/cks-xauth
; chromium-browser 
cannot open path of the current working directory: Permission denied
; cd /tmp; chromium-browser
Sorry, home directories outside of /home are not currently supported. 
See https://forum.snapcraft.io/t/11209 for details.

On Ubuntu 20.04, Canonical delivers 'chromium-browser' (the open source version of Chrome) as a Snap, which is the jargon name for a 'Snappy', well, thing. The installed and visible /usr/bin/chromium-browser is just a shell script that flails around and eventually runs /snap/bin/chromium, which is a symlink to /usr/bin/snap, which magically winds up running Chromium as a sandboxed and oddly set up program. Or trying, at any rate; it fails, as you can see, because our long standing system configuration is incompatible with how Canonical requires you to operate your Ubuntu systems now.

(It would be nice if the whole system told you the core problem right away and saved you the initial attempts to make things work.)

Although the messages do not say this, the Snap system ignores any $HOME environment variable that you might have set; what it cares about is where /etc/passwd says your home directory is (after any symlinks are resolved). The official documentation says 'a workaround is to bind mount the home directory outside /home into /home'. We currently have roughly 2600 user home directories that are not under /home; instead they are under an assortment of filesystems that are NFS mounted from our Linux ZFS fileservers. Bind mounting all of their filesystems into /home or changing all of them to be NFS mounted there is what we can politely call not feasible, and certainly massively disruptive even if it was feasible.

Given this, I have to say that Canonical's Snappy system does not appear to be designed for anything except small environments, such as desktops and laptops used by one or a few people. It's certainly not viable in our larger (but still small scale) multi-user Unix environment. At the moment, we can probably live without Chromium in 20.04, although some of our users are going to be upset. If Canonical packages more important things as Snaps instead of actual packages, we will probably be forced to do something much more drastic.

(One quite likely option is migrating from Ubuntu LTS to Debian, who will probably never be this narrowly focused on small desktops or so willing to throw away compatibility with different environments.)

Ubuntu2004SnapsHomeIssue written at 21:18:02; Add Comment

2020-04-27

My views on SystemTap as compared to eBPF for Linux kernel instrumentation

In the small lobste.rs discussion on yesterday's entry looking back at DTrace from an eBPF world, jmtd noticed that there was no mention of SystemTap in my entry. SystemTap (also) was once the promising rough Linux equivalent of DTrace, but it's fallen out of favour. I used SystemTap a very little bit back when it was new (for example, to trace the system calls of a setuid program) but did nothing with it since then.

As a sysadmin, I felt that SystemTap had two or perhaps three significant issues. The lesser and questionable issue was that SystemTap was hard to get started with; getting all of the packages on to your system was a little bit intricate in the beginning and the documentation was discouragingly spotty both initially and somewhat later on when I tried it again in 2013. The big issues were that SystemTap was quite slow to start doing things and worked by the scary method of building and loading a new Linux kernel module into the kernel; I believe the kernel module then live-patched the running kernel code on the fly at least some of the time.

These two issues were interrelated; SystemTap was slow because it had to do a great deal of work to create and load that kernel module, and part of why it had to be slow was that a loadable kernel module is very dangerous, and SystemTap was trying to do it without blowing up your system. Because SystemTap used loadable kernel modules, any bugs in its runtime or how it translated your SystemTap script to kernel code could potentially crash your system, which was a scary thing to have to worry about on a production system. The very last thing you want to risk with an instrumentation system is that it makes your problems worse.

All of this effectively made SystemTap a last ditch option for us, something that we would have turned to only if a problem got really bad and it was worth risking crashing a production server in an attempt to diagnose it. In practice our problems never got that bad; we were always able to deal with them in less risky ways.

My perception of eBPF so far is that it is far more stable and officially supported. What eBPF can do is deliberately limited, as compared to the unlimited power of a loadable kernel module, and the core kernel itself works to make eBPF limited and safe. This includes otherwise potentially risky things like hooking into kernel functions with kprobes. This limited power and limited capability for damage makes me much more willing to use eBPF on production systems (once it gets fully available, which is not before Ubuntu 20.04 at the earliest for us). We'll have to test it, of course, but I consider eBPF to be much more intrinsically safe than a loadable kernel module that is supposed to not have any bugs.

(eBPF has had security bugs, which is why we've turned off user access to it, but I don't think it's had many stability issues. Partly I believe this is because its safety gets a lot of attention, since it's part of the core kernel and increasingly heavily used.)

SystemTapEBPFMyViews written at 00:25:55; Add Comment

2020-04-17

We've disabled eBPF for normal users on our Linux machines

In late March, there was a Linux kernel security issue, CVE-2020-8835 (also). This was, to quote the Ubuntu CVE description:

In the Linux kernel 5.5.0 and newer, the bpf verifier (kernel/bpf/verifier.c) did not properly restrict the register bounds for 32-bit operations, leading to out-of-bounds reads and writes in kernel memory. The vulnerability also affects the Linux 5.4 stable series, starting with v5.4.7, [...]

We were lucky. Because we run the standard Ubuntu 18.04 server kernels instead of the HWE kernels, we were not affected by this and did not have to go through a fleet wide kernel update and reboot while working from home.

You might wonder why an issue in eBPF creates a security vulnerability, as I did at the time. The answer is that these days, eBPF programs can be used by ordinary users on their own sockets (contrary to what the current bpf(2) manpage says; per the exploit writeup, this was a change added in kernel 4.4). This means that bugs in the eBPF verifier and JIT translation that allow bad stuff to slip through are fatal; the entire system's security rests on them being completely right all of the time.

Fortunately there is a kernel sysctl that controls whether normal users can use eBPF, kernel.unprivileged_bpf_disabled; if set to '1', normal users can't use eBPF. You can probably guess what we immediately did when this CVE came out, even though it didn't apply to us; we immediately pushed out a local update to all of our machines that turned this on, disabling user eBPF programs. Much like user namespaces (which kept enabling kernel security issues) and overlayfs, we feel that allowing user eBPF programs is too much risk right now for too little potential gain. Perhaps someday there will be a compelling usage for this, but for now we'd rather avoid a whole class of potential security issues.

(If anything has broken as a result of this, people haven't told us.)

DisablingUserEBPF written at 23:24:22; Add Comment

2020-04-08

Why you want a Linux bootloader even on UEFI systems

One of the reactions to my entry on the complexity of Linux bootloaders is to say that all of this is unnecessary on modern UEFI systems. On a UEFI system, you can just put all of your Linux kernel environments in the EFI system partition and give them (U)EFI boot menu entries to let the user choose between them. Unfortunately there are a number of problems with this on anything except the simplest systems, both in practice and in general.

On the practical level, many systems have a very limited number of UEFI boot menu entries, which means that you can only support a handful of kernels and boot environments (for things like a recovery option); how many you can support also goes down if you need a dual-boot system. Often people want to keep a decent number of kernels available, although arguably this is mostly historical at this point. More importantly, UEFI BIOSes have historically been not very happy to have boot entries updated all of the time. Boot entries are EFI variables, and UEFI BIOSes have traditionally assumed that EFI variables wouldn't be updated very often. In addition to the problem of constantly changing the NVRAM, BIOSes have had bugs and problems here, simply because churning through lots of changes is unusual behavior that Windows doesn't do.

One general issue is that you are putting things that change and that you need to be available into the EFI system partition, which is a FAT32 filesystem on a single disk. If you want to have a server with redundant disks that can survive (and reboot) after the total failure of one disk, you need to keep your two (or more) EFI system partitions in sync with each other, and also insure that the UEFI boot menu can use both of them somehow. Some people will tell you that you can use Linux software RAID mirroring here with the right magic; in practice, this is reportedly rather fragile (there are many exciting ways for the sides of the mirror to get out of sync with each other, since sometimes they are accessed outside of the software RAID) and may not work with any particular UEFI BIOS. Certainly it's not likely to be something that UEFI BIOS vendors either support or test.

Another issue is that UEFI boot menu entries don't necessarily support Linux specific features like modifying the kernel command line, which can be necessary in special circumstances. In addition to kernel parameters, a variety of Linux initramfs and init systems assume that you can modify kernel command line parameters to control their behavior, to turn on extra debugging messages or to start the system in single user mode or the like. You could do this with additional UEFI boot menu entries, but then you run into practical limits on how many a UEFI BIOS will support and how many people can sort through.

So while you can directly load Linux through UEFI with the kernel's EFISTUB support (also), the result is not up to the standards that people want and expect from a Linux boot environment, at least on servers. It might be okay for a Linux machine with a single disk where the user will only ever boot the most recent kernel or the second most recent one (in case the most recent one doesn't work), and if that doesn't work they'll boot from some alternate Linux live media or recovery media.

WhyBootloaderOnUEFI written at 21:41:25; Add Comment

2020-04-06

Fedora 31 fumbles DNF modules, package updates edition

I still run 'dnf check-update' every so often (despite what I wrote before), and for a while it has been reporting it had update info for 'rust-ripgrep' version 12.0.0 (aka the 'ripgrep' package), among other updates. I have ripgrep 11.0.2 installed and use ripgrep a fair bit, so an upgrade was (and is) of interest to me. However, 'dnf update' never offered to install such an update. Today I dug into what is going on, and it has left me unhappy.

Modern versions of Fedora use DNF modules (also), which serve as the source of various programs. When you update Fedora, at least in my way of doing it, some number of these modules are silently enabled based on what programs you have installed (and perhaps what versions). Modules come in multiple versions (you can have 'the latest' and 'version X' and so on), so my first thought was that ripgrep 12.0.0 was coming from a version of the ripgrep module that I didn't have enabled. However, 'dnf module list' said that there was only one ripgrep module, providing the latest version, and it was enabled (and it also only had one profile):

ripgrep    latest [d][e]   default [d]  [...]

I will skip to the conclusion, rather than going through my investigation steps (which involved things like looking at Bodhi and digging through a mirror of the actual Fedora updates repository). Fedora has ripgrep 12.0.0 in the main Fedora updates repository, but not in the 'ripgrep' DNF module that is supposed to have the latest version. In fact it has a number of modular packages that have been orphaned this way. On my system, all of gimp, libgit2, meson, and ripgrep have more recent versions in the main Fedora updates repository than in DNF modules.

(In libgit2's case, it appears that I somehow got locked into the libgit2 0.27 DNF module instead of the 0.28 one that is now current in Fedora's updates. Since I never explicitly chose a DNF module version, this is a problem in itself; simply having some version of libgit2 installed when DNF modularized my system shouldn't lock me into it.)

You can see what normal Fedora upgrades your DNF modules are blocking off from you with a magic flag that is barely documented (with large caution notes) in the dnf manpage:

dnf --setopt=updates.module_hotfixes=true check-update

Fixing this is what you would call more chancy. First, you want to revert the DNF modules back to a state where they are neither explicitly enabled nor explicitly disabled, with 'dnf module reset ..':

dnf module reset ripgrep libgit2 meson gimp

However, for me this by itself only unlocked the libgit2 package upgrades (after I did 'dnf clean metadata'). To upgrade the other packages, I had to resort to that dangerous dnf magic option again:

dnf --setopt=updates.module_hotfixes=true update

Because I am a cautious person, I re-ran the check-update command with the 'fedora' repository also overriding modules, which revealed that even from the moment Fedora 31 was released, some modules were behind the main versions. I started out fixing only the ninja-build package ('1.9.0-1' in its module, '1.9.0-3' in the Fedora 31 release repository) but wound up fixing all of them, which wound up with me having no DNF modules enabled at all and no RPMs installed from them. If you want to fix everything all at once, you want:

dnf --setopt=updates.module_hotfixes=true --setopt=fedora.module_hotfixes=true update

As a nice change, 'dnf updateinfo info' now has nothing to report, as I expect it to be on a fully up to date system.

(Sadly I suspect several of these DNF modules will magically spring back to life when I upgrade to Fedora 32, and I will have to go fix them all again.)

(This elaborates on some tweets.)

FedoraUpdatesModuleFailure written at 23:17:01; Add Comment

2020-04-05

My normal process for upgrading from one Fedora version to another

Sensible people have a normal, relatively standard Fedora configuration and upgrade from one Fedora version to another through one of the straightforward, fully supported methods. Neither of those are true for me, at least for my office and home desktops. The officially supported upgrade methods require taking your machine down for the duration of the upgrade, which is generally hours in my case, and I have an unusual system configuration (which goes beyond how updating Fedora kernels is a complicated affair for me). So I normally follow a much more complicated multi-stage process to upgrade my machines.

The first step is to wait while carefully reading the list of common bugs for the just released Fedora version. I never jump into a new Fedora version right away; I always give it at least a week or two to settle down before I even think about upgrading any of my machines, even unimportant ones. If there are any common bugs that look like they'll affect me, I hold off until they're fixed.

I upgrade my Fedora machines using 'dnf distro-sync', with more or less the process documented in Upgrading Fedora using package manager, because I want to watch over the upgrade process and keep using my machine during it and this is the only form of truly live upgrade that Fedora has. But before I upgrade either of my work or my home desktops, I upgrade a series of less important and generally less complicated other machines.

The first upgrade done is on a virtual machine that's a more or less stock Fedora install (with Cinnamon), generally one installed from scratch with the previous Fedora version. This tells me if the live upgrade process works in general and what sort of things happen on a stock system. After that I upgrade my work laptop, which has a pretty standard Cinnamon based install (although one that has been successively upgraded from Fedora version to Fedora version). If all goes well and I don't run into any problems when I use the new Fedora on my laptop for a while, it's time for the main systems.

Both of my desktops have additional risk factors for an upgrade, in that I use ZFS on Linux and Wireguard. I keep a Fedora virtual machine image with this same general setup, and so my first step in upgrading the main machines is to upgrade it and make sure that the result appears to work. Sometimes this requires manually rebuilding DKMS kernel modules after the 'dnf distro-sync' and other things, which is good to find out in advance. After it upgrades cleanly I will finally upgrade my work machine, sort out anything that's now broken in my non-standard desktop environment, and then at some point upgrade my home machine as well.

I do some extra steps in the actual upgrade process on each important machine (my laptop and my two desktops), and deviate from the standard directions. First, I always download packages separately from doing the actual upgrade:

dnf --downloadonly --releasever=NN --setopt=deltarpom=false distro-sync
dnf --releasever=NN --setopt=deltarpom=false distro-sync

The important thing about this is that it makes the start of the actual upgrade have a predictable timing. Downloading all of the required packages can take an unpredictable amount of time; the last thing I want is for the download to run long and box me in to launching into the upgrade at a bad time. It's also easier to sort out package conflicts in two stages.

I run both the package download and the package upgrade under script to capture all of the output, inside a screen session to try to keep things alive if there is a session problem. I live daringly and do this inside X, but I take a series of precautions. First, I always have several additional already open terminal windows, some of them su'd to root. Second, I have additional text mode root logins (and regular logins) on other virtual terminals. Finally, I generally have some SSH sessions from another machine (such as my laptop), one of which is also attached to the screen session doing the upgrade (with 'screen -x'). All of this is intended to give me access to the system if things go wrong; if I can't open new terminals, or can't now su to root, or the entire X server locks up, or whatever. And in the mean time I keep my normal X environment running so I can do productive work rather than have to sit on my hands for what may be hours.

(My Fedora upgrades are much faster now that I live in the world of SSDs and even NVMe drives, but in the past with HDs they could take three or four hours just to install and upgrade all of the packages.)

Generally everything goes smoothly except sorting out a few package dependency issues (usually solved by removing unimportant old packages) and then making my unusual desktop environment work completely again in the new Fedora. It's actually been a while since I had actual upgrade problems (either in the process as a whole or with individual packages having their postinstall or postuninstall scripts go wrong for some reason).

MyFedoraUpgradeProcess written at 22:26:39; Add Comment

2020-04-03

Microsoft Teams' bad arrogance on (Fedora) Linux

For reasons beyond the scope of this entry I'm going to need to use Microsoft Teams to participate in online video meetings every so often. Microsoft Teams has a 'public preview' for Linux, available (only) as a RPM or DEB package as far as I know (here). Because I would rather download an app to a somewhat expendable system (my work laptop) than go through the likely pain of getting video conferencing working in some browser, I tried it. It did not go entirely well.

My first tweet:

Oh how clever (not): the Microsoft Teams .rpm postinstall script queues an at job to install its GPG key into RPM ... using a temporary file name in /tmp that the key was written to by the postinstall script. -50 points; at least embed the GPG key in the at job.

Really, it does. Here's the relevant section of the script:

TMP=$(mktemp /tmp/teams.gpgsig.XXXXXX)
cat > "$TMP" <<KEY
[.. elided ..]
KEY
if [ -f "$TMP" ]
then
  service atd start
  echo "rpm --import $TMP" | at now + 2 minutes > /dev/null 2>&1
fi

As you might suspect, this isn't the only thing that the postinstall script does. It also adds an enabled Microsoft package repository to your system (requiring signatures, at least, which is why they have to add their key). It's not documented that they'll do this, they certainly don't ask, all of this is done on the fly so the relevant yum.repos.d file isn't in the RPM's manifest, and I don't believe they restrict what packages can be installed from their repository (although from its URL it appears to be specific to Teams).

(Another fun thing that the RPM does is that it puts the actual Teams binary and its shared library .so files in /usr/share/teams. I do not know how to break it to Microsoft, but that is not what goes in /usr/share. Also, it is of course an Electron app.)

I started Teams once to make sure that I could use it and thought no more. Then today I needed to use my laptop again for a different purpose, started up the laptop, and discovered another obnoxious and arrogant Teams behavior:

So Microsoft Teams on Fedora feels free to add itself to your desktop startup after you run it just once (perhaps to try it out) and even throws up a big window when you log in. That's some arrogance, Microsoft. Not even Zoom goes that far.

To be clear here, I explicitly quit out of Teams (including its little applet window that it leaves behind when you close its regular windows) the first time I used Teams. It was entirely shut down before I logged out, and it really did add itself to my Cinnamon desktop's startup. And when it started it didn't just post up its applet; it threw up a splash screen and then its large main window, shoving that in front of everything on the desktop.

This is especially arrogant given that Microsoft itself spent literally years systematically removing the ability of programs installed on Windows to automatically add themselves to the Start menu and I believe launch themselves when you logged in, because they found that programs abused that left and right. But here is Microsoft deciding that they themselves are exempt from these best practices of not automatically throwing people into random programs when they log in.

I would like to say that this is how you ruin a platform, but of course Microsoft doesn't care about ruining Linux as a platform. The tiger does not really change its stripes, regardless of what noises it may make for strategic reasons.

PS: Zoom's RPM postinstall and postuninstall scripts contain their own horrors, but I don't think any of them are quite as bad as Teams. And, as mentioned, Zoom refrains from starting itself on login.

(Please do not suggest Jitsi. I am not the person making these choices, especially the ones related to Teams.)

MicrosoftTeamsBadArrogance written at 22:08:24; Add Comment

2020-04-02

We may face some issues with the timing of Ubuntu 20.04 and its effects

If everything goes as listed in Ubuntu's 20.04 release schedule, Ubuntu 20.04 LTS will be released April 23rd, and this release will start a roughly one year countdown on security updates for Ubuntu 16.04, which would stop a year later in April of 2021. Under normal circumstances, we would spend a chunk of May bringing up 20.04 in our general Ubuntu install system, deploy some initial systems for testing, and then ideally start targeting an August upgrade of our user-facing 18.04 machines (which we try to keep on the current version of Ubuntu). We would also start working through our decent number of 16.04 machines in order to replace them well in advance of the support deadline (we have both 16.04 and 18.04 machines because of how we handle Ubuntu LTS versions).

These are not normal times. It seems unlikely that we'll be back in our office before July (cf), and while we can do some amount of work on 20.04 remotely (especially now that I can run VMWare Workstation from home), this only goes so far. We need to test 20.04 on real hardware before we deploy it, and we definitely can't remotely upgrade existing servers to 20.04. Also, we're unlikely to be able to immediately launch into new things the moment we're back in the office; there will probably be a backlog of problems to be dealt with first.

If everything goes back to normal around the start of July, we can almost certainly still squeeze everything in. We may have to push 20.04 upgrades for some machines to inconvenient times (for users and for us). In particular, we don't have that many 16.04 machines and so I don't think we'd have any real problem meeting the end of support deadline for it. However, that's the optimistic case. A more extended period out of the office would start to push at the 16.04 deadline, with us potentially looking at upgrading multiple critical machines (such as all of our mail servers) within the span of a few months at most (while we're dealing with many other issues).

If there is a general extended period of disruption all over, it's possible that Canonical will take the unusual step of extending free Ubuntu 16.04 support before the normal one year time after 20.04 is released. Apart from that, Canonical has already said that Extended Security Maintenance will be available for 16.04; we may be forced to buy that for crucial and hard to migrate machines that we can't get to in time, at least as a bridge to buy us extra time.

(Hopefully we could arrange something lower than Canonical's standard rates, which are rather expensive for a university department.)

Ubuntu2004TimingIssues written at 23:50:48; Add Comment

2020-04-01

Why Linux bootloaders wind up being complicated

I'm not really a fan of GRUB2 for a variety of reasons, but at the same time I have to acknowledge that a significant part of why it's so large and sprawling is simply that good, flexible bootloaders for Linux have a complicated job, especially if they're designed to work for traditional BIOS MBR booting as well as UEFI. However, if you haven't poked into this area it may not be obvious how a bootloader's job goes from looking simple to actually being complicated.

Here's what a modern bootloader like GRUB2 has to go through to boot Linux (which is only the start of the story of how modern Linux systems boot):

  • In a traditional BIOS environment, a very small initial boot stage must load the full bootloader, sometimes in multiple steps; this requires a fixed and known place to put at least some of the next stage of the bootloader. In a UEFI system, the UEFI BIOS will directly load and start the main bootloader code, sparing the bootloader from having to deal with this.

  • The bootloader reads a configuration file (possibly several configuration files) from somewhere. At a minimum, this configuration file will let the bootloader identify a bunch of kernel environments that it can potentially boot. For modern Linux, booting a kernel requires at least the kernel itself, its initial ramdisk, and the kernel command line arguments. If the bootloader supports the Boot Loader Specification (BLS), these kernel environments are each in separate files in a directory and have to all be inventoried and loaded one by one.

    In a UEFI environment, this configuration information can live on the EFI system partition and be loaded through the UEFI services provided by the BIOS. However, requiring the bootloader configuration to live in the EFI system partition causes problems. In a BIOS environment, a modern bootloader is expected to load the configuration from some regular filesystem (and see later about that). So for both UEFI and BIOS MBR booting, a good bootloader will be able to read its main configuration from some Linux filesystem.

    (A UEFI bootloader might sensibly start from an initial configuration file on the EFI system partition that tells it where to find the main configuration.)

  • Optionally, the bootloader presents you with some menu that lets you pick what you want to boot (either what kernel or what other operating system on the machine in multi-boot setups). Often you want to control how these boot entries are named and how many of them are shown at once (with options to hide more of them in sub-menus).

    Modern bootloaders are increasingly expected to not disturb whatever graphics and logo the system BIOS has left on the screen, or to disturb it as little as possible if they present a menu. The desire (although not necessarily the reality) is that you turn on the machine, see a logo that sits there with a progress spinner of some sort, and then the system login screen fades in, all without flashing or abrupt visual changes. This generally requires the bootloader to understand something about PC graphics modes and how to deal with them (although I think there may be some UEFI services that help you here if you're booting with UEFI).

    As a quality of life issue, the bootloader should also let you (the person booting the system) temporarily modify things like kernel command line parameters because you can wind up with ones that cause your kernel to hang.

  • The selected kernel and initial ramdisk are loaded into memory from disk and the bootloader transfers control to them, with its job done. Properly transferring control to a Linux kernel requires some magic setup that's specific to Linux (for example to tell the kernel where the initial ramdisk is in memory).

    As with the bootloader's configuration, a UEFI bootloader can in theory require that all the kernels and initial ramdisks live on the EFI system partition so that it can load them through standard UEFI system services. However this is even more inconvenient for people, so in practice a well regarded bootloader must support loading kernels and initial ramdisks from regular Linux filesystems, and in any case this is required if the bootloader wants to support BIOS MBR booting.

Loading kernels, initial ramdisks, perhaps configuration files, and more from regular Linux filesystems opens up a world of complexity, because in practice people want their bootloader to support the entire storage stack, not just filesystems. If you have ext4 filesystems in LVM on top of a mirrored or RAID-6 software RAID array, the bootloader has to understand all levels of that in order to be able to load things. If the bootloader doesn't have a fixed location for what it loads, you have to be able to specify on what filesystem they're found and what the storage stack looks like. Generally this goes in the configuration file, which requires a whole language for specifying the various levels of the storage stack (such as the UUIDs for the /boot or / filesystem, the RAID array, and so on, cf and also).

(Sometimes this causes issues when there are new filesystems and storage stacks people would like you to support, such as the challenges with booting ZFS from Linux with GRUB.)

Often the bootloader requires too much code for all of the filesystems, storage stacks, graphical options, and so on to build it all into the core bootloader (especially if you want the core bootloader to work with relatively little RAM). In this situation GRUB2 turns to loadable modules, which means that you also need a way to find and load those modules. A bootloader that works only on UEFI can once again put everything on the EFI system partition and load its modules with UEFI services, but otherwise the core bootloader needs enough built in code to find and load modules from at least some sort of filesystems and storage stacks.

(The bootloader may also want to load fonts, graphics, and so on in order to support showing menus and progress indicators in the graphics mode that the BIOS has left things in. And a bootloader may want a fancier menu than just a bunch of text.)

Bootloaders often want to support additional quality of life features like a flexible default for what kernel (or boot entry) is selected, one time booting of a particular kernel, configurable timeouts before the boot continues (either with or without the boot menu displayed), and especially some sort of recovery mode if things are broken. You don't really want your bootloader to fall over if the default kernel can't be loaded for some reason, although a simple bootloader can leave this out and tell you to boot from a recovery stick instead.

(Such a simple bootloader won't be popular with people running servers or anyone who wants unattended boots and reboots to be reliable even if things go a little wrong.)

A bootloader that wants to properly support UEFI Secure Boot needs to use UEFI services to verify the signatures on the kernel, any modules it loads, and so on. I believe that this is theoretically straightforward but can be complicated in practice.

(This entry is sparked by this and its link to this.)

BootloadersComplexity written at 23:30:06; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.