I recently tweeted:

Today I discovered that DKMS has spent some time silently (re)building one of my modules for the wrong kernel because DKMS. Naturally it didn't work. Since it was my sensor monitoring, I didn't notice for a while.

There is a complicated story here. This happened on my office workstation, which needs a very out of tree version of the it87 module in order to read the motherboard sensors. Because of ongoing problems in the 5.11 kernel series with my Radeon RX 550 card, I've been repeatedly upgrading my kernel to the latest Fedora 5.11.x, finding out that the kernel is no good, and falling back to the last-good kernel, which is Fedora's 5.10.23.

Recently (while in the default state of running 5.10.23), I noticed that I didn't have my usual motherboard sensor readings. Examining kernel messages for it87 problems, I immediately found the smoking gun:

[Fri Apr 23 15:17:51 2021] it87: version magic '5.11.15-200.fc33.x86_64 SMP mod_unload ' should be '5.10.23-200.fc33.x86_64 SMP mod_unload '

At that point I had 5.11.15 installed, making it the highest-version kernel, but I was running 5.10.23. This should be a supported system configuration but apparently DKMS somehow installed the 5.11.15 version of it87 (built when I installed that version) into 5.10.23's module area as well as 5.11.15's. So I told DKMS to remove the module and rebuild it. Surprise:

[Sat May  8 17:43:45 2021] it87: version magic '5.11.15-200.fc33.x86_64 SMP mod_unload ' should be '5.10.23-200.fc33.x86_64 SMP mod_unload '

Since 'dkms status' showed that the it87 module was removed before I had DKMS rebuilt it, DKMS knew what the correct kernel version was (because it installed the new it87 module there), but it rebuilt it for the wrong kernel version. What I wound up doing was removing the 5.11.15 RPMs entirely. This finally made DKMS build the module right. Possibly I could also have made DKMS work right by explicitly specifying the kernel version in 'dkms build' and 'dkms install'.

In the future I'm probably going to explicitly specify the kernel version for every DKMS build and install, even if it's the current running kernel. I'm also going to have to check that DKMS installed modules are for the right kernel, especially if I'm building them in some unusual situation. And obviously I now have a mental note to check that all my sensors still work after every reboot.

Somewhat to my surprise, DKMS is actively maintained in the DKMS git repository. But it is still a 3,935 line Bash script (which is up slightly from 2016). It's really a marvel that it works as well as it does, but on the other hand it's somewhat terrifying that so many Linux systems depend on it working reliably.

(One of the fun things about using DKMS on Fedora is that it rebuilds the initramfs for every installed kernel every time you install a DKMS module, regardless of which kernel you're installing the module into and whether or not the module would be included in the initramfs. This takes a substantial amount of time and there's no way to turn it off.)

Update: This turns out to be a significant Fedora issue instead of a DKMS bug, although DKMS could do more to defend against it (since DKMS knows a particular feature can't work on Fedora).