DKMS kind of has a problem with its error messages
I generally like DKMS, I really do. Almost all of the time it provides a friction-free way for me to have ZFS on Linux in the face of both kernel upgrades (including upgrading Fedora distributions) and ZoL updates (which happen reasonably frequently, because I generally track the development version). But once in a while things don't go so well, and when this happens the error messages that DKMS spits out are not so helpful at identifying the cause (and how to fix things).
The latest issue is that recently, any time I upgraded ZoL,
the upgrade of the
zfs-dkms RPM would spit out an error like
[...] Upgrading : zfs-dkms-0.7.0-rc1_144_g24cdeaf.fc24.noarch Removing old zfs-0.7.0 DKMS files... -------- Uninstall Beginning -------- [...] DKMS: uninstall completed. Loading new zfs-0.7.0 DKMS files... Error! DKMS tree already contains: zfs-0.7.0 You cannot add the same module/version combo more than once. warning: %post(zfs-dkms-0.7.0-rc1_144_g24cdeaf.fc24.noarch) scriptlet failed, exit status 3 Non-fatal POSTIN scriptlet failure in rpm package zfs-dkms Non-fatal POSTIN scriptlet failure in rpm package zfs-dkms [...]
So, this error message creates several obvious questions here:
- Where is the DKMS tree that this is talking about?
/var/lib/dkms, but DKMS doesn't tell you and you'll have to search the manpage and so on fairly carefully to find out or guess.
- What specifically makes DKMS think that the tree already contains
The answer appears to be that
/var/lib/dkms/zfs/0.7.0/sourceexists as either a symlink or a directory. Here it's normally a symlink to
/usr/src/zfs-0.7.0(at least that what it is when things are stable, after I fixed DKMS's view of the world; I don't know what it was at the time of the error because DKMS didn't tell me). At least I think that's what it is.
You find out this answer by carefully reading bits of the large Bash shell script that is
/usr/sbin/dkms, which is is not something most people will do to troubleshoot problems.
Naturally this leads to the next question, which is why there was
/var/lib/dkms/zfs/0.7.0/source symlink after DKMS uninstall
had finished. A hint to the answer is lurking in output that is not
actually in this transcript, which should come right after the
DKMS: uninstall completed.' line:
------------------------------ Deleting module version: 0.7.0 completely from the DKMS tree. ------------------------------
The lack of this message means 'for some reason, there were still
/var/lib/dkms/zfs/0.7.0/ after DKMS
theoretically uninstalled everything, so DKMS didn't recursively
delete all of
/var/lib/dkms/zfs/0.7.0'. Or at least, there were
directories that didn't match the egrep pattern
(build|tarball|driver_disk|rpm|deb|source)$, because DKMS
explicitly excludes anything ending in those for this check. Again,
you find this out from the DKMS shell script.
(DKMS could perhaps warn you about this situation, and maybe even report what things are lingering. Maybe you don't want this all of the time, but DKMS is being run here in a special 'during RPM upgrades' mode and it can probably work out that something is bad here.)
In my case, this was a directory hierarchy called
4.7.4-200.fc24.x86_64, which was apparently a lingering remnant
of a much older version of the
zfs-dkms package on a much older
kernel. It's not clear how or why it wound up left there, floating
around and contaminating things, but having gone through this entire
tracing exercise I think that I'm reasonably confident that it was
causing my problems and removing it was the right answer.
(I'll only find out for sure the next time I upgrade my ZoL packages, which probably won't be for at least a day or two. I don't like to reboot my office workstation all the time, after all; it's nice to have a few days of stability.)
PS: I don't expect the DKMS script to ever get better errors. Why not? Well, it's a 3,882 line Bash shell script (on Fedora 24). Do you want to touch that if you don't absolutely have to? Yeah, that's what I thought. Probably the only way for it to get significantly better is to get completely rewritten in something other than Bash, and that would be a mistake because those almost 4,000 lines undoubtedly contain a lot of painful and hard-won experience about what works and what sort of terrible and peculiar things happen out in the world of people using DKMS. As usual, a from-scratch rewrite would get to rediscover all of those problems and so on for itself.
Link: Linux containers in 500 lines of code
Lizzie Dixon's Linux Containers in 500 lines of code is exactly what it says in the title, namely an example of setting up a relatively complete Linux container in 500 lines of code or so. The whole thing is interesting, but for me the really useful bits are the code examples and careful discussion of things like dropping capabilities and installing seccomp filters to block system calls.
(I feel I have a broad understanding of how Linux containers work and the general kernel pieces used to set them up, but I haven't really done anything with either capabilities or seccomp.)
Also, it turns out that reading the footnotes here was very
useful because it led to me discovering the Ubuntu sysctl
kernel.unprivileged_userns_clone, which lets you turn off
unprivileged use of user namespaces. I've wanted this for
some time so it's great to find
out that we can get it on most of our Ubuntu machines.
Why I'm interested in nftables, the theoretical Linux iptables replacement
Nftables probably first came to my attention in comments on my recent entry about how I don't understand Linux NAT as well as I should, although I may have heard of it in passing before then. At least in theory, it's the next generation replacement for iptables, which is broadly held to have some architectural issues and limitations. In practice, who knows, since nftables has been in progress for some time now (per the history section of the Wikipedia page). My current interest was sparked by running across Florian Westphal's recent blog entry introduction to nftables. Okay, particularly I got interested because of this statement:
The special 'route' type tells the netfilter core that it should perform a new route lookup after the packet has traversed the chain. [...] In the old iptables world, this property is hard-coded into the mangle table; in nftables, it is a base chain property.
I've had a certain amount of heartburn with the inflexibility of when iptables (re)routes packets, as discussed here. Being able to fix all of that is a clear win for nftables.
But that's not my only iptables problem. To be honest, my current set of iptables rules have slowly turned into a kind of a mess. Some of them are automatically generated, not particularly well, from a high level language I glued together a very long time ago because writing iptables rules by hand was tedious. Others have been glued on by hand since then, sometimes with hacks. And the whole collection is IPv4 only because making them work for IPv6 too requires too much changing things and manual duplication even though I would be happy to just replicate all the blocks over.
(This has led to a patchwork quilt of using various modern features like ipsets only in some situations. If I was redoing my iptables rules from scratch, I would make a lot more use of sets and rule chains than I did in the past. Past me didn't really get the advantage of putting things in custom chains and ipsets were years in the future when I wrote the initial collection.)
I've toyed with the idea of razing everything to the ground and rebuilding my rules with something like FireHOL, but as usual it seems like a lot of work to wind up back at the status quo. Razing everything to the ground and rebuilding it with nftables at least offers the appealing prospect that the result could be better than I started with; more featureful, easier to manipulate in various ways, and it would finally be able to easily do some things that I've always wanted to be able to make work.
(nftables seems to have quite powerful and general capabilities in the area of sets and (hash)maps, ones that seem to be in the area of OpenBSD pf's capabilities.)
Finally, I have to admit another motivation: I don't really like iptables. To make an analogy, iptables is basically firewall rule assembly language. You can do a lot in assembly once you know how, but writing it is always verbose and tedious because you have to spell out absolutely everything in detail. This assembly nature is why I wrote my program to auto-generate iptables rules from a higher level language way back when (and why I've kept it even as things became increasingly creaky and awkward). Nftables doesn't seem to be the kind of high level thing that OpenBSD pf is, but at least it has a bunch of powerful abstractions (including sets) that hold out promise to reduce the tedium and verbosity.
(My rule-generation program is so old that it originally generated ipchains rules.)