Wandering Thoughts: Recent Entries For 2008/07/24

How packaging systems should handle kernel updates

As a system administrator, I have some relatively strong opinions on how Linux distributions and packaging systems should handle kernel updates. Since I have just gone through the experience of yet another mass kernel update of our servers, I feel like writing them up:

• you should be able to have multiple kernels installed at once.
• it should be easy to tell which package's kernel you are running, and thus whether or not you are running the most current kernel.
• installing a kernel update should never overwrite an existing kernel image; it should always install a new kernel.

  (The one time when it is marginally acceptable to overwrite an existing kernel image is when the package update has literally no changes in the kernel; all it does is fix some packaging mistake. But that should be vanishingly rare anyways.)

• old kernels should get removed sooner or later, at a rate that is configurable but that has a sensible default. I do not need every kernel ever released for my distribution sitting around on my disks (especially if they have security holes).

  (Ideally there would be some sort of time-based minimum expiry, so I can say 'never remove a kernel until I haven't run it for a month'.)

• you should be able to 'pin' any particular kernel so that it is never removed. You really want this if you have a 'last known good' kernel and you want to keep experimenting with kernel updates to see if your distribution got it fixed this time around.

• the running kernel and its modules should never, ever get removed (or overwritten).

I don't know of any distribution that gets all of these right. Ubuntu fails spectacularly at never overwriting existing kernels; Red Hat has no good way to pin a particular kernel version so that it won't get removed (unless the yum versionlock plugin has very recently fixed its interaction with the 'keep the most recent N' plugin). No one has good kernel expiry.

Oh, and I prefer as much of this as possible to be in the kernel packages themselves, instead of in tools that are used to manage them, so that I cannot accidentally cause problems by using the wrong package management tool. This means that I somewhat prefer the Debian/Ubuntu approach of making different kernel versions have different package names to the Red Hat approach of making different kernel versions be different versions of the same package and putting the smarts in yum and other tools.

One thing that I dislike about typical debuggers

One of the things that I hate about typical debuggers is that they want me to hold their hand all the time. Okay, this is not entirely fair; they default to having me hold their hand, and generally don't do a really good job of supporting hands-off operation.

This is not really their fault, because debuggers have a different view of how they're going to be used than I do. A classical debugger (like gdb) expects to be used by people who are carefully narrowing in on the bug, taking cautious step after cautious step, and for this of course you want to interact with the debugger all the time (because ultimately you are looking for an anomaly, which requires a human's eye).

I don't use debuggers that way. Instead, I use them to extract a bunch of information from the program which I will then stare at for a while, so I want to set things up (simply, please) and then fire off a run without further intervention. The last thing I want to do is to slow down the process by having to interact with the debugger all the time.

Modern debuggers sort of support this; you can write command sets for breakpoints and so on that automatically dump information and then continue. But my impression is that both the command language and the interface is awkward, and in practice when I've tried to use gdb for this sort of thing I've wound up getting lost in the complexity of managing all of the breakpoints and so on by hand. I would actually rather have a script-driven debugger where I had to edit a file and then reload it to change what I was monitoring, because that's easier to keep track of and to modify.

(Having to keep changing a file and reloading it would of course be intolerable for the step by step 'narrowing in' style of debugging, which is why I don't expect debuggers to support it any time soon.)