Wandering Thoughts archives

What affects automatically removing old kernels on Ubuntu

I have griped before (and recently) about how much of a pain it is to try to keep the number of kernels that Ubuntu installs on your machines under control. Writing your own script to remove obsolete kernels is fraught with challenges, but as it turns out I think we can do what we want with 'apt-get autoremove' and some extra work.

First, as Ewen McNeill said in a comment here back in 2015, it's the case that 'apt-get autoremove' will not remove a held package, kernel or otherwise. This makes a certain amount of sense, even if it's inconvenient. We can't keep kernels unheld in general for reasons covered here and here, but we probably can write a script that unholds them, runs 'apt-get autoremove', and holds the remaining kernels afterwards.

(Note that holding Ubuntu packages doesn't convert them from automatically installed packages to manually installed ones; it just holds them. You can see this with apt-mark, which also makes a handy way to hold and unhold packages on the command line.)

If you run apt-get autoremove with your kernel packages not held, you'll notice that it doesn't remove all of them. This naturally made me curious about what controlled this, and at least in Ubuntu the answer is in /etc/apt/apt.conf.d/01autoremove-kernels:

// DO NOT EDIT! File autogenerated by
// /etc/kernel/postinst.d/apt-auto-removal
APT::NeverAutoRemove
{
   "^linux-image-4\.4\.0-45-generic$";
   "^linux-image-4\.4\.0-53-generic$";
[...]

This contains a list of kernel packages and package regular expressions that should not be autoremoved; generally it's going to contain your two most recent kernels. As the comment says, it's (re)created by a script when kernel packages are installed and removed. This script, /etc/kernel/postinst.d/apt-auto-removal, starts with a comment that does a pretty good job of explaining what it wants to do:

Mark as not-for-autoremoval those kernel packages that are:

• the currently booted version
• the kernel version we've been called for
• the latest kernel version (as determined by debian version number)
• the second-latest kernel version

In the common case this results in two kernels saved (booted into the second-latest kernel, we install the latest kernel in an upgrade), but can save up to four. Kernel refers here to a distinct release, which can potentially be installed in multiple flavours counting as one kernel.

The second rule here implies that if you install an old kernel by hand for some reason, it will get added to the manual exclusion list. Well, added to the current manual exclusion list, since the list is rebuilt on at least every kernel install.

Now, there is a very important gotcha with this whole setup: this list of kernels to never autoremove is only recreated when kernel packages are installed or otherwise manipulated. When you run 'apt-get autoremove', there is nothing that specifically preserves the kernel you are actually running right then. Normally you're probably booted into one of the preserved kernels. But you might not be; if you have to boot back into an old version for some reason and you then run 'apt-get autoremove', as far as I can see it's entirely possible for this to remove your kernel right out from underneath you. Possibly autoremove has specific safeguards against this, but if so I don't see them mentioned in the manpage and there's also this Ubuntu bug.

(As a result, our wrapper script is likely to specifically hold or keep held the actual running kernel.)

(I got some of this information from this askubuntu question and its answers.)

PS: This suggests that maximum safety comes from writing your own script to explicitly work out what kernels you can remove based on local policy decisions. Using 'apt-get autoremove' will probably work much of the time, but it's the somewhat lazy way. We're lazy, though, so we'll probably use it.