How Ubuntu and Fedora each do kernel packages

April 7, 2015

I feel the need to say things about the Ubuntu (and I believe Debian) kernel update process, but before I do that I want to write down how kernel packages look on Ubuntu and Fedora from a sysadmin's perspective because I think a number of people have only been exposed to one or the other. The Fedora approach to kernel packages is also used by Red Hat Enterprise Linux (and CentOS) and probably other Linux distributions that use yum and RPMs. I believe that the Ubuntu approach is also used by Debian, but maybe Debian does it a bit differently; I haven't run a real Debian system.

Both debs and RPMs have the core concepts of a package having a name, an upstream version number, and a distribution release number. For instance, Firefox on my Fedora 21 machine is currently firefox, upstream version 37.0, and release 2.fc21 (increasingly people embed the distribution version in the release number for reasons beyond the scope of this entry).

On Fedora you have some number of kernel-... RPMs installed at once. These are generally all instance of the kernel package (the package name); they differ only in their upstream version number and their release number. Yum normally keeps the most recent five of them for you, deleting the oldest when you add a new one via a 'yum upgrade' when a new version of the kernel package is available. This gives you a list of main kernel packages that looks like this:

kernel-3.18.8-201.fc21.x86_64
kernel-3.18.9-200.fc21.x86_64
kernel-3.19.1-201.fc21.x86_64
kernel-3.19.2-201.fc21.x86_64
kernel-3.19.3-200.fc21.x86_64

Here the kernel RPM with upstream version 3.19.3 and Fedora release version 200.fc21 is the most recent kernel I have installed (and this is a 64-bit machine as shown by the x86_64 architecture).

(This is a slight simplification. On Fedora 21, the kernel is actually split into three kernel packages: kernel, kernel-core, and kernel-modules. The kernel package for a specific version is just a meta-package that depends (through a bit of magic) on its associated kernel-core and kernel-modules packages. Yum knows how to manage all of this so you keep five copies not only of the kernel meta-package but also of the kernel-core and kernel-modules packages and so on. Mostly you can ignore the sub-packages in Fedora; I often forget about them. In RHEL up through RHEL 7, they don't exist and their contents are just part of the kernel package; the same was true of older Fedora versions.)

Ubuntu is more complicated. There is a single linux-image-generic (meta-)package installed on your system and then some number of packages with the package name of linux-image-<version>-<release>-generic for various <version> and <release> values. Each of these packages has a deb upstream version of <version> and a release version of <release>.<number>, where the number varies depending on how Ubuntu built things. Each specific linux-image-generic package version depends on a particular linux-image-<v>-<r>-generic package, so when you update to it it pulls in that specific kernel (at whatever the latest package release of it is).

Because of all of this, Ubuntu systems wind up with multiple kernels installed at once by the side effects of updating linux-image-generic. A new package version of l-i-g will depend on and pull in an entirely new linux-image-<v>-<r>-generic package, leaving the old linux-image-*-generic packages just sitting there. Unlike with yum, nothing in plain apt-get limits how many old kernels you have sitting around; if you leave your server alone, you'll wind up with copies of all kernel packages you've ever used. As far as the Ubuntu package system sees it, these are not multiple versions of the same thing but entirely separate packages, each of which you have only one version of.

This gives you a list of packages that looks like this (splitting apart the package name and the version plus Ubuntu release, what 'dpkg -l' calls Name and Version):

linux-image-3.13.0-24-generic   3.13.0-24.47
linux-image-3.13.0-45-generic   3.13.0-45.74
linux-image-3.13.0-46-generic   3.13.0-46.79
linux-image-3.13.0-48-generic   3.13.0-48.80

linux-image-generic             3.13.0.48.55

(I'm simplifying again; on Ubuntu 14.04 there are also linux-image-extra-<v>-<r>-generic packages.)

On this system, the current 3.13.0.48.55 version of linux-image-generic depends on and thus requires the linux-image-3.13.0-48-generic package, which is currently 'at' the nominal upstream version 3.13.0 and Ubuntu release 48.80. Past Ubuntu versions of linux-image-generic depended on the other linux-image-*-generic packages and caused them to be installed at the time.

I find the Fedora/RHEL approach to be much more straightforward than the Ubuntu approach. With Fedora, you just have N versions of the kernel package installed at once; done. With Ubuntu, you don't really have multiple versions of any given package installed; you just have a lot of confusingly named packages, each of which has one version installed, and these packages get installed on your system as a side effect of upgrading another package (linux-image-generic). As far as I know the Ubuntu package system doesn't know that all of these different named packages are variants of the same thing.

(A discussion of some unfortunate consequences of this Ubuntu decision is beyond the scope of this entry. See also.)

Sidebar: kernel variants

Both Ubuntu and Fedora have some variants of the kernel; for instance, Fedora has a PAE variant of their 32-bit x86 kernel. On Fedora, these get a different package name, kernel-pae, and everything else works in the same way as for normal kernels (and you have both PAE and regular kernels installed at the same time; yum will keep the most recent five of each).

On Ubuntu I believe these get a different meta-package that replaces linux-image-generic, for example linux-image-lowlatency, and versions of this package depend on specific kernel packages with different names, like linux-image-<v>-<r>-lowlatency. You can see the collection with 'apt-cache search linux-image'.

Both Fedora and Ubuntu have changed how they handled kernel variants over time; my memory is that Ubuntu had to change more in order to become more sane. Today their handling of variants strikes me as reasonably close to each other.


Comments on this page:

By dozzie at 2015-04-07 08:10:32:

In Debian stable you only get one kernel release, so there's no point in keeping N most recent ones. Debian testing and unstable, where kernel upgrades can happen, were never meant for casual use, so there's little optimization around their specific needs.

By cks at 2015-04-07 08:23:53:

My understanding is that Debian stable most definitely gets multiple kernels released for it, due to kernel security updates. These do not change the upstream kernel version but they do change the (abstract) Debian release number in the same way that Ubuntu changes theirs (note that all of the Ubuntu updates here are nominally the same upstream version, 3.13.0). And just as with Ubuntu (and RHEL), you definitely want the ability to keep multiple kernels around for several reasons and as far as I know Debian supports that.

(Your life is actually difficult if you don't support having multiple kernels installed at once because of kernel module loading in the face of internal kernel ABI versioning, so I can't imagine any Linux distribution not supporting it in some way.)

By dozzie at 2015-04-07 10:19:42:

@cks:

My understanding is that Debian stable most definitely gets multiple kernels released for it, due to kernel security updates. These do not change the upstream kernel version but they do change the (abstract) Debian release number

Yes, of course. But next version lands in the very same path as the previous one (/boot/vmlinuz-3.2.0-4-amd64 with /lib/modules/3.2.0-4-amd64 for modules for current kernel, package linux-image-3.2.0-4-amd64 with version 3.2.65-1+deb7u2). Thus, you end up with only one kernel on a typical machine.

You can have multiple kernels in Debian, it just happens that Debian only provides two kernel packages (linux-image-*-$arch and linux-image-*-rt-$arch), of which none is an upgrade to the other.

By cks at 2015-04-07 10:48:30:

@dozzie: .. oh. I did not realize that Debian overwrote kernels on updates. As a sysadmin, my reaction is that that is utterly terrible behavior for any number of reasons. The Ubuntu version at least only does that rarely; for Debian to do that all of the time is terrifying.

By Ewen McNeill at 2015-04-07 16:14:27:

My impression, having run both Debian systems and Ubuntu systems for years (Debian for well over a decade; Ubuntu for at least 5 years) is that both will bump a "ABI" version number on the kernel packages at times, but Debian seems to only bump it on calculated ABI incompatibility, whereas Ubuntu bumps it every kernel package release. The result is that on Debian you typically end up with a tiny handful of kernel package names in a stable release, and in Ubuntu it seems like you end up with at least one new kernel package name per month (if not more). (Looking at those two links, which Google turned up, it appears in theory the same criteria are used to bump the ABI version, but in practice it seems to change infrequently on Debian and constantly on Ubuntu.)

So, eg, a typical Debian system I have (which has been upgraded through several releases, point updates, and security updates) has:

ewen@mcneill:~$ dpkg -l | awk '/linux-image/ { print $2;}'
linux-image-2.6-686
linux-image-2.6.26-2-686
linux-image-2.6.32-5-686
linux-image-3.2.0-4-686-pae
linux-image-686-pae
ewen@mcneill:~$ 

whereas a typical Ubuntu system I have has a laundry list of kernel packages:

ewen@naosdell:~$ dpkg -l | awk '/linux-image/ { print $2;}'
linux-image-2.6.32-21-server
linux-image-2.6.32-25-server
linux-image-2.6.32-26-server
linux-image-2.6.32-27-server
linux-image-2.6.32-30-server
linux-image-2.6.32-33-server
linux-image-2.6.32-34-server
linux-image-2.6.32-39-server
linux-image-2.6.32-41-server
linux-image-2.6.32-45-server
linux-image-2.6.32-46-server
linux-image-2.6.32-47-server
linux-image-2.6.32-48-server
linux-image-2.6.32-54-server
linux-image-2.6.32-57-server
linux-image-2.6.32-58-server
linux-image-2.6.32-65-server
linux-image-server
ewen@naosdell:~$ 

(which is literally only Ubuntu 10.04 LTS security updates, and not even all of them, as that's a system I install them by hand and reboot infrequently.)

Yes, this does mean that a typical, minor, Debian security update will overwrite the kernel package files, with improved files. But that's also what happens with every other package on the system when you apply a security update, so it doesn't seem a terrible idea to me providing it is truly binary compatible. (Lots of kernel security updates affect only the internals of a single function, or small number of functions -- eg fixing a range check. I do still tend to hold off installing kernel updates until I'm planning on rebooting though, but don't panic about doing it instantly.)

Incidentally this means that Ubuntu and auto-install of security updates eventually fills /boot, leading to bad things to happen -- which usually won't happen on Debian. In recent (Ubuntu 14.04 LTS, and a couple of slightly earlier versions), doing "sudo apt-get --purge autoremove" is safe, and will always keep the running kernel version; in older Ubuntu versions you need to clear up carefully by hand ensuring your running version won't be removed (and/or that the running version is the latest installed version, as the latest version is always safe from auto-remove because of the meta package dependency). I ended up writing a python script to do the tidying up to deal with the fallout of those auto-updates, and attempt to be safe on all the current Ubuntu LTS releases.

Ewen

By Ben Hutchings at 2015-04-09 14:20:22:

I'm a Debian kernel maintainer.

We currently try very hard to avoid ABI bumps during a stable cycle because this requires rebuilding out-of-tree modules. For the last two stable releases we have been able to keep the kernel module ABI compatible, at least for those symbols we think are actually used by out-of-tree modules.

Ubuntu used to have the problem of not limiting how many differently named kernel packages were installed at once, thus /boot could fill up. More recently, APT was changed so that kernel packages that are automatically installed are also marked as auto-removable except for (I think) the running kernel and the two highest versioned installed kernels. So if linux-image-generic depended on linux-image-3.19.0-12-generic and caused the latter to be installed, but it no longer does, the package manager can treat the latter as no longer needed (e.g. if you run 'apt-get autoremove'). This change is also included in Debian.

Maintaining the kernel module ABI in Debian takes a fair amount of time that could be better spent on other things, so we may well move to something like the Ubuntu model in Debian 9 (stretch).

Written on 07 April 2015.
« What adblockers block
Your entire download infrastructure needs to use HTTPS »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue Apr 7 01:24:19 2015
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.