Ubuntu 22.04 LTS servers and phased apt updates

January 13, 2023

I was working on getting one of our 22.04 LTS servers up to date, even for packages we normally hold, when I hit a mystery and posted about it on the Fediverse:

Why does apt on this 22.04 Ubuntu machine want to hold back a bunch of package updates even with '--with-new-pkgs --ignore-hold'? Who knows, it won't tell me why it doesn't like any or all of:

open-vm-tools openssh-client openssh-server openssh-sftp-server osinfo-db python3-software-properties software-properties-common

(Apt is not my favorite package manager for many reasons, this among them.)

Steve suggested that it was Ubuntu's "Phased Update" system, which is what it turned out to be. This set me off to do some investigations, and it turns out that phased (apt) updates explain some other anomalies we've seen with package updates on our Ubuntu 22.04 machines.

The basic idea of phased updates is explained in the "Phasing" section of Ubuntu's page on Stable Release Updates (SRUs); it's a progressive rollout of the package to more and more of the system base. Ubuntu introduced phased updates in 2013 (cf) but initially they weren't directly supported by apt, only by the desktop upgrade programs. Ubuntu 21.04 added apt support for phased updates and Ubuntu 22.04 LTS is thus the first LTS version to subject servers to phased updates. More explanations of phased updates are in this askubuntu answer, which includes one way to work around them.

(Note that as far as I know and have seen, security updates are not released as phased updates; if it's a security update, everyone gets it right away. Phased updates are only used for regular, non-security updates.)

Unfortunately apt (or apt-get) won't tell you if an update is being held back because of phasing. This user-hostile apt issue is tracked in Ubuntu bug #1988819 and you should add yourself as someone it affects if this is relevant to you. Ubuntu has a web page on what updates are currently in phased release, although packages are removed from this page once they reach 100%. Having reached 100%, such a package is no longer a phased update, which will become relevant soon. If you can't see a reason for a package to be held back, it's probably a phased update but you can check the page to be sure.

(As covered in the "Phasing" section, packages normally move forward through the phased rollout every six hours, so you can have a package held back on some server in the morning and then be not-held in the afternoon. This is great fun for troubleshooting why a given server didn't get a particular update.)

Your place in a phased update is randomized across both different servers and different packages. If you have a fleet of servers, they will get each phased update at different times, and the order won't be consistent from package to package. This explains an anomaly we've been seeing in our package updates for some time, where different 22.04 servers would get updates at different times without any consistent pattern.

The phased update related apt settings available and some of the technical details are mostly explained in this askubuntu answer. If you want to opt out of phased updates entirely, you have two options; you can have your servers install all phased updates right away (basically putting you at the 0% start line), or you can skip all phased updates and only install such packages when they reach 100% and stop being considered phased updates at all. Unfortunately, as of 22.04 there's no explicit option to set your servers to have a particular order within all updates (so that you can have, for example, a 'canary' server that always installs updates at 0% or 10%, ahead of the rest of the fleet).

For any given package update, machines are randomized based on the contents of /etc/machine-id, which can be overridden for apt by setting APT::Machine-ID to a 32 hex digit value of your choice (the current version of apt appears to only use the machine ID for phased updates). If you set this to the same value across your fleet, your fleet will update in sync (although not at a predictable point in the phase process); you can also set subsets of your fleet to different shared values so that the groups will update at different times. The assignment of a particular machine to a point in the phased rollout is done through a relatively straightforward approach; the package name, version, and machine ID are all combined into a seed for a random number generator, and then the random number generator is used to produce a 0 to 100 value, which is your position in the phased rollout. The inclusion of the package name and version means that a given machine ID will be at different positions in the phased update for different packages. All of this turns out to be officially documented in the "Phased Updates" section of apt_preferences(5), although not in much detail.

(There is a somewhat different mechanism for desktop updates, covered in the previously mentioned askubuntu answer.)

As far as I can see from looking at the current apt source code, apt doesn't log anything at any verbosity if it holds a package back because the package is a phased update and your machine doesn't qualify for it yet. The fact that a package was a phased update the last time apt looked may possibly be recorded in /var/log/apt/eipp.log.xz, but documentation on this file is sparse.

Now that I've looked at all of this and read about APT::Machine-ID, we'll probably set it to a single value across all of our fleet because we find different machines getting updates at different times to be confusing and annoying (and it potentially complicates troubleshooting problems that are reported to us, since we normally assume that all 22.04 machines have the same version of things like OpenSSH). If we could directly control the position within a phased rollout we'd probably set up some canary machines, but since we can't I don't think there's a strong reason to have more than one machine-id group of machines.

(We could set some very important machines to only get updates when packages reach 100% and stop being phased updates, but Ubuntu has a good record of not blowing things up with eg OpenSSH updates.)

Comments on this page:

By Stephen Kitt at 2023-01-14 04:12:47:

Administrators who rely on DHCP might want to avoid setting their machine-id to the same value across systems, at least not without also checking their netplan configuration, because the machine-id is the default DHCP differenciator in current Ubuntu releases. I don’t have the settings handy, and it is of course possible to configure netplan so that it uses the MAC address instead. Since the machine-id is supposed to be unique it’s probable that other tools will start relying on it too :-/.

By Stephen Kitt at 2023-01-14 04:14:44:

Ah, ignore the above, setting the apt machine-id as you describe isn’t the same as setting /etc/machine-id!

It especially frustrates me that the phased-update packages will appear in apt list —upgradable but then not install with apt upgrade. I thought something was seriously wrong the first few times I saw it.

Oh gosh, I was losing my mind getting different updates on servers with almost the same set of packages. I kind of forced the upgrades using aptitude, but they kept happening every so often.

Thanks for diagnosing it. I'll probably go the same way about setting a shared machine-id, as it bothers me that the updates are not deterministic for the entire fleet.

These are the things that should have been announced in the apt_news, instead of putting up the same "there will be news here" banner with every apt update.

By Simon at 2023-01-15 13:52:46:

Wouldn't it be better to set APT::Get::Never-Include-Phased-Updates or APT::Get::Always-Include-Phased-Updates instead of setting APT::Machine-ID to the same value across machines? The later sounds risky, since given the name I wouldn't be surprised that in the future something will rely on it being unique. Since "unique" IDs not being unique is always a risks (cloned images, backups, etc., etc.) I would hope APT never breaks something critical in this case, but still.

By K.C. Marshall at 2023-01-15 15:41:32:

I ran into this when some updates were reported as "kept back" and I could not find any half-configured or half-removed or conflicting packages on the system. After a while and some searching I realized it was a feature "helping" me.

I wonder if any problems with packages have been successfully identified during the phasing period and if the overall result is any different from just releasing an updated, corrected package in short order.

By cks at 2023-01-18 23:42:15:

Belatedly: I feel that setting APT::Machine-ID to the same value across machines is okay here because it's explicitly documented as being for this purpose (in the apt_preferences manual page). If in the future apt decides to reuse it for something else where a common value causes problems, that would be an actively incompatible change. I feel I'm entitled to assume that such changes won't happen, especially within a single Ubuntu LTS release.

As far as phased updates detecting problems goes, the status page for currently phased updates does seem to periodically show package rollouts that have hit problems and been reset to 0%. So far I think I've mostly seem then in desktop software.

Written on 13 January 2023.
« A browser tweak for system administrators doing (web) network debugging
Your server BMCs can need to be rebooted every so often »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Jan 13 22:56:18 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.