Our upgrade wave of Ubuntu 18.04 machines has gone fine
Back in the early middle of 2021, I worried about our future upgrade wave of Ubuntu 18.04 machines because these weren't ordinary times and we weren't upgrading machines to Ubuntu 20.04. Ubuntu 22.04 came out as expected this April and started the countdown for Ubuntu 18.04's official support (which runs out next April), which also started our countdown to do something with all of our 18.04 machines. The good news (for us) is that upgrading our 18.04 machines is now mostly done and it wasn't particularly a problem. Looking back, I think this was due to a number of factors.
(Although our 18.04 to 22.04 upgrades haven't bean a problem, they did consume a bunch of our time and effort. I don't want to understate the amount of work my co-workers have put into developing and testing upgrades to various machines, and figuring out weird problems that ensued. Our upgrades to 22.04 haven't held us back but haven't entirely been trivial.)
First, we'd already been through Canonical's change in Ubuntu installers in 20.04, so we didn't have to customize that from scratch. Our 22.04 base install image needed almost no new local modifications, so it was easy to develop and was ready almost immediately when the official 22.04 images were released. Beyond that, it felt like there weren't all that many large changes between 20.04 and 22.04; much our local Ubuntu install system is unchanged or almost unchanged between 20.04 and 22.04.
Looked at numerically, a lot of our 18.04 machines were in our SLURM cluster. These machines are effectively cattle; once we figured out the general 18.04 to 22.04 SLURM upgrade path, doing them all was merely tedious. This left a collection of unique and somewhat unique servers, and even with them there was relatively little configuration and setup churn and change from 18.04 to 22.04.
(The install instructions for a lot of 22.04 versions are basically the 18.04 one with '18.04' changed to '22.04', and sometimes things deleted because they're not necessary any more.)
Some of this lack of churn is due to systemd. Systemd has essentially standardized init scripts, including across distribution versions, so many of the customizations we'd made in 18.04 and 20.04 dropped right in to 22.04 (a few of them weren't necessary any more; for instance, the 22.04 systemd doesn't seem to have the unmount storm during shutdown problem that 20.04 had). But it's also due to how major things like Apache haven't had a big release with major changes in years; both 18.04 and 22.04 use Apache 2.4, for example, so our Apache configurations needed basically no changes (although we aren going to try to make some.
We still have a few significant things left, like our fileservers and our IMAP server (which is also a fileserver these days for reasons outside the scope of this entry). But almost everything else is done and it doesn't feel like we've been having to rush it, and we already have a test 22.04 fileserver up and running (with basically no real changes in our setup needed from the 18.04 version, although the underlying versions of things like OpenZFS have changed for the better).
PS: Some of this is also from limiting our ambitions with 22.04, even when this meant dropping features from our regular installs. For example, Canonical's Snaps are incompatible with NFS mounted home directories and Ubuntu 22.04 only really supplies Firefox as a Snap, so our 22.04 machines don't have a system wide Firefox installed. We decided that this was okay.