2021-05-03
Our future upgrade wave of Ubuntu 18.04 machines
We have long had a mix of Ubuntu versions. The short explanation is that most machines users log in to (our login servers and compute servers) get upgraded every LTS version, while other machines that are less accessible only get upgraded every other LTS (the longer version is How we handle Ubuntu LTS versions). Under normal circumstances, this would currently give us a relatively even mix of 20.04 machines and 18.04 machines. These aren't normal times.
The result of these abnormal times is that we have a lot more 18.04 machines and a lot fewer 20.04 machines than we normally would. None of our user login machines have been upgraded, our mail servers had to move to 18.04 instead of 20.04, and until late last year we lacked much experience with 20.04, so the path of least resistance was using 18.04 for new or upgraded machines because it was a known quantity. This is fine by itself, as Ubuntu 18.04 LTS is a perfectly good Ubuntu release.
Our future issue is that having a lot of 18.04 machines (some of them very critical ones) means that when Ubuntu 22.04 comes out next April, we'll have a lot of machines to upgrade in less than a year (since 18.04 will stop being supported at the end of April 2023). This is probably more unique machines than we've ever had to upgrade in one cycle, even if we assume that the machines users log in to are mostly simple to rebuild. Some of the machines, such as our fileservers, will take extensive testing all on their own.
If we get enough time in the office this summer we may try to upgrade our user login machines to 20.04, even though it's a year behind the usual schedule. We have a test user login machine built, although it hasn't seen much use, and that would let us slip their upgrade to 22.04 until late, perhaps even the summer of 2023.
Beyond that, we could upgrade some machines early, moving we normally wouldn't touch from 18.04 to 20.04 just so we have fewer to move to 22.04 later. This would also give us more of a spread between LTS versions for the long term; otherwise, if we just upgrade all of our 18.04 machines to 22.04 we'll have much the same problem in 2026, with a lot of 22.04 machines to suddenly move to 26.04.
I have no conclusions, but at least this is now an issue I'm going to be thinking about.
(I'm aware that in some places, planning for 2026 would be a laughable idea. It may be optimistic even for us, but I've had long term planning pay off before and in general we exist in an environment with long term stability, although there are somewhat more clouds on the horizon than usual.)
Understanding OpenSSH's future deprecation of the 'ssh-rsa' signature scheme
OpenSSH 8.6 was recently released, and its release notes have a 'future deprecation notice' as has every release since OpenSSH 8.2:
Future deprecation notice
It is now possible to perform chosen-prefix attacks against the SHA-1 algorithm for less than USD$50K.
In the SSH protocol, the "ssh-rsa" signature scheme uses the SHA-1 hash algorithm in conjunction with the RSA public key algorithm. OpenSSH will disable this signature scheme by default in the near future.
More or less a year ago I flailed around about what this meant. Now I think that I understand more about what is going on, enough so to talk about what is really affected and why. Helping this out is that since the OpenSSH 8.5 release notes, OpenSSH has had the current, more explicit wording above about the situation.
When we use public key cryptography to sign or encrypt something, we generally don't directly sign or encrypt the object itself. As covered in Soatok's Please Stop Encrypting with RSA Directly, for encryption we normally use public key encryption on a symmetric key that the message itself is encrypted with. For signing, we normally hash the message and then sign the hash (see, for instance, where cryptographic hashes come into TLS certificates). OpenSSH is no exception to this; it has both key types and key signature schemes (or algorithms), the latter of which specify the hash type to be used.
(OpenSSH's underlying key types are documented best in the ssh-keygen's
manpage for the -t option. The -sk keytypes are FIDO/U2F keys,
as mentioned in the OpenSSH 8.2 release notes. The supported key signature
algorithms can be seen with 'ssh -Q key-sig'.)
What OpenSSH is working to deprecate is the (sole) key signature algorithm that hashes messages to be signed with SHA-1, on the grounds that SHA-1 hashing is looking increasingly weak. For historical reasons, this key signature algorithm has the same name ('ssh-rsa') as a key type, which creates exciting grounds for misunderstandings, such as I had last year. Even after this deprecation, OpenSSH RSA keys will be usable as user and host keys, because OpenSSH has provided other key signature algorithms using RSA keys and stronger hashes (specifically SHA2-256 and SHA2-512, which are also known as just 'SHA-256' and 'SHA-512', see Wikipedia on SHA-2).
Most relatively modern systems support RSA-based key signature schemes other than just ssh-rsa. Older systems may not, especially if they're small or embedded systems using more minimal SSH implementations. Even if things like routers from big companies support key signature schemes beyond ssh-rsa, you may have to update their firmware, which is something that not everyone does and which may require support contracts and the like. Unfortunately, anything you want to connect to has to have a key signature scheme that you support, because otherwise you can't authenticate their host key.
(OpenSSH Ed25519 keys also have a single key signature scheme associated with them, if you ignore SSH certificates; they are both 'ssh-ed25519'. Hopefully we will never run into a similar hash weakness issue with them. Since I just looked it up in RFC 8709 and RFC 8032, ed25519 signatures use SHA2-512.)