Wandering Thoughts archives

Some notes and considerations on SSH host key verification

Suppose, not entirely hypothetically, that you want to verify the SSH host keys of a server and that you're doing so with code that's reasonably under your control (instead of relying on, say, OpenSSH's ssh program). Then there are a number of things that you're going to want to think about because of how the SSH protocol works and how it interacts with security decisions.

The first thing to know is that you can only verify one type of host key in a single connection. As covered in RFC 4253 section 7.1, the client (you) and the server (the remote end) send each other a list of supported host key algorithms, and then the two of you pick one of the supported algorithms and verify the server's key in that algorithm. If you know multiple types of host keys for a server and you want to verify that the server knows all of them, you need to verify each type of key in a separate connection.

In theory, the client controls the preference order of the SSH host key types; you can say that you prefer ed25519 keys to RSA keys and the server should send its ed25519 key instead of its RSA key. In practice, a server can get away with sending you any type of host key that you said you'd accept, even if it's not your preference, because a server is allowed to claim that it doesn't have your preferred sort of host key (but good servers should be obedient to your wishes, because that's what the protocol requires). As a result, if you're verifying host keys you have a security decision to make: are you willing to accept any type of host key you have on file, or if you have your preferred type of host key on file, do you insist that the server present that type of key?

To be concrete, suppose that you have ed25519 and RSA keys for a server, you prefer ed25519 keys, and when you try to verify the server it offers you its RSA key instead of its ed25519 key. You could reject this on the grounds that either the server does not have the ed25519 key it should or that it's not following the protocol specification, or you could accept it because the server has a SSH host key that you have on file for it.

(As far as I can tell, OpenSSH's ssh command behaves the second way; it'll accept an RSA key even if you also have an ed25519 key for the server in your known_hosts.)

If you pick the first approach, you want to configure your SSH connection to the server to only accept a single key type, that being the best key type you have on file for the server. If you pick the second approach, you'll want to list all key types you have, in preference order (I prefer ed25519 to RSA and skip (EC)DSA keys entirely, while the current OpenSSH ssh_config manpage prefers ECDSA to ed25519 to RSA).

Under normal circumstances, the server will present only a single host key to be verified (and it certainly can only present a single type of key). This means that if you reject the initial host key the server presents, you will never be called on to verify another type of host key. If the server presents an ed25519 key and you reject it, you'll never get asked to verify an RSA key; the connection just fails. If you wanted to fall back to checking the RSA key in this case, you would have to make a second connection (during which you would only ask for RSA keys). In other words, if the server presents a key it must be correct. With straightforward code, your condition is not 'the server passes if it can eventually present any key that you know', your condition is 'the server passes if the first and only key it presents is one you know'.

PS: If you want to match the behavior of OpenSSH's ssh command, I think you're going to need to do some experimentation with how it actually behaves in various situations. I'm sure that I don't fully understand it myself. Also, you don't necessarily want to imitate ssh here; it doesn't necessarily make the most secure choices. For instance, ssh will happily accept a known_hosts file where a server has multiple keys of a given type, and pass the server if it presents a key that matches any one of them.

Sidebar: How a server might not know some of its host keys

The short version is re-provisioning servers. If you generate or record a server's host key of a given type, you need to also make sure that the server is (re-)provisioned with that key when it gets set up. If you miss a key type, you'll wind up with the server generating and presenting a new key of that type. This has happened to us every so often; for example, we missed properly re-provisioning ed25519 keys on Ubuntu 14.04 machines for a while.

My new Linux office workstation for fall 2017

My past two generations of office Linux desktops have been identical to my home machines, and when I wrote up my planned new home machine I expected that to be the case for my next work machine as well (we have some spare money and my work machine is six years old, so replacing it was always in the plans). It turns out that this is not going to be the case this time around; to my surprise and for reasons beyond the scope of this entry, my next office machine is going to be AMD Ryzen based.

(It turns out that I was wrong on how long it's been since I used AMD CPUs. My current desktop is Intel, but my previous 2006-era desktop was AMD based.)

The definitive parts list for this machine is as follows. Much of it is based on my planned new home machine, but obviously the switch from Intel to AMD required some other changes, some of which are irritating ones.

AMD Ryzen 1800X

Even though we're not going to overclock it, this is still the best Ryzen CPU. I figure that I can live with the 95W TDP and the cooling it requires, since that's what my current desktop has (and this time I'm getting a better CPU cooler than the stock Intel one, so it should run both cooler and quieter).

ASUS Prime X370-Pro motherboard

We recently got another Ryzen-based machine with this motherboard and it seems fine (as a CPU/GPU compute server). The motherboard has a decent assortment of SATA ports, USB, and so on, and really there's not much to say about it. I also looked at the slightly less expensive X370-A, but the X370-Pro has more than enough improvements to strongly prefer it (including two more SATA ports and onboard Intel-based networking instead of Realtek-based).

It does come with built in colourful LED lighting, which looks a bit odd in the machine in our server room. I'll live with it.

(This motherboard is mostly an improvement on the Intel version since it has more SATA ports, although I believe it has one less M.2 NVME port. But with two x16 PCIE slots, you can fix that with an add-on card.)

2x16 GB DDR4-2400 Kingston ECC ValueRAM

Two DIMMs is what you want on Ryzens today. We're using ECC RAM basically because we can; it's available and is only a bit more expensive than non-ECC RAM, runs fast enough, and is supported to at least some degree by the motherboard. We don't know if it will correct any errors, but probably it will.

(You can't get single-rank 16GB DIMMs, so that this ECC RAM is double-rank is not a drawback.)

The RAM speed issues with Ryzen is one of the irritations of building this machine around an AMD CPU instead of an Intel one. It may never be upgraded to 64 GB RAM over its lifetime (which will probably be at least five years).

Noctua NH-U12-SE-AM4 CPU cooler

We need some cooler for the Ryzen 1800X (since it doesn't come with one). These are well reviewed as both effective and quiet, and the first Ryzen machine we got has a Noctua cooler as well (although a different one).

Gigabyte Radeon RX 550 2GB video card

That I need a graphics card is one of the irritations of Ryzens. Needing a discrete graphics card means an AMD/ATI card right now, and I wanted one with a reasonably modern graphics architecture (and I needed one with at least two digital video outputs, since I have dual monitors). I sort of threw darts here, but reviewers seem to say that this card should be quiet under normal use.

As a Linux user I don't normally stress my graphics, but I expect to have to run Wayland by the end of the lifetime of this machine and I suspect that it will want something better than a vintage 2011 chipset. A modern Intel integrated GPU would likely have been fine, but Ryzens don't have integrated graphics so I have to go with a separate card.

(The Prime X370-Pro has onboard HDMI and DisplayPort connectors, but a footnote in the specifications notes that they only do anything if you have an Athlon CPU with integrated graphics. This disappointed me when I read it carefully, because at first I thought I was going to get to skip a separate video card.)

EVGA SuperNOVA G3 550W PSU

Commentary on my planned home machine pushed me to a better PSU than I initially put in that machine's parts list. Going to 550W buys me some margin for increased power needs for things like a more powerful GPU, if I ever need it.

(There are vaguely plausible reasons I might want to temporarily put in a GPU capable of running things like CUDA or Tensorflow. Some day we may need to know more about them than we currently do, since our researchers are increasingly interested in GPU computing.)

Fractal Design Define R5 case

All of the reasons I originally had for my home machine apply just as much for my work machine. I'm actively looking forward to having enough drive bays (and SATA ports) to temporarily throw hard drives into my case for testing purposes.

LG GH24NSC0 DVD/CD Writer

This is an indulgence, but it's an inexpensive one, I do actually burn DVDs at work every so often, and the motherboard has 8 SATA ports so I can actually connect this up all the time.

Unlike my still-theoretical new home machine (which is now unlikely to materialize before the start of next year at the earliest), the parts for my new office machine have all been ordered, so this is final. We're going to assemble it ourselves (by which I mean that I'm going to, possibly with some assistance from my co-workers if I run into problems).

On the bright side of not doing anything about a new home machine, now I'm going to get experience with a bunch of the parts I was planning to use in it (and with assembling a modern PC). If I decide I dislike the case or whatever for some reason, well, now I can look for another one.

(However, there's not much chance that I'll change my mind on using an Intel CPU in my new home machine even if this AMD-based one goes well. The 1800X is a more expensive CPU, although not as much so as I was expecting, and then there's the need for a GPU and the whole issues with memory and so on. Plus I remain more interested in single-thread CPU performance in my home usage. Still, I could wind up surprising myself here, especially if ECC turns out to be genuinely useful. Genuinely useful ECC would be a bit disturbing, of course, since that implies that I'd be seeing single-bit RAM errors far more than I think I should be.)