TLS server certificate verification has two parts (and some consequences)
One of the unusual and sometimes troublesome parts of TLS is that verifying a TLS server's certificate actually has two separate parts, each critical. The first part is verifying that you have a valid certificate, one that is signed by a certificate chain that runs up to a known CA, hasn't expired, hasn't been revoked (or is asserted as valid), perhaps appears in a CT log, and so on. The second, equally critical part is making sure that this valid certificate is actually for the server you are talking to, because there are a lot of valid certificates out there and more or less anyone can get one for some name. Failing to do this opens you up to an obvious and often trivial set of impersonation attacks.
However, there is an important consequence of this for using TLS outside of the web, which is that you must know the name of the server you're supposed to be talking to in order to verify a server's TLS certificate. On the web, the server name is burned into the URL; you cannot make a request without knowing it (even if you know it as an IP address). In other protocols that also use TLS, this may not be true or it may not be clear what name for the server you should use (if there are levels of aliases, redirections, and so on going on, possibly including DNS CNAMEs).
The corollary of this is that it's much harder to use TLS with a
protocol that doesn't
give you start with a server
name somehow. If the protocol is 'I broadcast a status packet and
something responds' or 'someone gives me a list of IP addresses of
resources', you sort of have a problem. Sometimes you can resolve
this problem by fiat, for example by saying 'we will do a DNS PTR
query to resolve this IP address to a name and then use the name',
and sometimes you can't even get that far.
(You can also say 'we will not verify the server name', but then you only have part of TLS.)
That's all very abstract, so let's go with two real examples. The first one is 801.2X network authentication, which I tangled with recently. When I dealt with this on my phone, I was puzzled why various instructions said to make sure that the TLS certificate was for a specific name (and I even wondered if this meant that the TLS certificate wasn't being verified at all). But the reason you have to check the name is that the 801.2X protocol doesn't have any trustworthy way of asserting what authentication server should be called; almost by definition, you can't trust anything the 801.2X server itself claims about what it should be called, and the only other information you have is (perhaps) the free-form name of the network (as, for example, the wireless SSID). The server name and the trust has to be established out of band, and on phones that's through websites with instructions.
(On Linux you have to explicitly configure the expected server name in advance if you want security.)
The second example is wanting to use DNS over TLS or DNS over HTTPS to talk to the DNS servers you find through DHCP or have in a normal resolv.conf. In both of these cases, the protocol and the configuration file only specify the DNS servers by IP address, with no names associated with them (and the IPs may well be RFC 1918 private addresses). It's possible to turn this into a server name if you want to through DNS, but you wind up basically having to trust what the DNS server is telling you about what its TLS server name should be.
(You can augment DHCP and resolv.conf with additional information about the server names you should look for, but then you need to define the format and protocol for this information, and you need more moving parts in order to get your TLS protected DNS queries.)
PS: Sometimes the first part of TLS is sufficiently important by itself, because blocking passive eavesdropping can be a significant win. But it's at least questionable, and you need to consider your threat models carefully.
Comments on this page:Written on 20 September 2019.