Chris's Wiki :: blog/web/WebServerMTLSHazards

The extra hazards of mutual TLS authentication (mTLS) in web servers

March 15, 2023

Today I wound up reading a 2018 Colm MacCárthaigh thread on mutual TLS (mTLS) (via, via). I was nodding vaguely along to the thread until I hit this point, which raised an issue I hadn't previously thought about:

Directly related to all of this is that it takes an enormous amount of code to do mTLS. An ordinary TLS server can more or less ignore X509 and ASN.1 - it doesn't need to parse or handle them. Turn on MTLS and all of a sudden it does!

In mutual TLS (authentication), clients send the server a certificate just as the server sends one to clients. The server can then authenticate the client, just as the client authenticates the server (hence 'mutual TLS'). In both cases, by 'authenticates' we mean verifying the TLS certificate and also extracting various pieces of information in it.

As we should all know by now, verifying TLS certificates is not a simple process. Nor is extracting information from TLS certificates; over the years that have been an assortment of bugs in this area, such as KDE missing the difference between text blobs and C strings. In a conventional TLS environment that complexity lives only in the TLS client, which must have an X.509 and thus ASN.1 parser in order to understand and verify the server's TLS certificate.

In mTLS, the server has to verify the client's TLS certificate as well and then generally extract information from it (often people want some sort of identification of the client). This means that it needs to parse and verify X.509 certificates, complete with an ASN.1 parser. This is a lot more code than before and it's directly exposed to unauthenticated clients (you can't verify a signed TLS certificate without extracting information about the Certificate Authority from it).

If you have a narrow, specific circumstance you can potentially write somewhat narrow code that (hopefully) rejects client certificates that contain any oddities that shouldn't be present in yours, and so simplify this situation somewhat. However, you can't do this in general purpose code, such as code in general web servers or language libraries; that code pretty much needs to handle all of the oddities that are legal in mTLS client certificates. That means ASN.1 parsing that's as full as for server certificates and more or less full TLS certificate verification. Most people will probably not use cross-signed CA intermediate certificates with partially expired certificate chains for their mTLS client certificates, but if it's allowed, well.

Pragmatically, there have been a fair number of OpenSSL security issues that were only exploitable if you could get an OpenSSL client to talk to a server with a specially crafted certificate or otherwise compromised. With mTLS in your web server, congratulations, your web server is now a 'client'; these vulnerabilities may partially or completely apply. And your web server is generally a juicy target.

PS: mTLS in things that are merely using HTTPS as a transport is a somewhat different matter, although you still have new problems (such as those OpenSSL issues). My system administrator's view is that the extra code for mTLS is probably much less well tested in actual use and so probably has more bugs than plain HTTPS with some form of server certificate verification.

Comments on this page:

By Nolan at 2023-03-16 07:59:07:

One potential saving grace here is that checking certs (and extracting a subset of client ID) is easy to sandbox. You don't even need any of the fancy seccomp modes, the basic SECCOMP_SET_MODE_STRICT that just allows read(2), write(2) and exit(2) is good enough.

So finding an RCE in the TLS/X.509/ASN code just means that you can lie about your certificate and ID, which is not good, but not much worse than having no mTLS.

No idea if any servers actually implement this.

By Simon at 2023-03-16 22:11:33:

Nowadays a (web) server is a literal TLS client in many security critical functions (distro updates, other code distribution, interface with other servers/services, other protocols, etc., etc.). So most likely you rely on that part of the TLS stack anyway. So I'm not sure that this is a particular convincing argument.

(There are of course other things that are TLS client auth specific, but having to rely on the handling of X509 and ASN.1 isn't.)

By cks at 2023-03-16 23:21:11:

A web server program is generally not involved as a TLS client for things like updates; that is handled by a separate part of the overall system. Mutual TLS requires the web server program to process the client TLS certificate; that's the whole point.

In addition, client TLS to public HTTPS servers is also much less risky because exploiting issues generally requires a public HTTPS server to either present a specially crafted certificate without a signature or persuade a public Certificate Authority to sign a TLS certificate with all sorts of questionable things (and then make it public via Certificate Transparency). Many TLS CAs are extremely restrictive in what they will sign and what data they take from your Certificate Signing Request (CSR), to the point that it's unlikely an attacker would be able to sneak things into a properly signed certificate.

Meanwhile, in mTLS an attacker can likely present all sorts of unsigned garbage to the server without much consequences, forcing a web server to parse it all. The odds are also higher that they can get questionable things signed (either legitimately or by compromising an internal CA), and there's much less auditing of what does get signed. (I believe there are multiple people running TLS certificate linters against the public CT logs.)

From 185.220.101.172 at 2023-03-30 20:26:38:

Ok, if we are talking only about the "web server program" (like apache, nginx) that's more true, although I'm not sure looking at only part of the system makes that much sense when making a "this adds complexity" argument. But even there this is only partially true, since for example for revere proxy usage you probably already have TLS client support in your web server program.

Quite a bit of the whole X.509, ASN.1, etc. handling has to happen before checking the signatures, regardless of server or client certs, so the "present all sorts of unsigned garbage" applies in both cases.

In the setups we are talking here you will often find internal CAs, but that's true for both client as well as server authentication.

I agree that it's more likely that someone messes up what their internal CA signs. On the other hand with internal CAs you can be much more restrictive than a public CA can be, all you need is a mapping of DNS names to public key, all other stuff like CSRs, ACME, etc. is not required.

For client auth you have the advantage that the list of CAs you have to accept will be much sorter than for server auth when connecting to public TLS servers.

So I agree that TLS has a lot of complexity I would love to see go away. But the presented argument doesn't convince me given how much we already rely on exactly those parts of TLS for critical things.

To be clear, I expect client auth specific bugs, since this code path has got much less attention. So this is something you have to consider when using it. But you also have to look at all the complexity you get with other forms of authentication.

PS: What you write about Certificate Transparency will be likely true for certs signed by public CAs. But keep in mind that beside Chromium based browsers (and Google's crawlers), your client usually doesn't check if the cert has been (promised to be) included in a log, so it doesn't have to be.

Written on 15 March 2023.

«	Some notes on searching the systemd journal with journalctl

NFS filehandles from Linux NFS servers can be client specific