The extra hazards of mutual TLS authentication (mTLS) in web servers
Today I wound up reading a 2018 Colm MacCárthaigh thread on mutual TLS (mTLS) (via, via). I was nodding vaguely along to the thread until I hit this point, which raised an issue I hadn't previously thought about:
Directly related to all of this is that it takes an enormous amount of code to do mTLS. An ordinary TLS server can more or less ignore X509 and ASN.1 - it doesn't need to parse or handle them. Turn on MTLS and all of a sudden it does!
In mutual TLS (authentication), clients send the server a certificate just as the server sends one to clients. The server can then authenticate the client, just as the client authenticates the server (hence 'mutual TLS'). In both cases, by 'authenticates' we mean verifying the TLS certificate and also extracting various pieces of information in it.
As we should all know by now, verifying TLS certificates is not a simple process. Nor is extracting information from TLS certificates; over the years that have been an assortment of bugs in this area, such as KDE missing the difference between text blobs and C strings. In a conventional TLS environment that complexity lives only in the TLS client, which must have an X.509 and thus ASN.1 parser in order to understand and verify the server's TLS certificate.
In mTLS, the server has to verify the client's TLS certificate as well and then generally extract information from it (often people want some sort of identification of the client). This means that it needs to parse and verify X.509 certificates, complete with an ASN.1 parser. This is a lot more code than before and it's directly exposed to unauthenticated clients (you can't verify a signed TLS certificate without extracting information about the Certificate Authority from it).
If you have a narrow, specific circumstance you can potentially write somewhat narrow code that (hopefully) rejects client certificates that contain any oddities that shouldn't be present in yours, and so simplify this situation somewhat. However, you can't do this in general purpose code, such as code in general web servers or language libraries; that code pretty much needs to handle all of the oddities that are legal in mTLS client certificates. That means ASN.1 parsing that's as full as for server certificates and more or less full TLS certificate verification. Most people will probably not use cross-signed CA intermediate certificates with partially expired certificate chains for their mTLS client certificates, but if it's allowed, well.
Pragmatically, there have been a fair number of OpenSSL security issues that were only exploitable if you could get an OpenSSL client to talk to a server with a specially crafted certificate or otherwise compromised. With mTLS in your web server, congratulations, your web server is now a 'client'; these vulnerabilities may partially or completely apply. And your web server is generally a juicy target.
PS: mTLS in things that are merely using HTTPS as a transport is a somewhat different matter, although you still have new problems (such as those OpenSSL issues). My system administrator's view is that the extra code for mTLS is probably much less well tested in actual use and so probably has more bugs than plain HTTPS with some form of server certificate verification.