2023-10-02
What we did when we couldn't gracefully roll over an OpenVPN TLS root certificate
A few years ago I wrote an entry on my failure to gracefully roll over a TLS root certificate for OpenVPN, where I described the problem and what hadn't worked for me, but didn't describe what we did about it (partly because at the time I wrote the entry, we hadn't dealt with the problem). Recently a commentator was interested in what we ended up doing, because they're in a similar situation. Unfortunately we didn't find a solution, so we had to work around the problem with more or less brute force.
The problem was that the current TLS root certificate for our OpenVPN server was expiring; this TLS certificate had been imported by all of the clients that people were using to connect to our server. Since we couldn't arrange any sort of roll over or renewal for the TLS root certificate, our only way out was to generate a new one. To make the switch less abrupt, we set up a new OpenVPN server under a new name (using the new TLS root certificate). Once the new server was up and working, we updated our support site to refer to the new OpenVPN server and its new TLS root certificate, and then contacted all of the old users to tell them they had a month or two to switch over, until the old TLS root certificate expired (and their OpenVPN client would probably start refusing to connect to the old server).
Generally, people could switch to the new OpenVPN server by downloading the new TLS root certificate, installing it into their current OpenVPN configuration, and then changing the host name of the server. Some OpenVPN clients probably took more work than this, perhaps up to deleting the old configuration and setting up a completely new one. As far as we know, no one had major problems with the switch.
(Once the old TLS root certificate had expired and usage of the old OpenVPN server had gone to nothing, we turned it off, took out the old name from DNS, and so on.)
Our new OpenVPN TLS root certificate was (is) set up to expire in mid 2037. I chose not to make it any further out than that because of the year 2038 problem, in that I didn't want to find out the hard way that some OpenVPN client had problems with TLS root certificate expiry times past that point. If we're still using OpenVPN in 2037 I will be a little bit disappointed.
(TLS certificates with expiry times past 2049 have to use a different internal representation of the expiry time, and client handling of that may also turn out to have bugs.)
As a side note, our VPN usage metrics suggest pretty strongly that OpenVPN clients do care about the expiry time of the server's TLS root certificate. A number of people had sessions that were constantly connected until shortly after the TLS root certificate expiry time; within a few hours they were all gone.
(We import basic usage metrics into our Prometheus setup, which makes it easy to go back a few years to look at just what happened one day in late August of 2021.)