Recently I wrote about my failure to arrange a graceful TLS root certificate rollover for our OpenVPN servers. This might leave you wondering what we're doing about this instead, and the answer is that we've opted to use a brute force solution, because we know it works.

Our brute force solution is to set up a new set of OpenVPN servers (we have two of them for redundancy), using a new official name and with it a new TLS root certificate that is good for quite a while (I opted to be cautious and not cross into 2038) and with it a new host certificate. With the new servers set up and in production, we've updated our support site so use the new official name and the new TLS root certificate, so people who set up OpenVPN from now onward will be using the new environment.

Since these servers are using a new official name, they and the current (old) OpenVPN servers can operate at the same time. People with the new client configuration go through our new servers; people with the old client configuration and old TLS certificate go through our old servers. There's no flag day where we have to change the TLS root certificate on the old servers, and in fact they won't change; we're going to run them as-is right up until the TLS root certificate expires and no one can connect to them any more.

This leaves us with all of the people who are currently using our old OpenVPN servers with the expiring TLS root certificate. We're just going to have to contact all of them and ask them to update (ie change) their client configuration, changing the OpenVPN server name and getting and installing the new TLS root certificate. This is not quite as bad as it might sound, because we were always going to have to contact the current people to get them to update their TLS root certificate. So they only have to do one extra thing, although that extra thing may be quite a big pain.

(Some environments have nice, simple OpenVPN configuration systems. But on some platforms, the configuration is 'open a text editor and ...', and one of them is probably not one you're thinking of.)

Doing the change this way 'costs' us two extra servers for a while, which we have to spare, and more importantly it meant that we needed a new official name for our OpenVPN service. This time around this was acceptable, because our old official name was in retrospect perhaps not the best option. If we have to do this again, we may have a harder time coming up with a good new name, but hopefully next time around we'll be able to roll over the TLS root certificate instead of having to start all over from scratch.

(From my perspective, the most annoying thing about this is that I just rebuilt the OpenVPN servers in January in order to update them to a modern OpenBSD. If I'd known all of this back then, we could have gone straight to our new end state and saved one round of building and installing machines.)