We can't use Let's Encrypt on our production systems right now

February 19, 2016

I really like Let's Encrypt, the new free and automated non-profit TLS Certificate Authority. Free is hard to beat, especially around here, and automatically issued certificates that don't require tedious interaction with websites are handy. And in general I love people who're striking a blow against the traditional CA racket. Unfortunately, despite all of that, there's basically no prospect of us using LE certificates in production around here.

The problem is not any of the traditional ones you might think of. Browsers trust the LE certificates, and that LE only does basic 'Domain Validation' (DV) certificates is not an issue because those are what we use anyways. And I have no qualms about using a free CA; CAs are in a commodity business and LE is easier to deal with than the alternatives due to their automation. It's not even the short 90-day duration on their certificates (although that's a potential issue).

The problem for us is that Let's Encrypt (currently) has relatively low rate limits, and especially it has a limit of five certificates per domain per week. Even if LE interpreted this very liberally (applying it to just our department's subdomain instead of the entire university), this is probably nowhere near enough for our usage. We have more than five different servers doing TLS ourselves, never mind all of the web servers run by research groups or even individual graduate students. This isn't just an issue of having to carefully schedule asking for certificates (and the resulting certificate renewals); it's also a massive coordination problem among all of the disparate people who could request certificates. As far as I can tell, using LE certificates in production here would mean giving a very large number of people the power to stop us from being able to renew (production) certificates. That's just not a risk we can take, especially since you have to renew LE certificates fairly often.

(Sure, we'd renew well ahead of time and if there were problems we could buy a commercial TLS certificate to replace the LE one. But if we're going to have problems very often we can save ourselves the heartburn and the fire drill by just buying commercial certificates in the first place. The university may not value staff time very highly in general but our time is still worth some actual money, and commercial certificates are cheap.)

I do feel sad about this, as I'd certainly like to be able to use LE certificates in production here (and I'd prefer to use them, especially with automatically handled renewal). But I suspect that a big university is always going to be a corner case that LE's rate limits simply won't deal with. If the university got seriously into 'TLS for all web sites', we're probably talking about at least thousands of separate servers.

(This doesn't mean that I have no use for LE certificates here. But that's another entry.)

Sidebar: my views on multiple names on the same certificate

TLS certificates can be issued with multiple names by using SANs, which means that you can theoretically cut down the number of distinct certificates you need by cramming a bunch of names on to one certificate. LE is especially generous with how many SANs you can attach to one certificate.

My personal dividing line is that I'm only willing to put multiple names into a TLS certificate when all of the names will be used on the same server. If I'm putting fifteen virtual host names into a certificate that will be used on a single web server, that's fine. If I'm jamming fifteen different web servers into one TLS certificate and so I'm going to have fifteen copies of it (and its key) on fifteen hosts, that's not fine. I should get separate certificates, so that the damage is more limited if one of those hosts gets compromised.

Comments on this page:

By Jack Dozier at 2016-02-19 09:53:29:

Have you looked at InCommon's Certificate Service? I worked at a mid-sized 2 year and after comparing costs we found that we could join InCommon and get their unlimited certificate service and still save money compared to buying from the private market.

You make a compelling argument.

I'd suggest approaching LE and see if they have anything in the pipeline that you might be able to (beta) test for them when the time comes around.

By cks at 2016-02-21 19:54:01:

Jack: I hadn't heard of InCommon before now and they're interesting. Unfortunately I think the university as a whole would have to join (if only in order to make the costs make sense), and that creates relatively big political and coordination problems. I may suggest it to the central powers that would be making the decisions, but I honestly suspect it wouldn't go anywhere for various reasons.

By cks at 2016-02-22 01:37:15:

To add an update: I just looked at the InCommon website in more detail, and unfortunately they're only open to institutions in the US. We're in Canada, so I expect they'd tell us that we're ineligible.

By Jack Dozier at 2016-02-22 08:50:57:

I completely understand on the political landscape issue. I'm from a two-year with a highly centralized IT, so it made good sense for us. The only people on campus who could spell TLS have desks within shouting distance of each other.

Sorry to hear about it only being available in the US. That worry had crossed my mind, but I figured it was better to mention than not.

Greek Universities have created their own CA a few years ago. A virtual team of people took the bullet and went through all the steps required to become trusted in browsers. This CA now issues certificates for all of them -> https://www.harica.gr/

By Ewen McNeill at 2016-02-23 02:51:33:

FWIW in discussion with a Mozilla developer at a conference today about the Lets Encrypt rate limits their reading of the rate limits seemed to be that only the requests/IP was likely to be an issue for larger organisations (with few public IPs). And that an IP rich organisation could just ensure production requests came from a different IP from "general rabble".

They didn't seem concerned about the 5 certificates per registered domain per week. Reading that page again, it looks like the text is supposed to mean "per FQDN", not just "per DNS delegation from global registry". I've just to prevent people from repeatedly getting new 90 day certs for identical FQDN (ie "reissues"). So it might be worth investigating that limit further.

Finally Let's Encrypt is still in Open Beta at present. So the developer suggested that Let's Encrypt were keen to hear from organisations that it wouldn't work for as is, so things can be tweaked. There definitely are some organisations (eg some web hosts with "tick for TLS" options) which already have thousands of certs issued.

So maybe it's just the documentation which needs clarifying to be "per FQDN required in certificate" rather than "per 2LD" or "per 3LD".



Written on 19 February 2016.
« Two models of dealing with cookies in Firefox with addons
My two usage cases for Let's Encrypt certificates »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Feb 19 01:49:30 2016
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.