We may not want to use OCSP stapling in our web servers
Not too long ago, I was modernizing some of our Apache TLS settings. As I usually do, I went to the Mozilla SSL configuration generator (which is what I think you should do) and more or less accepted its recommendations for our Apache and OpenSSL version. These days, that includes OCSP stapling. I didn't really think much about this. Then some things happened recently, and today my co-workers asked more or less if we should be doing OCSP stapling at all. Unfortunately, the more I think about that question, the more I think that we should probably not configure OCSP stapling in our web servers.
One big motivation for OCSP stapling was OCSP must-staple, but that seems basically dead; I can find people talking about it, but I can't find any site actually using it, including people who set it up once upon a time. I'm not surprised by this, because OCSP must-staple can easily break your website if anything goes wrong with OCSP stapling and people generally don't like that sort of 'feature'. People may try it out, but sooner or later something explodes and then they stop. We're certainly never going to use OCSP must-staple unless it becomes a hard requirement in a popular browser.
If you're not using OCSP must-staple, the major thing that OCSP stapling means is that browsers that make OCSP checks can save some time when they connect to you. Desktop Firefox definitely does OCSP checks by default (although I believe mobile Firefox only does it for EV certificates), and Chrome definitely doesn't. I'm not sure about the state of Safari, either on iOS or OS X (although see this ssl.com article). Unfortunately, Firefox is no longer a major desktop browser, so how much OCSP stapling matters likely depends mostly on how Safari behaves.
(Currently, Apple appears to use OCSP stapling information under some circumstances, but it's not the only way to provide what they want.)
Pragmatically, a lot of sites that I would have expected to use OCSP stapling don't actually do so. None of Mozilla, Let's Encrypt, Twitter, Facebook, or Apple use OCSP stapling, although Apple's support site currently does, as does Amazon (and Amazon Canada). Infrequent use of OCSP stapling (and OCSP stapled data) matters, because the less it's used (in Apache and elsewhere), the more chances there are for all of the code related to it to have issues, especially in Apache.
On the whole, the more I look at the combination of the benefits of OCSP stapling, the potential hazards, and the limited use I can see out in the field (even and especially from places that I'd expect to support it), the less I feel like we should be using OCSP stapling. OCSP stapling seems to be at least a little bit bleeding edge, not well established practice, and there's no need for us to live out there.
How I now think you want to configure Apache for OCSP stapling
When I initially set up OCSP stapling while I was modernizing our Apache TLS configurations, I followed the standard setup from the Mozilla SSL configuration generator (as is my usual habit). For OCSP stapling, the configuration this generates was (and is) just:
SSLUseStapling On SSLStaplingCache "shmcb:logs/ssl_stapling(32768)"
Then recently our web servers at work couldn't get a good answer from Let's Encrypt's OCSP servers, for reasons that aren't clear to us. Some people experienced issues where their Firefox would refuse to connect to these web servers, rejecting things with a SEC_ERROR_OCSP_TRY_SERVER_LATER error.
(It's possible that this actually comes from Firefox itself directly querying the LE OCSP servers. These people were inside the same networks as the web servers and our problem could well have been a firewall or network reachability issue to LE's OCSP servers, instead of an issue on LE's side.)
This experience made me look into what happens when OCSP stapling runs into errors, and also into the Apache documentation. The result of this is that I now think that you should always tell Apache to not return OCSP errors, by also adding the Apache configuration option for this:
(The default is '
This does the most aggressive version of handling OCSP problems; if set to off, the documentation says 'only responses indicating a certificate status of "good" will be included in the TLS handshake'. Expired responses, any errors, and any other certificate status causes Apache to not include OCSP stapling information at all.
(You may also want to see this Apache bug.)
PS: Firefox still defaults to checking certificate status through
OCSP if necessary, but you can change this if you want to. The
normal preferences only let you turn this off entirely, but if you
about:config and set
security.OCSP.enabled to the value
of '2', Firefox will do OCSP checks for EV certificates but not for
normal ones. Given the increasing disuse of EV certificates, I don't
think it's worth bothering; just turn off OCSP checking entirely.
OCSP stapling and what web servers and browsers do in the face of errors
OCSP is an attempt to solve some of the problems of certificate revocation by having your browser or other TLS client check if TLS certificates are (still) valid when it sees them. OCSP stapling is an attempt to fix the privacy, performance, and infrastructure problems that OCSP creates by having the web server include ('staple') a sufficiently recent proof of its good OCSP certificate status in the TLS handshake; this proof is requested by the web server from its Certificate Authority's OCSP server every so often. As they say, so far so good. However, this raises two questions: what should the web server do if it can't get the OCSP proof from its CA when it asks, and what should browsers do in reaction to that?
It's possible to mark TLS certificates as 'OCSP must-staple', which means that the certificate is asserting that you should always see it with a stapled OCSP proof. If you're using such a certificate, it shouldn't matter what the web server does if it can't get an OCSP reply; no matter what, browsers should refuse to accept the certificate without it (whether they actually do is another matter, cf). However, most TLS certificates are not marked this way for various reasons (including that it's kind of dangerous because your CA can cut you off, either deliberately or accidentally).
For ordinary TLS certificates without OCSP must-staple, the web server has a choice of what to do in its conversation with the browser when it can't get a positive signed OCSP response from the CA. Either it can not include any OCSP stapling information at all, or it can pass on an OCSP error indication (or the signed OCSP response that says 'bad certificate' or 'unknown'). If the web server passes on OCSP errors to the browser, the browser may ignore them and continue with TLS as usual, ignore them and make its own OCSP query (even if it might not normally do an OCSP query), or report a TLS error and abort the connection.
(If the web server passes on a signed OCSP server response of 'bad certificate' or 'unknown', probably the browser should respect it.)
The safest thing for a web server to do is to only ever pass on a positive OCSP response. If your CA's OCSP server ever says anything other than 'your certificate is good', you don't say anything to the client (although if it says 'bad certificate' or 'unknown', you probably want to raise loud alarms in your monitoring system). In a perfect world, CA OCSP servers would always be operational and reliable, but in this world they aren't, so for most websites it's more likely that the CA's OCSP server has a problem than that you have a genuinely bad TLS certificate. When you're silent, at worst the browser will make its own OCSP query and get the same result, and at best either it won't query at all or it will query and get a good result.
As far as browsers go, Firefox will sometimes (or perhaps always) abort the TLS connection if it receives back certain OCSP errors, not just when it gets signed OCSP replies that contain bad statuses. For instance, if it receives an OCSP 'try later' status from the web server, it errors out with SEC_ERROR_OCSP_TRY_SERVER_LATER. This is not necessarily the greatest thing ever, because the 'try later' OCSP error is unsigned and so doesn't necessarily actually come from the CA's OCSP server. For that matter, the web server may decide to make up a 'try later' OCSP error status if it has some problem talking to the OCSP server. Chrome appears to ignore such 'try later' OCSP responses at least some of the time.
(Both web servers and browsers talk to OCSP servers over HTTP, not HTTPS, so they're vulnerable to man in the middle attacks for any responses that aren't signed. And of course the web server can give you any unsigned error status it wants to.)
PS: As far as OCSP must-staple goes, it appears that Cloudflare's blog doesn't use it any more, despite their 2017 blog entry pushing for it. Neither does Scott Helme. At this point I'd like to find a site that does use it just so I can check which of my tools actually report that.
Why I use both uBlock Origin and uMatrix
Isn’t uMatrix an advanced version of uBlock Origin? What’s the rationale behind using both of them?
(Writing this has given me a new appreciation for the difference between the blocklist sources included in uMatrix and the filter lists included in uBlock Origin. The blocklists uMatrix uses just list hosts, because that's what uMatrix deals with. uBlock Origin's filter lists include all sorts of sophisticated matching rules to make HTML elements disappear, as well as some host lists. Having just checked it now, I believe that all of the default uMatrix hosts lists are also in uBlock Origin, although they may not all be enabled by default.)
My Firefox addons as of Firefox '74' (the current development version)
As I write this, Firefox 72 is the just released version of Firefox and 73 is in beta, but my primary Firefox is still a custom hacked version that I build from the development tree, so it most closely corresponds to what will be released as Firefox 74 in a certain amount of time (I've lost track of how fast Firefox makes releases). Since it's been about ten versions of Firefox (and more than a year) since the last time I covered my addons, it's time for another revisit of this perennial topic. Many of the words will be familiar from the last time, because my addons seem to have stabilized now.
My core addons, things that I consider more or less essential for my experience of Firefox, are:
- Foxy Gestures
probably still the best gestures extension for me for modern
versions of Firefox (but I can't say for sure, because I no
longer investigate alternatives).
(I use some custom gestures in my Foxy Gestures configuration that go with some custom hacks to my Firefox to add support for things like 'view page in no style' as part of the WebExtensions API.)
- uBlock Origin (Github) is my standard 'block ads
and other bad stuff' extension, and also what I use for selectively
removing annoying elements of pages (like floating headers and
(Github) is my primary tool
cookies as far as I know, and in any case uMatrix gives me finer
- Cookie AutoDelete
deals with the small issue that uMatrix doesn't actually block
cookies, it just doesn't hand them back to websites. This is
probably what you want in uMatrix's model of the world (see my
entry on this for more details), but
I don't want a clutter of cookies lingering around, so I use
Cookie AutoDelete to get rid of them under controlled circumstances.
(However unaesthetic it is, I think that the combination of uMatrix and Cookie AutoDelete is necessary to deal with cookies on the modern web. You need something to patrol around and delete any cookies that people have somehow managed to sneak in.)
Additional fairly important addons that would change my experience if they weren't there:
(Github) gives me the ability
to edit textareas in a real editor. I use it all the time when
writing comments here on Wandering Thoughts, but not
as much as I expected on other places, partly because increasingly
people want you to write things with all of the text of a paragraph
run together in one line. Textern only works on Unix (or maybe
just Linux) and setting it up takes a bit of work because of how
it starts an editor (see this entry),
but it works pretty smoothly for me.
(I've changed its key sequence to Ctrl+Alt+E, because the original Ctrl+Shift+E no longer works great on Linux Firefox; see issue #30. Textern itself shifted to Ctrl+Shift+D in recent versions.)
- Open in Browser
(Github) allows me
to (sometimes) override Firefox's decision to save files so that
I see them in the browser instead. I mostly use this for some
PDFs and some text files. Sadly its UI isn't as good and smooth
as it was in pre-Quantum Firefox.
- Cookie Quick Manager (Github) allows me to inspect, manipulate, save, and reload cookies and sets of cookies. This is kind of handy every so often, especially saving and reloading cookies.
The remaining addons I use I consider useful or nice, but not all that important on the large scale of things. I could lose them without entirely noticing the difference in my Firefox:
- Certainly Something
(Github) is my
TLS certificate viewer of choice. I occasionally want to know the
information it shows me, especially for our own sites.
- HTTP/2 Indicator
(Github) does what
it says; it provides a little indicator as to whether HTTP/2 was
active for the top-level page.
- Link Cleaner cleans
the utm_ fragments and so on out of URLs when I follow links. It's
okay; I mostly don't notice it and I appreciate the cleaner URLs.
(It also prevents some degree of information leakage to the target website about where I found their link, but I don't really care about that. I'm still sending
Refererheaders, after all.)
- HTTPS Everywhere, basically just because. But in a web world where more and more sites are moving to using things like HSTS, I'm not sure HTTPS Everywhere is all that important any more.
Some of my previous extensions have stopped being useful since last time. They are:
- My Google Search URL Fixup,
because Google changed its search pages (as covered above for
Stylus) and it became both unnecessary and non-functional. I
should probably update its official description to note this, but
Google's actions made me grumpy and lazy.
- Make Medium Readable Again (also, Github) used to deal with a bunch of annoyances for Medium-hosted stuff, but then Medium changed their CSS and it hasn't been updated for that. I can't blame the extension's author; keeping up with all of the things that sites like Medium do to hassle you is a thankless and never-ending job.
I still have both of these enabled in my Firefox, mostly because it's more work to remove them than to let them be. In the case of MMRA, perhaps its development will come back to life again and a new version will be released.
(There are actually some branches in the MMRA Github repo and a bunch of forks of it, some of which are ahead of the main one. Possibly people are quietly working away on this.)
(There are somewhat scary directions in the Decentraleyes wiki on making it work with uMatrix. I opted to skip them entirely.)