Wandering Thoughts archives


We may not want to use OCSP stapling in our web servers

Not too long ago, I was modernizing some of our Apache TLS settings. As I usually do, I went to the Mozilla SSL configuration generator (which is what I think you should do) and more or less accepted its recommendations for our Apache and OpenSSL version. These days, that includes OCSP stapling. I didn't really think much about this. Then some things happened recently, and today my co-workers asked more or less if we should be doing OCSP stapling at all. Unfortunately, the more I think about that question, the more I think that we should probably not configure OCSP stapling in our web servers.

One big motivation for OCSP stapling was OCSP must-staple, but that seems basically dead; I can find people talking about it, but I can't find any site actually using it, including people who set it up once upon a time. I'm not surprised by this, because OCSP must-staple can easily break your website if anything goes wrong with OCSP stapling and people generally don't like that sort of 'feature'. People may try it out, but sooner or later something explodes and then they stop. We're certainly never going to use OCSP must-staple unless it becomes a hard requirement in a popular browser.

If you're not using OCSP must-staple, the major thing that OCSP stapling means is that browsers that make OCSP checks can save some time when they connect to you. Desktop Firefox definitely does OCSP checks by default (although I believe mobile Firefox only does it for EV certificates), and Chrome definitely doesn't. I'm not sure about the state of Safari, either on iOS or OS X (although see this ssl.com article). Unfortunately, Firefox is no longer a major desktop browser, so how much OCSP stapling matters likely depends mostly on how Safari behaves.

(Currently, Apple appears to use OCSP stapling information under some circumstances, but it's not the only way to provide what they want.)

Pragmatically, a lot of sites that I would have expected to use OCSP stapling don't actually do so. None of Mozilla, Let's Encrypt, Twitter, Facebook, or Apple use OCSP stapling, although Apple's support site currently does, as does Amazon (and Amazon Canada). Infrequent use of OCSP stapling (and OCSP stapled data) matters, because the less it's used (in Apache and elsewhere), the more chances there are for all of the code related to it to have issues, especially in Apache.

On the whole, the more I look at the combination of the benefits of OCSP stapling, the potential hazards, and the limited use I can see out in the field (even and especially from places that I'd expect to support it), the less I feel like we should be using OCSP stapling. OCSP stapling seems to be at least a little bit bleeding edge, not well established practice, and there's no need for us to live out there.

OCSPStaplingMaybeNot written at 22:22:37; Add Comment


How I now think you want to configure Apache for OCSP stapling

When I initially set up OCSP stapling while I was modernizing our Apache TLS configurations, I followed the standard setup from the Mozilla SSL configuration generator (as is my usual habit). For OCSP stapling, the configuration this generates was (and is) just:

SSLUseStapling On
SSLStaplingCache "shmcb:logs/ssl_stapling(32768)"

Then recently our web servers at work couldn't get a good answer from Let's Encrypt's OCSP servers, for reasons that aren't clear to us. Some people experienced issues where their Firefox would refuse to connect to these web servers, rejecting things with a SEC_ERROR_OCSP_TRY_SERVER_LATER error.

(It's possible that this actually comes from Firefox itself directly querying the LE OCSP servers. These people were inside the same networks as the web servers and our problem could well have been a firewall or network reachability issue to LE's OCSP servers, instead of an issue on LE's side.)

This experience made me look into what happens when OCSP stapling runs into errors, and also into the Apache documentation. The result of this is that I now think that you should always tell Apache to not return OCSP errors, by also adding the Apache configuration option for this:

SSLStaplingReturnResponderErrors off

(The default is 'on'.)

This does the most aggressive version of handling OCSP problems; if set to off, the documentation says 'only responses indicating a certificate status of "good" will be included in the TLS handshake'. Expired responses, any errors, and any other certificate status causes Apache to not include OCSP stapling information at all.

(You may also want to see this Apache bug.)

PS: Firefox still defaults to checking certificate status through OCSP if necessary, but you can change this if you want to. The normal preferences only let you turn this off entirely, but if you go into about:config and set security.OCSP.enabled to the value of '2', Firefox will do OCSP checks for EV certificates but not for normal ones. Given the increasing disuse of EV certificates, I don't think it's worth bothering; just turn off OCSP checking entirely.

ApacheOCSPStaplingSettings written at 01:08:34; Add Comment


OCSP stapling and what web servers and browsers do in the face of errors

OCSP is an attempt to solve some of the problems of certificate revocation by having your browser or other TLS client check if TLS certificates are (still) valid when it sees them. OCSP stapling is an attempt to fix the privacy, performance, and infrastructure problems that OCSP creates by having the web server include ('staple') a sufficiently recent proof of its good OCSP certificate status in the TLS handshake; this proof is requested by the web server from its Certificate Authority's OCSP server every so often. As they say, so far so good. However, this raises two questions: what should the web server do if it can't get the OCSP proof from its CA when it asks, and what should browsers do in reaction to that?

It's possible to mark TLS certificates as 'OCSP must-staple', which means that the certificate is asserting that you should always see it with a stapled OCSP proof. If you're using such a certificate, it shouldn't matter what the web server does if it can't get an OCSP reply; no matter what, browsers should refuse to accept the certificate without it (whether they actually do is another matter, cf). However, most TLS certificates are not marked this way for various reasons (including that it's kind of dangerous because your CA can cut you off, either deliberately or accidentally).

For ordinary TLS certificates without OCSP must-staple, the web server has a choice of what to do in its conversation with the browser when it can't get a positive signed OCSP response from the CA. Either it can not include any OCSP stapling information at all, or it can pass on an OCSP error indication (or the signed OCSP response that says 'bad certificate' or 'unknown'). If the web server passes on OCSP errors to the browser, the browser may ignore them and continue with TLS as usual, ignore them and make its own OCSP query (even if it might not normally do an OCSP query), or report a TLS error and abort the connection.

(If the web server passes on a signed OCSP server response of 'bad certificate' or 'unknown', probably the browser should respect it.)

The safest thing for a web server to do is to only ever pass on a positive OCSP response. If your CA's OCSP server ever says anything other than 'your certificate is good', you don't say anything to the client (although if it says 'bad certificate' or 'unknown', you probably want to raise loud alarms in your monitoring system). In a perfect world, CA OCSP servers would always be operational and reliable, but in this world they aren't, so for most websites it's more likely that the CA's OCSP server has a problem than that you have a genuinely bad TLS certificate. When you're silent, at worst the browser will make its own OCSP query and get the same result, and at best either it won't query at all or it will query and get a good result.

As far as browsers go, Firefox will sometimes (or perhaps always) abort the TLS connection if it receives back certain OCSP errors, not just when it gets signed OCSP replies that contain bad statuses. For instance, if it receives an OCSP 'try later' status from the web server, it errors out with SEC_ERROR_OCSP_TRY_SERVER_LATER. This is not necessarily the greatest thing ever, because the 'try later' OCSP error is unsigned and so doesn't necessarily actually come from the CA's OCSP server. For that matter, the web server may decide to make up a 'try later' OCSP error status if it has some problem talking to the OCSP server. Chrome appears to ignore such 'try later' OCSP responses at least some of the time.

(Both web servers and browsers talk to OCSP servers over HTTP, not HTTPS, so they're vulnerable to man in the middle attacks for any responses that aren't signed. And of course the web server can give you any unsigned error status it wants to.)

PS: As far as OCSP must-staple goes, it appears that Cloudflare's blog doesn't use it any more, despite their 2017 blog entry pushing for it. Neither does Scott Helme. At this point I'd like to find a site that does use it just so I can check which of my tools actually report that.

OCSPStaplingAndErrors written at 23:39:37; Add Comment


Why I use both uBlock Origin and uMatrix

In response to my entry on my current Firefox addons, hwj asked a good question in the lobste.rs comments:

Isn’t uMatrix an advanced version of uBlock Origin? What’s the rationale behind using both of them?

While it's true that uMatrix and uBlock Origin have overlapping functionality (and are written by the same person), they have different purposes and focuses. uBlock Origin's focus is blocking ads and other undesired things as an out of the box experience with little configuration needed. uMatrix's focus is on exerting tight and highly specific control over what resources a page is allowed to load and use, including Javascript and cookies (and requires a lot of configuration).

(One significant difference in features is that uBlock Origin can remove HTML elements from HTML pages, while uMatrix has no support for this. Selectively removing HTML elements is extremely important for blocking ads, but it's not relevant if you're blocking entire HTTP requests. I believe that uMatrix's HTML modifications are limited to blocking inline Javascript.)

You can block Javascript with uBlock Origin and it's somewhat easier in simpler cases than using uMatrix, but you don't have the fine control over Javascript that uMatrix gives you (and this improves my experience of the web). Nor do you get the control over cookies and other types of resources, which is a deliberate simplification on uBlock Origin's part. At the same time, uMatrix doesn't give you sophisticated adblocking or things like making unwanted page elements go away, including those annoying permanent headers and footers.

So the reason that I use both of them is that they do different things for me. uBlock Origin removes ads and unwanted page elements, while uMatrix blocks Javascript, cookies, and so on. If I just wanted adblocking, element zapping, and blocking Javascript, I could probably use uBlock Origin alone, but I definitely want cookie blocking as well and I usually like the fine-grained control uMatrix gives me over other things as well.

(Writing this has given me a new appreciation for the difference between the blocklist sources included in uMatrix and the filter lists included in uBlock Origin. The blocklists uMatrix uses just list hosts, because that's what uMatrix deals with. uBlock Origin's filter lists include all sorts of sophisticated matching rules to make HTML elements disappear, as well as some host lists. Having just checked it now, I believe that all of the default uMatrix hosts lists are also in uBlock Origin, although they may not all be enabled by default.)

UBlockOriginAndUMatrix written at 21:32:46; Add Comment

My Firefox addons as of Firefox '74' (the current development version)

As I write this, Firefox 72 is the just released version of Firefox and 73 is in beta, but my primary Firefox is still a custom hacked version that I build from the development tree, so it most closely corresponds to what will be released as Firefox 74 in a certain amount of time (I've lost track of how fast Firefox makes releases). Since it's been about ten versions of Firefox (and more than a year) since the last time I covered my addons, it's time for another revisit of this perennial topic. Many of the words will be familiar from the last time, because my addons seem to have stabilized now.

My core addons, things that I consider more or less essential for my experience of Firefox, are:

  • Foxy Gestures (Github) is probably still the best gestures extension for me for modern versions of Firefox (but I can't say for sure, because I no longer investigate alternatives).

    (I use some custom gestures in my Foxy Gestures configuration that go with some custom hacks to my Firefox to add support for things like 'view page in no style' as part of the WebExtensions API.)

  • uBlock Origin (Github) is my standard 'block ads and other bad stuff' extension, and also what I use for selectively removing annoying elements of pages (like floating headers and footers).

  • uMatrix (Github) is my primary tool for blocking Javascript and cookies. uBlock Origin could handle the Javascript, but not really the cookies as far as I know, and in any case uMatrix gives me finer control over Javascript which I think is a better fit with how the web does Javascript today.

  • Cookie AutoDelete (Github) deals with the small issue that uMatrix doesn't actually block cookies, it just doesn't hand them back to websites. This is probably what you want in uMatrix's model of the world (see my entry on this for more details), but I don't want a clutter of cookies lingering around, so I use Cookie AutoDelete to get rid of them under controlled circumstances.

    (However unaesthetic it is, I think that the combination of uMatrix and Cookie AutoDelete is necessary to deal with cookies on the modern web. You need something to patrol around and delete any cookies that people have somehow managed to sneak in.)

  • Stylus has become necessary for me after Google changed their non-Javascript search results page to basically be their Javascript search results without Javascript, instead of the much nicer and more useful old version. I use Stylus to stop search results escaping off the right side of my browser window.

Additional fairly important addons that would change my experience if they weren't there:

  • Textern (Github) gives me the ability to edit textareas in a real editor. I use it all the time when writing comments here on Wandering Thoughts, but not as much as I expected on other places, partly because increasingly people want you to write things with all of the text of a paragraph run together in one line. Textern only works on Unix (or maybe just Linux) and setting it up takes a bit of work because of how it starts an editor (see this entry), but it works pretty smoothly for me.

    (I've changed its key sequence to Ctrl+Alt+E, because the original Ctrl+Shift+E no longer works great on Linux Firefox; see issue #30. Textern itself shifted to Ctrl+Shift+D in recent versions.)

  • Open in Browser (Github) allows me to (sometimes) override Firefox's decision to save files so that I see them in the browser instead. I mostly use this for some PDFs and some text files. Sadly its UI isn't as good and smooth as it was in pre-Quantum Firefox.

  • Cookie Quick Manager (Github) allows me to inspect, manipulate, save, and reload cookies and sets of cookies. This is kind of handy every so often, especially saving and reloading cookies.

The remaining addons I use I consider useful or nice, but not all that important on the large scale of things. I could lose them without entirely noticing the difference in my Firefox:

  • Certainly Something (Github) is my TLS certificate viewer of choice. I occasionally want to know the information it shows me, especially for our own sites.

  • HTTP/2 Indicator (Github) does what it says; it provides a little indicator as to whether HTTP/2 was active for the top-level page.

  • Link Cleaner cleans the utm_ fragments and so on out of URLs when I follow links. It's okay; I mostly don't notice it and I appreciate the cleaner URLs.

    (It also prevents some degree of information leakage to the target website about where I found their link, but I don't really care about that. I'm still sending Referer headers, after all.)

  • HTTPS Everywhere, basically just because. But in a web world where more and more sites are moving to using things like HSTS, I'm not sure HTTPS Everywhere is all that important any more.

Some of my previous extensions have stopped being useful since last time. They are:

  • My Google Search URL Fixup, because Google changed its search pages (as covered above for Stylus) and it became both unnecessary and non-functional. I should probably update its official description to note this, but Google's actions made me grumpy and lazy.

  • Make Medium Readable Again (also, Github) used to deal with a bunch of annoyances for Medium-hosted stuff, but then Medium changed their CSS and it hasn't been updated for that. I can't blame the extension's author; keeping up with all of the things that sites like Medium do to hassle you is a thankless and never-ending job.

I still have both of these enabled in my Firefox, mostly because it's more work to remove them than to let them be. In the case of MMRA, perhaps its development will come back to life again and a new version will be released.

(There are actually some branches in the MMRA Github repo and a bunch of forks of it, some of which are ahead of the main one. Possibly people are quietly working away on this.)

I have some Firefox profiles that are for when I want to use Javascript (they actually use the official Mozilla Linux Firefox release these days, which I just updated to Firefox 72). In these profiles, I also use Decentraleyes (also), which is a local CDN emulation so that less of my traffic is visible to CDN operators. I don't use it in my main Firefox because I'm not certain how it interacts with me blocking (most) Javascript setup, and also much of what's fetched from CDNs is Javascript, which obviously isn't applicable to me.

(There are somewhat scary directions in the Decentraleyes wiki on making it work with uMatrix. I opted to skip them entirely.)

Firefox74Addons written at 01:56:48; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.