How Firefox could support automatically using local DNS over HTTPS servers
On the surface, one of the challenges for Firefox automatically using different DNS over HTTPS servers is that Firefox considers your ISP to be a threat. This means that Firefox doesn't want to just use your local DNS over HTTPS server any more than it wants to just use your normal local DNS server. Firefox's use of DNS over HTTPS is explicitly to avoid surveillance from various parties, including the local network, so to do this it needs to go straight to a trusted (public) DNS over HTTPS server.
But there is a leak in this security model, in the form of Firefox's canary domain for disabling its automatic DNS over HTTPS. Any local network can already tell Firefox to disable DNS over HTTPS, defeating this anti-snooping measure. This is necessary because Firefox can't reliably detect when DNS over HTTPS to a public DNS server won't work properly for the local network, so networks with special name resolution setups need some way to signal this to Firefox.
(As a practical matter, Firefox not supporting a way to disable its automatic DNS over HTTPS to public DNS servers would result in a certain amount of the remaining Firefox users dropping it, because it didn't work reliably in their network. So Mozilla's hand is forced on this, even though it allows ISPs to step in and snoop on people again.)
Since Firefox already supports disabling automatic DNS over HTTPS entirely through a network doing magic tricks with the canary domain, it could also support a way of using the canary domain to signal that Firefox should use a local DNS over HTTPS server. This is no worse than turning off DoH entirely (in both cases your DNS queries are going to the network operator), and has some advantages such as potentially enabling encrypted SNI.
(Firefox's threat model might say that it can't enable ESNI with an untrusted local DNS over HTTPS server that was picked up automatically.)
Sensible heuristics for when to use DNS over HTTPS can't work for us
If Firefox is using DNS over HTTPS in general, it has various heuristics for whether or not to use it to resolve any particular name; for instance, right now it doesn't use DNS over HTTPS for any domain in your DNS suffixes (this happens even if you explicitly turned on DNS over HTTPS, which disables checking for the canary domain). Presumably other browsers will also have their own set of heuristics when they implement DNS over HTTPS, and at some point the set of heuristics that various browsers use may even be documented.
(This DNS over HTTPS bypass is intended to deal with two cases; where the name you're looking up doesn't exist in public DNS, and where the name has a different IP address.)
Almost six months ago I wrote a cautiously optimistic entry about Firefox, DNS over HTTPS, and us, where I hoped that Firefox's heuristics would be able to deal with our split horizon DNS setup where some names resolve to different IPs internally than they do externally. Unfortunately, I've changed my mind (based on experience and experimentation since then); I now believe that no sensible set of heuristics can cover all of our cases, and so anyone using DNS over HTTPS (with external resolvers) will sooner or later be unable to connect to some of the websites run by people in the department.
The fundamental issue that sinks the entire thing here is that people sometimes want to host websites on their machines here but give them names not under our university domains (for various good reasons, such a regular yearly conference that just happens to be hosted here this time). We do know what these domains are, because we have to set up the split DNS for them, but it's definitely not appropriate to add them to our DNS suffixes and they have different public DNS servers than our internal resolvers.
(In some cases they have public DNS servers that aren't even associated with us, and we carefully shadow bits of their domain internally to make it work. We prefer to host their DNS, though.)
I can't think of any sensible heuristic that could detect this situation, especially if you don't want to leak information about the person's DNS lookups to the local resolver. You could detect connection failures and try a non DNS over HTTPS name lookup, but that leaks data in various circumstances and even if it works there's a long connection delay the first time around.
So I think we're going to always have the 'disable DNS over HTTPS' canary domain signal in our local DNS servers, and we'll hope that someday this signal is respected even for Firefox users who have explicitly turned on DNS over HTTPS (unless they explicitly picked a new option of 'always use DoH even if the local network signals that it shouldn't be used and might give you broken results'). This makes me a little bit sad; I'd like heuristics that magically worked, so we could let people use DNS over HTTPS and hide things from us that we don't want to know anyway.
Some notes on the state of DNS over HTTPS in Firefox (as of March 2020)
Recently, we decided to add the magic marker that's used to explicitly disable DNS over HTTPS to our local DNS resolvers as a precaution against various things. Being sensible people, we then attempted to verify that we'd gotten it right, by explicitly enabling DNS over HTTPS in a sysadmin's test Firefox and then trying things with and without the canary domain. This failed and left us very puzzled, and it was only through a lucky bit of happenstance that we kind of discovered what seems to be going on (although what's going on is documented if you pay attention). So here are some notes.
First and most importantly, the canary domain is ignored if you've explicitly enabled DNS over HTTPS. We found this out from a tweet by Jan Schaumann (via), who filed a Mozilla bug over this behavior. The result of the bug was to cause Mozilla to update their documentation to mention this, both here (in a bit that's easy to miss in passing) and here (where it's made more obvious). The Mozilla bug (bug #1614751) contains a way of manipulating about:config settings to pretend that Firefox enrolled you in DoH so that you can test that you properly added the canary domain to your DNS resolver, in comment #1.
(It's possible that Mozilla will someday be persuaded to disable DoH when the canary domain is present no matter what the user asked for. In the mean time, people who have explicitly turned on DoH won't be able to connect to some of the web servers that we host due to our split horizon DNS.)
To make your life more confusing when you're testing, Firefox never uses DNS over HTTPS for domains in your DNS suffix list (which can come from DHCP or be explicitly configured, for example in /etc/resolv.conf on Unix systems). This can mean that either you need to manipulate your host settings to scrub out your usual DNS suffix list or you need a split horizon hostname that is not under them. Fortunately we were able to find some eventually, which allowed us to see that Firefox was still looking them up with DoH despite the canary domain being theoretically configured.
While testing Firefox, you can look at the state of its DNS stuff in about:networking. The 'DNS' tab will show you what Firefox thinks are your DNS suffixes and what names it has recently resolved, with or without DNS over HTTPS (the 'TRR' column, which is true if DoH was used). You can also directly do address lookups with the 'DNS Lookup' tab; addresses looked up here show up in the 'DNS' tab, so you can see if they were resolved with DNS over HTTPS (if the IP address isn't a sufficient sign).
I believe that Mozilla no longer documents any specific claims about what Firefox will do to detect names and situations where DNS over HTTPS doesn't work. Empirically, using an internal top level domain (we use '.sandbox') appears to result in Firefox not using DoH for the lookup, but I don't know if this happens because Firefox knows that this TLD doesn't exist or because it does a DoH lookup, fails to find the name, and retries through the local resolver.
(I can think of ways to find out, but they require more work than I want to bother with and anyway, Mozilla is likely to change all of these behaviors over time.)
Logging out of HTTP Basic Authentication in Firefox
We make significant use of HTTP Basic Authentication in our local web servers, because if you use Apache it's a nice simple way of putting arbitrary things behind password protection. It's not the most user-friendly thing these days and it's probably not what you want if you also need to handle things like user registration and password resets, but in our environment all of those are handled separately. However, it does have one little drawback, which is logging out.
Normal user web authentication schemes are pretty much all implemented with browser cookies and often a backend session database. This means that 'logging out' is a matter of removing the cookies, marking the session as logged out, or both at once. Logging you out is a straightforward and unobtrusive thing for a website to do (and it can even do so passively), and even if a website doesn't support logging out you can do it yourself by scrubbing the site's cookies (and Firefox is making it increasingly easy to do that).
There is no equivalent of this for HTTP Basic Authentication. The browser magically remembers the authentication information for as long as its running, there's no way for the website to gracefully signal to the browser that the information should be discarded, and Firefox doesn't expose any convenient controls for it to the user (Firefox doesn't even seem to consider HTTP Basic Authenticaion to be 'cookies and site data' that it will let you clear). Traditionally the only way to 'log out' from HTTP Basic Authentication was to quit out of your entire browser session, which is a bit obtrusive in these days of tons of windows, tabs, and established sessions with other websites.
Recently I learned that you can do better, although it's a bit obtrusive and not particularly user-friendly. The magic trick is that you can overwrite Firefox's remembered HTTP Basic Authentication user and password with a new, invalid pair by using a URL with embedded credentials. If you're currently authenticated to https://example.com/app, you can destroy that and effectively log out by trying to access 'https://bad:email@example.com/app'. The drawback is that you'll get an authentication challenge popup that you have to dismiss.
(Chrome apparently no longer supports specifying credentials in URLs this way, so this trick won't work in it. Hopefully Firefox is not going to go the same way, at least not before it adds some sort of UI to let you discard HTTP Basic Authentication credentials yourself. MDN does describe this as deprecated in the MDN page on HTTP authentication, so it may be going away someday even in Firefox.)
You can definitely enter such an URL by hand (or modify the existing URL
of the page in the URL bar to insert the '
bad@bad:' credentials bit)
and it works. I believe that Firefox will still support links on web
pages that have credentials embedded in them, so you could put a 'log
out by trying to use this' link on your index page, but I haven't tested
it. You'd only want to do this on websites aimed at technical users,
because following such a link will provoke a HTTP Basic Authentication
challenge that you have to cancel out of.
PS: You can apparently clear this information through 'Clear Recent History' by invoking that and then carefully selecting only 'Active Logins'. Since clearing any degree of history is an alarming thing for me and a mistake could be very bad (I keep my browser history forever), I'm not fond of this interface and don't expect to ever use it. People who are less attached to their browser history than I am (and so are more tolerant of slips than I am) may like it more.