Wandering Thoughts archives

My view of setting up sane web server application delegations

One of the things that drives the appeal of CGI scripts is the easy deployment story compared to, say, a bunch of programs that implement web applications is the deployment configuration problem. When you have a bunch of applications on the same machine (and using the same IP), you need a central web server to take incoming requests and dispatch them out to the appropriate individual web app (based on incoming host and URL), and this central web server needs to be configured somehow. So how do you do this in a way that leads to easy, sane deployment, especially in a multiple user environment where not everyone can sit there editing central configuration files as root?

My views on this have come around to the idea that you want some equivalent of Apache per-directory .htaccess files in a directory structure that directly reflects the hosts and URLs being delegated. There are a couple of reasons for this.

First, a directory based structure creates natural visibility and enforces single ownership of URL and host delegations. If you own www.fred.com/some/url, then you are in charge of that URL and everyone underneath it. No one can screw you up by editing a big configuration file (or a set of them) and missing that /some/url has already been delegated to you somewhere in hundreds of lines of configuration setup; your ownership is sitting there visible in the filesystem and taking it over means taking over your directory, which Unix permissions will forbid without root's involvement.

Second, using some equivalent of .htaccess files creates delegation of configuration and control. Within the scope of the configuration allowed in the .htaccess equivalent, I don't need to involve a sysadmin in what I do to hook up my application, control access to it, have the native master web server handle some file serving for me, or whatever. Of course the minimal approach is to support none of this in the master server (with the only thing the .htaccess equivalent can do is tell the master server how to talk to my web app process), but I think it's useful to do more than that. If nothing else, directly serving static files is a commonly desired feature.

(Apache .htaccess is massively powerful here, which makes it quite useful and basically the gold standard of this. Many master web servers will probably be more minimal.)

To the extent that I can get away with it, I will probably configure all of my future Apache setups this way (at least for personal sites). Unfortunately there are some things you can't configure this way in Apache, often for good reason (for example, mod_wsgi).

(This entry is inspired by a Twitter conversation with @eevee.)

Sidebar: doing this efficiently

Some people will quail at the idea of the master web server doing a whole series of directory and file lookups in the process of handling each request. I have two reactions to this. First, this whole idea is probably not appropriate for high load web servers because on high load web servers you really want and need more central control over the whole process. If your web server machine is already heavily loaded, the last thing you want to do is enable someone to automatically set up a new high-load service on it without involving the (Dev)Ops team.

Second, it's possible to optimize the whole thing via a process of registering and (re)loading configuration setups into the running web server. This creates the possibility of the on-disk configuration not reflecting the running configuration, but that's a tradeoff you pretty much need to make unless you're going to be very restrictive. In this approach you edit your directory structure and then poke the web server with some magic command so that it takes note of your change and redoes its internal routing tables.

It's time for me to stop using lighttpd

There's another SSL configuration vulnerability going around; this one is called Logjam (also). Part of the suggested fixes for it is to generate your own strong Diffie-Hellman group instead of using one of the default groups, and of course another fix is yet more SSL parameter fiddling. There have been quite a lot of SSL/TLS related issues lately, and many of them have required SSL parameter fiddling at least in the short term.

I've had a long-standing flirtation with lighttpd and my personal site has used it since the start. But this latest SSL issue has crystallized something I've been feeling for a while, which is that lighttpd has not really been keeping up with the SSL times. Lighttpd cannot configure or de-configure a number of things that people want to; for example, it has no option to disable TLS v1.0 or SSL compression (although the latter is probably off in OpenSSL by now). OCSP stapling? You can forget it (from all appearances). In general, the last release of lighttpd 1.4.x was a year ago, which is an eternity in SSL best practices land.

For a while now I've been telling people when they asked me that I couldn't recommend lighttpd for new deployments if they cared about SSL security at all. Since I care increasingly much about SSL myself, it's really time for me to follow my own advice and move away from lighttpd to something else (Apache is the most likely candidate, despite practical annoyances in my environment). It'll be annoying, but in the long run it will be good for me. I'll have a SSL configuration that I have much more trust in and that is much better supported by common resources like Mozilla's SSL configuration generator and configuration guidelines.

There's certainly a part of me that regrets this, since lighttpd is a neat little idea and Apache is kind of a hulking monstrosity. But in practice, what matters on the Internet is that unmaintained software decays. Lighttpd is in practice more or less unmaintained, while Apache is very well maintained (partly because so many people use it).

(Initially I was going to write that dealing with Logjam would push me over the edge right away, but it turns out that the Logjam resources page actually has settings for lighttpd for once.)

On the modern web, ISPs are one of your threats

Once upon a time, it was possible to view the Internet as a generally benevolent place as far as your traffic was concerned. Both passive eavesdroppers and man in the middle attacks were uncommon and took generally aggressive attackers to achieve (although it could be done). Eavesdropping attacks were things you mostly worried about on (public) wifi or unusual environments like conference networks.

I am afraid that those days are long over now. On the modern Internet, ISPs themselves are one of your threats (both your ISP and other people's ISPs). ISPs routinely monitor traffic, intercept traffic, modify traffic on the fly both for outgoing requests (eg) and for incoming replies from web servers ('helpfully' injecting hostile JavaScript and HTML into pages is now commonplace), and do other malfeasance. To a certain extent this is more common on mobile Internet than on good old fashioned fixed Internet, but this is not particularly reassuring; an increasing amount of traffic is from mobile devices, and ISPs are or will be adding this sort of stuff to fixed Internet as well because it makes them more money and they like cash.

(See for example the catalog of evil things various ISPs are doing laid out in We're Deprecating HTTP And It's Going To Be Okay (via). Your ISP is no longer your friend.)

The only remedy that the Internet has for this today is strong encryption, with enough source authentication that ISPs cannot shove themselves in the middle without drastic actions. This is fundamentally why it's time for HTTP-only software to die; the modern Internet strongly calls for HTTPS.

This is a fundamental change in the Internet and not a welcome one. But reality is what it is and we get to deal with the Internet we have, not the Internet we used to have and we'd like to still have. And when we're building things that will be used on today's Internet it behooves us to understand what sort of a place we're really dealing with and work accordingly, not cling to a romantic image from the past of a friendlier place.

(If we do nothing and keep naively building for a nicer Internet that no longer exists, it's only going to get worse.)

It's time to stop coddling software that can't handle HTTPS URLs

A couple of years ago I moved my personal website from plain HTTP to using HTTPS. When I did that, one of the lessons I learned was that there were a certain number of syndication feed fetchers that didn't support HTTPS requests at all. My solution at the time was to sigh and add some bits to my lighttpd configuration so they'd be allowed to still fetch the HTTP version of my syndication feeds Now I'm in the process of moving this blog from HTTP to HTTPS and so I've been considering what I'll do about issues like this for here. This time around my decision is that I'm not going to create any special rules; anything fetching syndication feeds or web pages from here that can't do HTTPS (or follow redirections) is flat out of luck.

There are some pragmatic reasons for this, but ultimately it comes down to that I think it's now clearly time that we stopped accepting and coddling software that can only deal with HTTP URLs. The inevitable changes of the Internet have rendered such software broken. It's clear that HTTPS is increasingly the future of web activity and also clear that a decent number of sites will be moving to it via HTTP to HTTPS redirection. Software that cannot cope with both of these is decaying; the more sites that do this, the more pragmatically broken the software is.

I'm not going to say that you should never give in and accommodate decaying, broken software; if nothing else, I certainly have made some accommodations myself. But when I do that, I do it on a case by case basis and only when I've decided that it's sufficiently important; I don't do it generally. Coddling broken software in general only prolongs the pain, not just for you but for everyone. In this case, the more we accommodate HTTP only software the more traffic remains HTTP (and subject to snooping and alteration) instead of moving to HTTPS. HTTPS is not ideal, but it's clear that an HTTPS network is an improvement over the HTTP one we have today in practice.

This is likely going to hurt me somewhat (and already has, as some Planets (also) that carry Wandering Thoughts apparently haven't coped with this). But even apart from the pragmatic impossibility of trying to pick through all of the request to here to see which aren't successfully transitioning to HTTPS, I'm currently just not willing to coddle such bad software any more. It's 2015. You'd better be ready for the HTTPS transition because it's coming whether you like it or not.

The reason I feel like this now when I didn't originally is pretty simple: more time has passed. The whole situation with HTTP and HTTPS on the Internet has evolved significantly since 2013, and there is now real and steadily increasing momentum behind the HTTPS shift. What was kind of wild eyed and unreasonable in 2013 is increasingly mainstream.