Wandering Thoughts archives

I love Apache (well, like it at least)

There is somewhat of a meme around the sysadmin and web world that Apache is a bad web server that no one should use, or at least not use for very long. And if you're using Apache or thinking about configuring something with Apache, maybe you should think about how to migrate away or pick a setup that will make that easy. I once sort of felt this way and experimented with (and still use) alternatives like lighttpd, but I no longer do so; these days I think that Apache is your best default choice for a web server.

I will put it like this: choosing Apache is a lot like choosing to write code with a high-level 'scripting' language like Python or Ruby instead of C++, Java, or Go. The result is not always as fast as it could be but often it will be pretty good (sometimes excellent), much of what most people are doing doesn't need the flat out speed, and the whole thing is easy to set up, quite convenient, and does a lot for you, including things you may not initially realize that you want or need.

Apache is not likely to be the highest performing or least resource using web server you can configure (although you might be surprised). However in real life most of the web sites most people set up do not need to support massive loads (and when they do you may find that the bottlenecks are not in the web server). And what Apache gives you is almost unmatched support for easily set up and deployed web app systems. CGI scripts are very simple; PHP files can simply be dropped into place; Python applications need only mod_wsgi; even Ruby on Rails apparently has an Apache module. If you want to use FastCGI and app daemons and other higher-capacity things, well, Apache supports that too. Not infrequently it will give your simple applications practical performance boosts, for example by compressing their HTTP responses when possible.

(In my personal interest of TLS security I've also wound up feeling that Apache is in practice at the forefront of allowing you to set up good TLS configurations. Other web servers can have spotty support for various things, but Apache is almost always there. These days I recommend that people who care about TLS look very carefully at anything other than Apache.)

Apache can have what I've called the Apache configuration tax, where for simple setups (like static file service) other web servers can have simpler setups with less configuration work to do. On the other hand I've found that sensible Apache configurations create very simple Apache setups. And any complex server design is probably going to be complex to express in a configuration, regardless of what web server you're using.

All of this leaves me feeling three things about Apache. First, Apache is a perfectly sensible choice as your web server, one that shouldn't need defending any more than a choice of a more sexy and talked about web server like nginx (or lighttpd, which no longer seems to be the latest hotness). Second, Apache probably should be your default choice of web server. Unless you already have special expertise and tooling for other web servers or unless you can clearly see why Apache is a bad fit for your target environment, just going with Apache will probably make your life easier and work just as well. And finally, that Apache deserves more love in general. It's really time for Apache to (re)take its place as a perfectly respectable web server, not the embarrassing relative we don't talk about.

(I'll admit that this is kind of a rant because I've decided that I don't want to feel defensive any time I talk about using Apache. But my views have really shifted on this over time, as I used to share the common attitude that Apache was outdated or at least a catastrophic performer (not really, actually, to my surprise in that situation). My personal site still runs lighttpd, but I'm not convinced that that's the right decision; it persists partly out of inertia and partly because Fedora's Apache configuration setup is terrible.)

How a Firefox update just damaged practical security

Recently, Mozilla pushed out Firefox 34 as one of their periodic regular Firefox updates. Unfortunately this shipped with a known incompatible change that broke several extensions, including the popular Flashblock extension. Mozilla had known about this problem for months before the release; in fact the bug report was essentially filed immediately after the change in question landed in the tree, and the breakage was known when the change was proposed. Mozilla people didn't care enough to do anything in particular about this beyond (I think) blacklisting the extension as non-functional in Firefox 34.

I'm sure that this made sense internally in Mozilla and was justified at the time. But in practice this was a terrible decision, one that's undoubtedly damaged pragmatic Firefox security for some time to come. Given that addons create a new browser, the practical effect of this decision is that Firefox's automatic update to Firefox 34 broke people's browsers. When your automatic update breaks people's browsers, congratulations, you have just trained them to turn your updates off. And turning automatic updates off has very serious security impacts.

The real world effect of Mozilla's decision is that Mozilla has now trained some number of users that if they let Mozilla update Firefox, things break. Since users hate having things break, they're going to stop allowing those updates to happen, which will leave them exposed to real Firefox security vulnerabilities that future updates would fix (and we can be confident that there will be such updates). Mozilla did this damage not for a security critical change but for a long term cleanup that they decided was nice to have.

(Note that Mozilla could have taken a number of methods to fix the popular extensions that were known to be broken by this change, since the actual change required to extensions is extremely minimal.)

I don't blame Mozilla for making the initial change; trying to make this change was sensible. I do blame Mozilla's release process for allowing this release to happen knowing that it broke popular extensions and doing nothing significant about it, because Mozilla's release process certainly should care about the security impact of Mozilla's decisions.

Browser addons can effectively create a new browser

In many ways, what a browser is is defined by its user interface. These days all browsers display web pages and run JavaScript; what really differentiates them is the experience of using them. The logical consequence of this is that there are any number of browser addons that change either the user interface itself or just the experience of using the browser to such a degree that you can wind up with what might as well be a different browser. Your browser is still based on Chrome or Firefox but it is not Chrome or Firefox as such.

(Certainly this is the case for me with my extensions. Adding gestures to Firefox significantly changes the UI I experience, while NoScript and other screening tools give me a much different and more pleasant view of the web.)

The direct consequence of this is that in many cases, people's core addons are not optional. If your addons stop working, what you wind up with is effectively a different browser; its UI is different, its behavior is different. This means that from a user's perspective, breaking addons can be the same as breaking the browser. Regardless of the technical details about what happened, you wind up in a browser that doesn't work right, one that no longer behaves the way it used to.

(A corollary is that once your browser is broken, you may well have no particular reason to stay with the underlying base it was built on. Humans being humans, you are probably much more likely to feel angry that your browser has been broken and switch to a browser from some other people.)

This is of course not the case for all addons or all people. Some addons have too small an effect, and not everyone will do much with addons that can have major effects on the UI or the browsing experience. But even small addons may have big effects for some people; if you use an addon that tweaks a site that you use all the time in a way that strongly affects your use of the site, you've kind of got a different browser. Certainly losing the addon would significantly change your experience even though that one site is only a small part of the web. I'm sure there are people using extensions related to the big time-consuming social websites who fall into this category.

(If you install a gestures extension but rarely use gestures, or install NoScript but whitelist almost everything you run into, you're not really changing your browsing experience much from the baseline browser.)

The unreasonable effectiveness of web crawlers

I have a few test copies of Wandering Thoughts and all of CSpace sitting around here and there; I use them for things like trying out new CSS and other layout stuff, playing with code changes, testing the full weight of CSpace in different web environments, and so on. As it happens, one of those copies sometimes exists on my personal domain. I don't link to these copies from anywhere, of course, as they're test things and I access them from direct URLs. So you can imagine my surprise when one day I discovered that Googlebot and several other crawlers were rummaging through that copy on my personal domain. Of course I became very curious about how they could possibly have found it.

The answer turned out to be that lurking in the DWiki install on my personal domain was a single stray file copied over from CSpace that had a link to '/~cks/'. This link had been there for years but equally had led to nothing for those years until I brought up the test install and left it there. Crawlers had been trying the link all that time and getting 404s on it, but within a few days of the link switching to working Googlebot tried the URL again, found a page there, and started crawling through the links it found (another crawler also showed up). And it was crawling quite enthusiastically at that, not going all that slowly.

(Fortunately I noticed almost immediately and turned the whole thing off again. This was mostly luck; I was watching the logs because I'd been doing some experimentation, so I actually noticed the explosion in traffic volume. Normally I don't look at the logs there for long periods of time.)

What this has shown me rather vividly is that web crawlers are unreasonably effective. If there's a link to something lurking somewhere, no matter how obscure, they're likely to find it, follow it, and crawl everything behind it. Of course I already knew this in theory, since there have been all sorts of stories over the years of search engines indexing things that no one expected them to stumble over (or to stumble over that fast), but it's one thing to read the stories and another thing to have it happen to you.

(The next time around I'll try to remember to put access restrictions up for whatever I'm testing. And to do it before I bring up the test setup.)