Wandering Thoughts: Recent Entries

Some stuff on 'time since boot' timestamps

From today on Twitter:

@standaloneSA: Is it just me, or does it seem silly that the #NetFlow timestamp field for the flow references "ms since the router booted". Seems obtuse.

@thatcks: @standaloneSA It's probably easy to implement in the router and it creates an absolute ordering w/o worries about time going backwards.

In the Twitter way, this is a little bit cryptic so I'm going to elaborate on my guess here.

Suppose that routers were supposed to generate an absolute timestamp for their events instead of this relative one, for example UTC in milliseconds. This would create two problems.

First, routers would somehow need to know or acquire the correct UTC time (with millisecond resolution) and then maintain it. This is to some degree a solved problem but it adds complexity to the router. It also leads to the second problem, because a router is unlikely to boot with the correct UTC time (down to the millisecond).

The second problem is that the moment you have a system generating an absolute timestamp you need to deal with the certainty that the correct time, as the system sees it, will jump around. The router will boot will some idea of the UTC time but it's quite likely to be a bit off (remember that we're calling for millisecond accuracy here), then over time it will converge on the correct UTC time. As it does so, its version of UTC time may go forward abruptly, go backwards abruptly, or go forward more slowly than UTC time is really advancing. Backwards time jumps screw up event ordering completely, and all of the options screw up the true relative time between events; if you have two events timestamped UTC1 and UTC2, you actually have only a weak idea how long it is between them.

The valuable property that milliseconds since boot has is that it is a clear monotonic timestamp. It only ever goes forward and it goes forward at what should be a very constant rate, which means that it creates a clear order of events and a clear duration between any two events (well, for events from the same stream of monotonic timestamps). Monotonic timestamps are not a substitute for absolute time but neither is absolute time a substitute for monotonic timestamps; you really need both, which means that you need a map between them.

There are two possible places to build such a map: each device can do its own or it can be done in a central aggregator. I believe that the right answer is to do it in the central aggregator because this means that you have only a single version of absolute time, the aggregator's view (each device, aggregator included, may have a slightly different view of the current 'correct' absolute time for the reasons outlined above). Using only a single version of absolute time means that you have a single coherent map of all of the monotonic timestamps to (some) absolute time.

(Of course you need devices that generate monotonic timestamps to tell you when they reset their timestamps, eg when they boot.)

My impression is that using elapsed time since boot is actually common in a number of environments. For example, Linux kernel messages are usually reported this way these days (which has its own issues if you're trying to work backwards to roughly when in absolute time something happened).

My Firefox 12 extensions and addons

In light of yesterday's entry about my failed Firefox Nightly experiment and the potential that some of my extensions are the root cause of my Firefox problems, I'm going to run down the current set of Firefox extensions that I use in my main browser (updating previous discussions from the Firefox 7 era, which alarmingly was less than a year ago). This time around I'm going to group them by purpose:

Safe browsing:

• NoScript to disable JavaScript for almost everything. I browse with JS blocked and only enable it selectively on sites when I have to (and almost always temporarily). I consider this more an issue of safety than of performance; I simply don't trust most JavaScript from most sites to not do things that will make me unhappy.

  (NoScript also takes care of blocking Flash, Java, and so on.)

• CookieSafe 3.0.5, with the actual addon here. I browse through a filtering proxy and it blocks ordinary cookies, but it can't do anything about cookies I get over HTTPS or via JavaScript. I use CookieSafe to block those (there's some more explanation here).

  (For me, CookieSafe 3.1a10 has an explosive interaction with NoScript that hangs Firefox in some sort of infinite JavaScript loop, so I am still on 3.0.5 aka the 2011-12-10 version of CookieSafe.)

User interface:

• All-in-One Gestures (specifically my tweaked version of it). I turn off A-i-O autoscroll because the native Firefox autoscroll is better (and has been for years). A-i-O hasn't been updated in ages but still seems to be the best, most reliable gesture extension in my brief experimentation.

  (FireGestures is actively developed but the last time I tried it there was an odd bug with changing font size settings; however, that was a while back. It would be my leading alternate here.)

• Status-4-Evar restores the old Firefox bottom status bar so that I can see the full display of link targets and have a useful page load status display.

Fixing annoying websites, especially Google's:

• GreaseMonkey combined with the Google Link Cleanup user script to remove Google's tracking links from search results. I hate these tracking links with a burning passion for two reasons; first, I have no interest in letting Google know what search results I've followed and second, Google's tracking links screw up my history so that I can't see which search results I've already read and which are new.

• Stylish combined with a number of mostly personally written styles to fix various website misdesigns. The most important is a version of this user style to disable the left option sidebar in Google searches (because I hate it and I use Google all the time). I also have Compact Google Reader in the Firefox instance I use with Google Reader, for similar reasons.

  (This entry and its comments have a bunch of discussion about ways to fix Google's layout issues.)

  I could probably replace my use of Stylish with more GreaseMonkey user scripts, but I started with Stylish and I prefer fixing things with CSS alterations than with JavaScript (even if the JavaScript just inserts CSS alterations). Certainly there seem to be plenty of 'fix Google stuff' GreaseMonkey user scripts, eg this one for Google Reader (which I have not tried).

Improving my life:

• It's All Text! handily deals with how browsers make bad editors. The more I have it available the more I use it (and the longer comments and so on I wind up leaving, because I can actually edit them sensibly; this may not be a plus, all things considered).

Modern versions of Firefox also give you a JavaScript based PDF viewer addon for free. I have not done much with this and in fact currently have it turned off.

Of these extensions, I consider NoScript, All-in-One Gestures, GreaseMonkey, and Stylish to be completely essential. I can sort of live without the others, so as an experiment I am trying that to see if it makes a difference in Firefox memory usage and the number of zombie compartments that build up. If I am serious about this, I probably should migrate away from Stylish to GreaseMonkey for everything on the grounds that the latter is probably more actively used and maintained and so any leaks it has are more likely to get fixed promptly.

(Unfortunately I suspect that A-i-O is a likely candidate to be a leaky extension, since it hasn't been updated in ages.)

My experiment with Firefox Nightly builds: a failure

Ever since my old Firefox build started crashing and I was forced to update to current versions, I've had serious memory issues with Firefox. I used to be able to leave Firefox running for weeks (or months) with basically stable memory usage. Now, Firefox will steadily bloat up from under a GB of resident memory at its initial steady state to, say, 1.5 GB in a few days at most. Although my current machine has 16 GB of RAM, Firefox progressively gets slower and slower as its resident memory grows; by the time it reaches around 1.5 to 1.6 GB resident the performance is visibly dragging and I have to restart.

Recently I stumbled across this Mozilla blog entry on Firefox memory usage, which discusses how current Firefox builds have changes that reduce memory leaks, especially a drastic reduction in zombie compartments (see this entry for more). Ever since I discovered the verbose about:memory information, I've noticed that I have zombie compartments that linger from my ordinary browsing; the longer I browse, the more zombie compartments build up. A Firefox change that actually dropped zombie compartments seemed very promising, certainly promising enough to build a current version of Firefox and see what happened.

(Thus this is not quite an experiment with the literal Nightly builds, although it should be very close; as far as I understand, they're built from the same source repository (see also) that I was using.)

Unfortunately, the experiment turned out to be mostly a failure, although a sort of interesting one; in some ways Firefox improved but in other ways it got significantly worse. I tweeted a cryptic short form version, and I feel like elaborating on it now.

What improved was Firefox's responsiveness as its resident memory grew. Firefox 12 visibly starts slowing down with as little as 1.2 or 1.3 GB of resident memory; the current Firefox code was still running almost as well as at start when it reached 2 GB or more of resident memory, and it might have kept going even as it bloated more. What did not improve was everything else. I still saw zombie compartments (probably just as many as before) and if anything Firefox memory usage grew faster than under Firefox 12, reaching 2 GB resident in a day or two. But the worse thing was that at home, Firefox would soon get into a state where it was constantly using CPU (apparently talking with the X server). In this state it would not shut down gracefully; I could quit Firefox and it would close all its windows, but the process would not exit and would continue consuming the CPU talking with the X server.

(I had to use 'kill -9' to get it to exit, and this happened more than once with builds across several days. It was also odd CPU usage; it showed clearly in top but did not affect the load average and didn't lag the X server that I could tell.)

Unclean shutdowns aren't something that I considered acceptable in this situation so I am now back to Firefox 12, memory bloat slowdown and all.

It's possible that the current Firefox codebase will improve as it marches towards release, eliminating the memory bloat and 100% CPU usage while preserving responsiveness as its memory usage grows. I could live with that and it certainly would be an improvement over the status quo. (In some ways, simply eliminating the CPU usage would be a bit of an improvement over the status quo, although I don't like Firefox consuming several GB of my RAM for no good reason.)

(Despite the result, I don't regret doing this experiment; it was worth trying and it didn't particularly explode in my face.)

Sidebar: dealing with this with Chrome or by disabling extensions

Chrome is not something I consider an acceptable alternative to Firefox, so switching to it is not an option.

One piece of advice the Mozilla people give about this sort of memory bloat is 'disable unnecessary addons'. Well, I don't have any of those; all of the addons I have loaded are ones that I consider either absolutely necessary (to the point where I would not browse without them) or important for how I use Firefox.

(I suppose there's one or two that I don't use very often, like It's All Text!, but it would be actively painful periodically.)

A basic step in measuring and improving network performance

There is a mistake that I have seen people make over and over again when they attempt to improve, tune, or even check network performance under unusual circumstances. Although what set me off now is this well intentioned article, I've seen the same mistake in people setting off to improve their iSCSI performance, NFS performance, and probably any number of other things that I've forgotten by now.

The mistake is skipping the most important basic step of network performance testing: the first thing you have to do is make sure that your network is working right. Before you can start tuning to improve your particular case or start measuring the effects of different circumstances, you need to know that your base case is not suffering from performance issues of its own. If you skip this step, you are building all future results on a foundation of sand and none of them are terribly meaningful.

(They may be very meaningful for you in that they improve your system's performance right now, but if your baseline performance is not up to what it should be it's quite possible that you could do better by addressing that.)

In the very old days, the correct base performance level you could expect was somewhat uncertain and variable; getting networks to run fast was challenging for various reasons. Fortunately those days have long since passed. Today we have a very simple performance measure, one valid for any hardware and OS from at least the past half decade if not longer:

Any system can saturate a gigabit link with TCP traffic.

As I've written before in passing, if you have two machines with gigabit Ethernet talking directly to each other on a single subnet you should be able to get gigabit wire rates between them (approximately 110 MBytes/sec) with simple testing tools like ttcp. If you cannot get this rate between your two test machines, something is wrong somewhere and you need to fix it before there's any point in going further.

(There are any number of places where the problem could be, but one definitely exists.)

I don't have an answer for what the expected latency should be (as measured either by ping or by some user-level testing tool), beyond that it should be negligible. Our servers range from around 150 microseconds down to 10 microseconds, but there's other traffic going on, multiple switch hops, and so on. Bulk TCP tends to smooth all of that out, which is part of why I like it for this sort of basic tests.

As a side note, a properly functioning local network has basically no packet loss whatsoever. If you see any more than a trace amount, you have a problem (which may be that your network, switches, or switch uplinks are oversaturated).

The one area today where there's real uncertainty in the proper base performance is 10G networking; we have not yet mastered the art of casually saturating 10G networks and may not for a while. If you have 10G networks you are going to have to do your own tuning and measurements of basic network performance before you start with higher level issues, and you may have to deliberately tune for your specific protocol and situation in a way that makes other performance worse.

The death of paging on the web

I've written about the problem of permanent headers and footers before (around a year ago), but I'm seeing more and more of them these days. What this confirms for me is that paging is dead on the modern web.

By this I don't mean long pages; I'm not one of those people who think that all of your content has to be 'above the fold', immediately visible as what people see (and the available evidence from actual experimentation apparently says otherwise). What I mean is getting to that content by paging, advancing in nearly full page increments (usually by hitting the space bar in your browser). Given that permanent headers or footers (or both) screw this up, and given that permanent headers and footers are increasingly popular, I can only conclude that paging isn't really used any more; otherwise, header and footer based designs would be wretched experiences and test badly (and on the modern web, people do at least do A/B tests).

Instead, I think that on the modern web everyone has scroll wheels (or some other way of scrolling, for example on tablets) and they scroll through articles and pages with them. Only an insignificant number of people still navigate with paging.

Now I'll add a personal confession here: since I started my scroll wheel mouse experiment, I've found myself increasingly scrolling web pages instead of paging them. I don't know why, but there's just something about it that feels right (and this is on pages without obnoxious headers and footers). I think that part of it is that the boundaries of things on the web page often don't align naturally with what I'd get by paging; by partially scrolling the page I can make things line up right (this is especially visible to me if the page content includes images).

(Looking back, I've had middle mouse button based scrolling in my browser for years and have used it too instead of paging. So I should have seen this one coming.)

I don't know what this means for web page design going forward, but I suspect that it means something (I also suspect that current web designers do know what it implies; I am not exactly current on the field). There have to be things you design differently if you expect almost everyone to scroll your page around so that things can catch their eye as they move past.

(I probably won't ever put a permanent header or footer on a page I design (at least not a full-width one), but that's a personal thing. Also it would have to be something awfully important to the page to deserve a permanent full-time presence in front of the viewer. My bias is that almost all headers and footers I've seen aren't that important; in fact, they're often rather presumptuous that way, which is part of the reason I dislike them.)

All your servers should have Linux's magic SysRq enabled

This is effectively another lesson learned from our recent building power shutdown. I will put it simply:

All of your servers should have magic SysRq enabled.

There are reasons to not do this on client machines (but not necessarily very good ones), but none on your servers (which certainly should have their hardware and consoles in a secure location).

What magic SysRq is good for on servers (above everything else) is giving you a last ditch chance to shut down or reboot the machine in something approaching an orderly way. I'm not just talking about if the system goes crazy, because it's also quite possible for ordinary system shutdowns to hang, especially if you're shutting down a group of systems that have complex NFS filesystem relationships and something went down out of order. If this happens and you don't have magic SysRq support available, you're plain out of luck; all you can do is pull the power and hope that nothing is going to explode because it hasn't been killed, had its data synced to disk, or whatever.

With magic SysRq you have at least a chance of doing something about this. You can force a kernel level sync, a kernel level unmount of as many filesystems as possible, and even hit processes with signals if you think it's going to do any good. And then you can reboot the machine (and afterwards, possibly pull the power to keep the machine down).

PS: you should explicitly enabled magic SysRq in your standard server install setup, even if your distribution normally defaults to leaving it on; distribution defaults can change over time. Also, note that if you have a serial console you generally need a getty listening on it in order to make magic SysRq work.

(You can check to see if magic SysRq is enabled by looking at the value of /proc/sys/kernel/sysrq; a 1 means that it is, a 0 means that it isn't.)

Using rsync to pull a directory tree to client machines

Suppose that you have a decent sized directory tree that you want some number of clients to mirror from a master server (with the clients pulling updates instead of the master pushing them), perhaps because you've just noticed undesired NFS dependencies. Things in the directory tree are potentially sensitive (so you want access control), it's updated at random, and it's not in a giant VCS tree or something; this is your typical medium-sized ball of local stuff. The straightforward brute force approach is to use rsync with SSH; give the clients special SSH identities, put them in the server's authorized_keys, and have them run 'rsync -a --delete' (or some close variant) to pull the directory tree over. However, this has the problem that normal rsync is symmetric; if you allow a client to pull from you, you also allow a client to push to you (assuming that the server side login has write access to the directory tree, and yes let's make that assumption for now).

(You also have to set the SSH access up so that the clients can't run arbitrary commands on the server.)

Rsync's solution to this is its daemon mode, which can restricted to operate in read only mode. Normally rsync wants to be run this way as an actual daemon (listening on a port and so on), but that requires us to use rsync's weaker and harder to manage authentication, access control, and other things. I would rather continue to run daemon mode rsync over plain SSH and take advantage of all of the existing, proven SSH features for various things.

(The rsync manpage suggests hacks like binding the rsync daemon to only listen on localhost on the server and then using SSH port forwarding to give clients access to it. But those are hacks and require making various assumptions.)

How to to do this is not obvious from the documentation, so here is the setup I have come up with for doing this on both the server and the clients. First, you need an rsyncd.conf configuration file on the server. Don't use the normal /etc/rsyncd.conf; it's much more controllable to use your own in a different place. It should look something like:

use chroot = no

[somepath]
comment = Replication module
path = /some/path
read only = true
# if necessary:
uid = 0
gid = 0

(The '[somepath]' bit is what rsync calls the module name and can be anything meaningful for you; you'll need it on the client later. The comment is optional but potentially useful. You need to explicitly specify uid and gid if the server login is UID 0 for access to the directory tree and you need to keep that; otherwise rsync will drop privileges to a default UID.)

Next, you need a script on the server that will force an incoming SSH login to run rsync in daemon mode against this configuration file and do nothing else. We will set this as the command= value in the server login's authorized_keys to restrict what the incoming SSH connection from clients can do. This looks like:

#!/bin/sh
exec /usr/bin/rsync --server --daemon --config=/your/rsyncd.conf .

Note that this completely ignores any arguments that the client attempts to supply. However, this doesn't matter; as far as I can tell, the command line that the clients send will always be 'rsync --server --daemon .', regardless of what command line options and paths you use on the clients. (Certainly this is the only command line that clients seem to send for requests that you actually want to pay attention to.)

On the server, the login that you're using for this should have a .ssh/authorized_keys file with entries for the client SSH identities. These entries should all force incoming logins to run the command above and block various other activities (especially port forwarding, which could otherwise be done without command execution at all as Dan Astoorian mentioned in a comment here):

command="/your/rsyncd-shell",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty [...]

A from="..." restriction is optional but potentially recommended. Even a broad one may limit the fallout from problems.

Finally, on the client you need to run rsync with all of the necessary arguments. You probably want to put this in a script:

#!/bin/sh
rsync -a --delete --rsh="/usr/bin/ssh -i /client/identity" LOGIN@MASTER-HOST::somepath /some/path/

Potentially useful additional arguments for rsync are -q and --timeout=<something>. In a production script you probably also want an option to mirror the directory tree to somewhere other than /some/path on the client.

If you run this from cron, remember to add some locking to prevent two copies from running at once. If the directory tree is large and you have enough clients, you may want to add some amount of randomization of the start times for the replication in order to keep load down on the master server.

(There may be a better way to do this with rsync; if you know of one, let me know in the comments. For various reasons we're probably not interested in doing this with any other tool, partly because we already have rsync and not the other tools. Another tool would have to be very much better than rsync to really be worth switching to.)

Things I will do differently in the next building power shutdown (part 2)

Back at the start of last September, we had an overnight building wide power shutdown in the building with our machine room and I wrote a lessons-learned entry in the aftermath. Well, we just had another one and apparently I didn't learn all of the lessons that I needed to learn the first time around. So here's another set of things that I've now learned.

Next time around I will:

• explicitly save the previous time's checklist. If nothing else, the 'power up' portion makes a handy guide for what to do if you abruptly lose building power some day.

  (I sort of did this last time, not through active planning but just because I reflexively don't delete basically any of this sort of stuff. But I should do it deliberately and put it somewhere where I can easily find it, instead of just leaving it lying around.)

  Having last time's list isn't the end of the work, because things have undoubtedly changed since then. But it's a starting point and a jog to the memory.

• start preparing the checklist well in advance, like more than a day beforehand. Things worked out in the end but doing things at the last moment was a bit nerve wracking.

  (There's always stuff to do around here and somehow it always felt like there was plenty of time right up until it was Friday and we had a Monday night shutdown.)

• update and correct the checklist immediately afterwards to cover things that we missed. My entry from last time is kind of vague; I'm sure I knew the specifics I was thinking of at the time, but I didn't write them down so they slipped away. I was able to reconstruct a few things from notes and email in the wake of last time, but others I only realized in the aftermath of this one.

• add explanatory notes about why things are being done in a certain order and what the dependencies are. Especially in the bustle of trying to get everything down or up as fast as possible, it's useful to have something to jog our minds about why something is the way it is and whether or not it's that important.

  (Our checklists for this sort of thing are not fixed; they're more guidelines than requirements. We deviate from them on the fly and thus it's really useful to have some indication of how flexible or rigid things are.)

• if any machines are being brought down and then deliberately not being brought back up, explicitly mention this so that people don't get potentially confused about a 'missing' machine.

My entry from last time was very useful in several ways. I reread it when I was preparing our checklist for this time and it jogged my memory about several important issues; as a result our checklist for this time around was (I think) significantly better than for last time (and also noticeably longer and more verbose). This time I at least made new mistakes, which is progress that I can live with.

I will also probably try to put more explanation into the checklist the next time around. I'm sure it's possible to put too much of it in, but I don't think that's been our problem so far. In the heat of the moment we're going to skim anyways, so the thing to do is to break the checklist up into skimmable blocks with actions and things to check off and then chunks of additional explanation after them.

(In a sense a checklist like this serves two purposes at once. During the power down or power up it is mostly a catalog of actions and ordering, but beforehand it's a discussion and a rationale for what needs to be done and why. Without the logic behind it being written out explicitly, you can't have that discussion; once you have that logic written out, you might as well leave it in to jog people's memories on the spot.)

On a side note, a full power up is an interesting and useful way to find problematic dependencies that have quietly worked their way into your overall network, ones that are not so noticeable when your systems are in their normal steady state. For example, DHCP service for several of our networks now depends on our core fileserver, which means that it can only come up fairly late in the power up process. We're going to be fixing that.

(There is a chain of dependencies that made this make sense in a steady state environment.)

Third party Linux kernel modules should build against non-running kernels

For my sins, I have to deal with third-party kernel modules (some of them open source, some of them not so much). When you deal with third party modules, you get to rebuild them every time you update the kernel. At one level, almost everyone has this down pat these days; you run a command or two and you're done, with everything handled properly.

(I would like to say that this is a very welcome development. I've been around Linux long enough to remember when it was much more work and pain.)

At another level almost everyone gets this wrong, because they all force you to build their modules against the current running kernel and only the running kernel. This probably sounds fine for ordinary users but it makes sysadmins angry because it makes our lives more difficult. There are two problems.

First and obviously, it increases the service downtime. When you force us to build modules against the running kernel, the upgrade process is to install the new kernel, reboot the machine to activate it (which takes down the service, not just because of the reboot but because we're without your module), sit there rebuilding your modules with the service down, and then either activate the service or even reboot the machine again. If we could rebuild against a non-running version, we could install the new kernel, rebuild your modules, and then reboot; the only downtime would be the actual reboot.

Second, it makes kernel upgrades more dangerous. With third party modules there's always the chance that the module will not rebuild against your new kernel. If we can only rebuild against the running kernel, we have to reboot the machine (taking the service down) before we find out whether or not your modules will rebuild happily. If we can build against a non-running version, we can install the new kernel, try to rebuild, and if it fails we know we don't even have to schedule a downtime because there's no point.

(Yes, in theory everyone has test machines. This is still annoying, and sometimes you actually don't.)

I somewhat sympathize with the people building third party kernel modules; I'm sure that building against the running kernel simplifies your life and it's nice to be able to immediately test that your newly compiled module can be loaded. But don't make it mandatory and do provide an 'I know what I'm doing, just compile the thing against <X>' option. Sysadmins will thank you.

As a side note, being able to rebuild kernel modules in advance makes it possible to do opportunistic kernel upgrades. If we have to reboot the machine for some other reason, or even if the machine crashes and reboots on its own, we can switch it into a new kernel in the process. This is especially valuable for unattended reboots at odd hours, where there won't be a sysadmin there to rebuild the modules by hand at the time.

(Another case is situations with mass reboots, where you don't want to be babysitting individual machines as they come up in order to get them back into service.)

Why I wound up using Django

Eevee just wrote (and I just read) a good article on Python web development. It was an interesting experience for me, because a great deal of the article is about a middle level in Python webdev that I've basically skipped right over. I've worked at the level of CGI and WSGI and I've worked in the big framework land of Django, but I've never played around in the world of relatively simple and component-ized Python web frameworks that Eevee mostly talks about (to the extent that I merged them in with Django when I wrote about modern Python web app options). And that raises an obvious question: why did I wind up using Django instead of one of the smaller and simpler alternatives?

While I could say various things, the honest answer is twofold: good PR on Django's part and 'batteries included', with the first one being more important. The good PR insured that I knew of Django and indeed thought of it as the natural choice once I knew that I needed a framework. The batteries included nature meant that I was getting everything I needed in one place, pre-integrated for me, with no omissions that might have pulled me out to look at anything else.

Do I feel bad about falling for Django's PR and batteries included monolith? No. Not at all. While I have some sympathies to the smaller and leaner frameworks (I'm ordinarily a smaller is better person), I feel that frameworks are special in a way. The thing about frameworks is that they are only a means to an end, ie a working web app. Almost all web apps are small and undemanding enough that the differences between good enough frameworks are not important; any one of them will do. What really matters is getting your application written, and for that the inclusion of batteries matters a lot.

Let me be blunt. For most people selecting a framework, the last thing they want to do is then try to pick out a templating system, a database ORM layer, a forms handling package, and so on. This is especially so because all of those things are what really gets the work done for most projects; without them, a framework is only a skeleton that does, well, not very much (it's less a framework and more a request/response handling library) and provides very little help for your project. This means that including batteries by default is quite important in practice (even if it may cause political difficulties). I don't think that you have to make the default choice the only option, but if you want me to pay your framework much attention you need something that's pre-chosen and ideally pre-assembled for me; doing otherwise means that your framework simply requires too much additional time when there are other options.

Even if Django had not had the PR edge that it does, even if it had simply been one among a number of equally well known contenders, I think I would have chosen it purely for the batteries included nature. I had a project to get done and Django offered me a single solution for everything I needed so I could immediately jump into building my actual app.

Counting your syndication feed readers

Suppose that your blog has a syndication feed, and you would like to know how many people are reading it. While you could just look at your web stats to see how many different people pull your feed, the numbers aren't you'll get won't be too useful. Some of those feed requests are from aggregators and not all aggregators report readership numbers to you (and some of them have no idea, for example planet-style aggregators that just put the result up on a web page), plus not everyone who pulls your feed is still actually reading your entries; you have to assume that a certain amount of feed subscriptions are just quietly rotting away unread.

(These effects compound, of course. I have no idea if Google Reader's reported numbers exclude people who are subscribed but not reading, for example.)

I will cut to the chase: the hack solution to this is images. You can crudely track actual entry readership by putting a (unique) image in your entry, one that you host, and then counting requests for it in your own logs. Of course this works best if you already write the sort of blog where you include images in your entries, either as part of them or simply for visual flavour and design. Otherwise people may start to wonder why your entries periodically include random images.

(There are all sorts of issues with this which make it only a crude measure. Carefully making the image uncacheable by proxies and serving cookies with it will help with some of them.)

There are at least three sorts of images you can use for this; little invisible 'web bug' images, more or less unimportant images that are just used to add visual appeal and design to entries, or images that are actual relevant parts of your entries. Setting aside the issue of being nice to your readers, the last choice is the pragmatically best one. If the images are actually an integral part of your entries your readers will make sure that they see them even in the face of feed reader issues (going so far as to visit your entry directly if necessary). Less important images can drop out without the readers caring (or even noticing), so you will undercount such people.

(Web bugs are especially likely to get screened out, of course. I wouldn't be surprised if some feed readers and aggregators do it automatically by now.)

My strong impression is that nothing other than real images inline in your entries will work reliably. Many aggregators and even feed readers apparently screen out JavaScript, much or all CSS including background images, and probably all sorts of other tricks. However screening out plain inlined images generally doesn't happen, partly because it degrades the reading experience too much to be popular.

PS: I'm sure this is well known in the part of the web world that cares about readership stats. I just feel like writing it down here for the record.