2010-08-21
A sudden realization about Unix access time updates and disk mirrors
I have a small confession: we have only recently turned off file access time updates on our fileservers. While turning off atime is one of the most common pieces of advice for how to speed up overall filesystem performance (see here for one explanation why), we haven't felt that our filesystems were particularly slow and sysadmins famously have a lot of inertia.
(What spurred me into action was being exposed to a bunch of stuff on ZFS performance tuning, which caused me to start thinking again about some long-standing minor performance issues we have from time to time.)
In the process of doing this I came to a belated realization about atime updates on filesystems that are mirrored:
Atime updates steal IOPs from all of your disks.
(In this they are not unique; all write IO does this.)
If you have an IO load that is primarily random reads (which we do overall), your limiting factor is IO operations per second; a modern disk is seek limited and can give you only somewhere between 100 and 150 IOPs a second. The usual assumption about random reads is that on average they'll keep all disks equally busy, so you get ~100 * N IOPs/sec out of your overall system.
But on a mirrored system, writes have to be written to all of the disks, which takes an IOP a disk and thus reduces the IOPs/s level of all of them. This means that atime updates can have a larger effect on your IO load than is immediately obvious. In this they are not alone; any other source of quiet asynchronous write IO can also have such an indirect impact.
(In our case it is probably still not enough to make an obvious difference, and certainly turning off atime updates didn't produce any immediately obvious performance improvements.)
This whole idea really should not have been so slow to show up in my mind, because I have certainly read about this issue a lot in the context of scaling traditional SQL databases through master/slave replication. Slave database replicas work great for dealing with read load, but you still get killed when your write load goes up since writes have to be applied to all slaves as well as on the master.
(I probably first ran into this idea in Brad Fitzpatrick's various presentations about scaling LiveJournal.)
Sidebar: backups and access time
One of the reasons that turning off access time updates was an easy decision for us is that our backup system makes them relatively useless anyways because it works by reading things through the filesystem and thus updates every file's access time when it backs them up.
The traditional two uses of access time are detecting files that haven't been looked at for a while and detecting what files have been looked at recently. Here, a file's atime is never older than the most recent backup (which will be in the past week or two), no matter how long it's been since the user looked at it, and a recently accessed file may just mean that the backup system has backed it up recently.
2010-08-15
Why I change font sizes
A commentator on my recent entry on how programs should make it easy to change the font size asked me to explain why I'd do this and said that they didn't even change the font size in Firefox, which boggles me quite a lot. When I boil it down enough, there are two reasons.
The first reason is the reason that I keep changing font sizes in Firefox: because (web) designers are crazily in love with not merely small font sizes, but absurdly small font sizes. I cannot count the number of websites that think that ridiculously small fonts are not only readable but desirable. More rarely there are websites that set things in fonts that are too big, at least on my machine. To a certain extent, this problem infects the designers of other browser-like text interfaces, although generally it can be fixed by adjusting the system default font size.
(Note that setting a minimum font size is not the answer for reasons I wrote about here.)
The second major reason is that my display isn't big enough to include everything I want to include at my default font size (ie, the font size that I find most readable). Since some things are less important than others, I am willing to reduce their font size in order to fit more text into the same physical space. While web designers may come to their senses, this problem is likely to be eternal.
There are two additional reasons that come up periodically. Sometimes I am vaguely watching information that is generally unimportant; because it's unimportant, I want to have it in a relatively small font size not just so that it takes up as little room as possible but also to de-emphasize it so that it stops drawing as much of my attention. Next, sometimes I am working with something that has a fixed size in characters, such as an 80x24 interface (because I am talking to something over a serial line and it knows that I have an 80x24 VT100). In this situation I often might as well crank up the text size for maximum readability and emphasis (generally the larger a window is, the more attention I pay to it).
Looking back at my first entry, most of these reasons apply primarily to browsers (broadly construed) and to terminal windows. Ignoring HTML and rich text content, editors and mail readers and the like are much more a case where I will set a text size once and then stick to it, although this is partly because my normal editors and mail readers don't give me any convenient way of detaching a content window and then shrinking it down because I'm just going to use it as reference for something else I'm doing.
(My broadly construed version of browsers is 'any program that lets the content designer monkey around with the text size and design'. If the content designer can shrink the font size, sooner or later they will and sooner or later I will disagree violently with that shrinking.)