Wandering Thoughts archives

Practical RAID-1 read balancing

If you're doing performance analysis of a RAID-1 setup, one of the interesting questions is 'which drive gets read from when?'

(Measuring how balanced the current IO load is is useful, but it doesn't tell you how a changed load will affect the balance.)

Since seeks are the expensive thing on modern drives, you want your RAID-1 system to deal in requests, not in individual blocks, and to send a sequence of sequential reads off to the same disk. If your IO isn't strictly sequential, ideally the system keeps track of the last known head position for each disk (influenced by writes as well as by reads), and issues to the disk that is some combination of positioned closest and least loaded.

But all of this is theory. What you really need to do is measure, because you can never be sure just what a RAID-1 system is doing. Sometimes it can really surprise you.

For example, I once dealt with a reasonably fancy hardware RAID-10 controller where which disk a read would go to was statically determined, not based on load or anything. The controller divided the RAID-10 array up into slices (64K at the time); reads from odd slices went to the first disk in a mirror pair and reads from even slices went to the second. In extreme cases, half the array could be completely idle while the other half was melting down. (Presumably that didn't happen too often.)

We only found this out by running a test program against an otherwise idle test array and watching the front panel disk lights, as part of trying to sort out how the array distributed IO over the drives in general. Much to our surprise, in one test half the disks went active and the other half didn't. All I can say is thank goodness for front panel lights.

Sidebar: RAID-10 versus RAID-01

Because I always get them confused: RAID-10 is striped mirrors, RAID-0 on top of RAID-1. RAID-01 is the reverse, mirrored stripes, and is much less resilient against failures of two or more disks; RAID-01 dies unless all failed drives are in the same stripe, whereas RAID-10 will survive unless both sides of a single mirror die.

A surprising effect of RAID-1 resynchronization

Today I got to run into an interesting performance impact of having a RAID-1 mirror resync running on a big partition of a live system.

An important system was having performance problems today, so we were poking around it. When we watched the disk statistics, we noticed that only the first disk was seeing read traffic; the second disk was loafing along with just occasional bursts of writes. Looking more closely we noticed that a RAID-1 resync of a big partition was in progress; because the system was loaded, the resync's IO bandwidth had been choked and it hadn't gotten very far, only 5% or so in a 100G partition.

Then the light dawned. Normally, reads are distributed over both sides of a RAID-1 mirror. However, at the moment only 5% of the second disk was valid; a read for something in the remaining 95% could only be be done by the first disk. No wonder the first disk was running hot and the second disk was seeing virtually no reads.

Like everybody, I already knew about the direct IO impact of a RAID-1 resync. But the choking effect of not being able to read from both disks for most of the filesystem hadn't previously occurred to me.

Sidebar: what's a RAID-1 resync?

A RAID-1 resync is what happens when the two disks in a RAID-1 mirror cease to be identical copies of each other, usually due to some calamity (power loss, system crash, disk failure). When this happens, one of the mirrors is identified as the most up to date and its data gets dumped to the other disk to bring them back into sync.

The obvious effect of a RAID-1 resync is that it adds extra IO to the system: reads on the first disk, writes on the second disk. However, any decent RAID system has various things to limit this IO so that it happens more or less when the disks are idle and doesn't steal IO bandwidth from real work.

Wanted: RSS feeds for vendor software updates

Vendors, I have a simple request for you: give me an RSS feed for your security fixes and system updates. No, please don't suggest that I subscribe to your email announcement list, because these days email is just too much of a hassle. Accept it; RSS is the new lightweight notification format, and notification of new updates is just what I want.

And don't just give us one global RSS feed of everything. Be modern; let people get narrow feeds as well as broad ones. Easier yet, tag all of your updates and let people subscribe to tags and combinations of tags.

Existing feeds

Debian Security has a feed for Debian security updates. The version that includes the first paragraph, so you can see some details about each issue, is here [RSS feed].

Fedora Core doesn't have an explicit feed, but we can fake it by getting an RSS feed of the Fedora announcements list from gmane.org. The 'full text of all messages' version is here [RSS feed]. The Fedora announcements list isn't just package updates, but it's close enough for my purposes.

Red Hat Enterprise Linux has an announcements mailing list too, and thus a similar feed here [RSS feed].

Note that you can use gmane.org this way for any project that has a reasonably non-noisy announcements mailing list (if gmane.org doesn't already carry it, you can ask them to subscribe). Go wild.

(gmane.org is quite cool and worth a look in general; I wrote about it before in GmanePointer.)

Automation promotes action

I've written before about the advantages of scripts, but my own recent experiences have reminded me of another one, which I can snappily phrase as automation promotes action. Or in other words: when you've scripted something so that it's easy to do, you do it more often.

I build Firefox from CVS myself, for various reasons, but I run it on a different machine than I build on. Bringing up a new build up on the right machine involves about three or four steps; nothing particularly intricate, but just long enough to be vaguely annoying. (Among other things, I save the old build in case I want or need to revert to it; since I'm on the bleeding edge, this happens every so often.)

For a long time I didn't script this, because it was just small enough to seem silly and just dangerous enough to make me nervous. Recently I bit the bullet and spent the time to write a newfox script that I was happy with. Predictably, I now feel much happier about doing build updates and they happen somewhat more often; the difference between typing newfox and typing N somewhat long commands turns out to actually matter after all.

Call it a lesson learned. Apparently every so often I need to experience these things, not just know them intellectually.

An advantage of using a non-standard shell

One of the surprising advantages of using a completely non-standard shell (in my case, Byron Rakitzis' Unix version of rc) is all of the things that are not automatically set up for you by modern systems. As someone I read found out recently, these days this includes colourized ls listings, colourized super-intelligent vi, and so on.

(I might not mind colourization so much if it paid any attention to what the baseline terminal colours were, but I haven't seen that happen yet. And it tends to come out as fruit salad even on the best of days.)

The other thing I tend to get to skip is all sorts of $LANG settings for internationalization. These are occasionally OK, but a lot of the time they annoy me by changing, for example, sort's output or the order ls puts files in. I'm a creature of sufficient Unix habit that I get perturbed if these shuffle. (And my scripts can get perturbed too. Yes, I should work out the magic to use the old fashioned collation and sorting order without drop-kicking the rest of the internationalization stuff. Someday. When I have to.)

Red Hat has a well developed (and heavily used) system for sticking standard shells with things, driven out of /etc/profile.d. Perusing the files in that directory can be interesting, and sometimes alarming. (Which leads to the discovery that the easy way to spay ls is to create an empty $HOME/.dircolors file.)

Fortunately Red Hat has stopped making /usr/bin/vi be vim; that caused me to have a vi symlink in $HOME/bin that pointed to the 'real' (non-fancy) /bin/vi (I put $HOME/bin before system directories precisely so I can fix this sort of thing). These days, the full vim experience has to be invoked as vim, and if you ask for vi (without aliases that redirect it to vim) you get the plain thing. (Unfortunately if you ask for ex you're out of luck; that they still override.)

The sysadmin's life

A one-line addition to httpd.conf: 30 seconds.

Restarting Apache: 30 seconds.

Making sure that the addition was the right thing to do to an old version of Apache and wouldn't kill the server: much longer.

(And of course the much longer doesn't look too much like work, and doesn't entirely feel like work.)

And more of the sysadmin's life:

Accidentally using kill -USR2 instead of kill -USR1 to gracefully restart the Apache server: priceless. Whoops.

Trying to find exactly where Apache gets started on boot, to restart it with the right arguments, and being unable to do so: nerve-wracking.

Discovering that the machine is actually running Solaris 8 instead of Solaris 9: twitch-inducing. (So much for installing the Solaris 9 recommended patch set during this unexpected downtime.)

Discovering that the web server does not in fact get automatically restarted on reboot, nor do a number of other things: bad.

Discovering that the hostname changed on reboot, to 'test': AUGH. (It turns out that in Solaris 8, /etc/nodename is what sets the host's name. /etc/hostname.<interface> is not.)

I think I have well and truly stubbed my toes today. On the flipside, I know somewhat more about magic Apache things now, including useful bits like 'apachectl startssl' as the magic way of saying 'start Apache with SSL'.

(The management apologizes for this entry being somewhat less coherent than usual. Chris had a little bit of a shock today.)