Wandering Thoughts archives

The advantage of a SAN

It's simple: the advantage of a SAN is that it decouples your storage space from your servers.

This means that you can deal with each side independently; you can add storage or move storage without having to change the server hardware, and you can replace (and add) servers without having to do anything to your disks. In turn this means that you can use common garden variety servers instead of big monster servers that are designed to hold loads of disks, which means it is much easier to maintain and replace them.

(You may also be able to use relatively commodity disk enclosures, instead of monster huge specialized ones, although this depends on your SAN.)

This decoupling also simplifies failover and recovering from hardware problems in general, because it means that a server failing doesn't automatically take down some disk space and vice versa. (Merely using a SAN does not necessarily make your environment more reliable, though.)

Painless long term storage management without disturbing users

One of the things I have become conscious of recently is that managing your storage over the long term without lots of pain and without disturbing your users requires an additional set of features that you may not realize you need. (Partly I came to this realization by blithely proposing a design for our next generation of storage without really thinking about the issues, and then having people gently point out the problems I had missed.)

(By 'painless' I mean that it does not take a lot of sysadmin effort to deal with, so you do not have people babysitting dump and restore runs all weekend. To avoid disturbing users you can't change around your filesystem layout or have to unmount and remount NFS filesystems or CIFS shares or whatever; ideally they shouldn't notice that anything has happened.)

There are two major sorts of changes that happen over the long term: you replace old hardware, especially old disks, with newer and bigger stuff, and you add more storage, often in the process of replacing hardware. (Over the long term you are going to replace everything, including the physical servers themselves.)

So you need migration and growth: you need to be able to migrate data from old storage to new storage, and once this is done you need to be able to grow things to use the extra space on the new storage. It is reassuring to also be able to mirror data between your old storage and your new storage and run them both at the same time; this gives you a live test of the new storage before you are completely committed to it.

(In some sorts of migration systems you get mirroring for free, because you migrate data by creating a mirror and then discarding the old side of it.)

These features have to be provided by your storage system, because you cannot make them painless and transparent without the storage system's cooperation. As a consequence of this, you need to be able to make both the new and the old storage visible to the storage system on a single machine at the same time.

And of course all of this has to be transparent, which means that you have to be able to do it while your systems are live, with users banging on the filesystems and so on.

Some thoughts on Slashdot's moderation system

Slashdot's comment moderation system is one of those wisdom of crowds things that are periodically held up as exemplars; community feedback, allowing the best to bubble to the top and so on. People who actually read Slashdot comments, even only showing high scoring ones, may be aware that it doesn't necessarily work this way.

One of the reasons for this is that Slashdot's '+1 Funny' moderation option tacitly encourages off-topic comments, because it encourages people to be funny even if there is nothing to their comment besides humour. As a result, an appreciable amount of the high-moderated comments on any random story are merely random humour.

The moral here is that people do what you reward them for, so be very careful what you reward them for. Especially in communities, where there is public feedback that amplifies those rewards.

As an occasional moderator, I also have to say that Slashdot's comment moderation interface is not great for its intended goal. The view of comments that I really want is 'most recent comments first, only for comments below a rating threshold', because this is what I want to pick out missing gems. (A comment that is already high-rated probably doesn't need me boosting it further.)

RAID-5 versus RAID-6

For those who haven't encountered it yet, RAID-6 is RAID-5 with two parity blocks per stripe instead of one; it thus has a two disk overhead for parity, compared to RAID-5's one disk.

We're interested in RAID-6 because it is more or less a better version of RAID-5 plus a hot spare. With RAID-5, if you lose a drive the disk array is non-redundant for the time it takes to rebuild onto your hot spare; if you lose a second disk during this time, the entire array is lost. With large modern disk drives of the sort we use, a rebuild can take many hours, especially if your RAID array is active with real work at the time.

(RAID-6 does pay a penalty on writes, which we think is not too important in our environment. This may be counterbalanced by the fact that it keeps your 'hot spare' drive active, so it can't quietly die on you.)

RAID-6 makes the most sense if you are using all of the drives in a controller in a single large disk array, which is what we generally do. If you're making multiple arrays, RAID-5 plus a single global hot spare may have significantly less overhead (depending on how many arrays you're making).

Unfortunately, few vendors of standalone SAN RAID controllers seem to include RAID-6 (at least in their baseline products; it may be available in high-end gear or as an extra-cost option). Software implementations are more common, with Solaris ZFS's 'raidz2' and Linux's software RAID being two of them.

Sidebar: RAID-5 efficiency figures

For my own interest:

Common sizes for standalone SATA RAID controllers are 12 disks (2U) or 15 disks (3U).