2007-06-04
RAID-5 versus RAID-6
For those who haven't encountered it yet, RAID-6 is RAID-5 with two parity blocks per stripe instead of one; it thus has a two disk overhead for parity, compared to RAID-5's one disk.
We're interested in RAID-6 because it is more or less a better version of RAID-5 plus a hot spare. With RAID-5, if you lose a drive the disk array is non-redundant for the time it takes to rebuild onto your hot spare; if you lose a second disk during this time, the entire array is lost. With large modern disk drives of the sort we use, a rebuild can take many hours, especially if your RAID array is active with real work at the time.
(RAID-6 does pay a penalty on writes, which we think is not too important in our environment. This may be counterbalanced by the fact that it keeps your 'hot spare' drive active, so it can't quietly die on you.)
RAID-6 makes the most sense if you are using all of the drives in a controller in a single large disk array, which is what we generally do. If you're making multiple arrays, RAID-5 plus a single global hot spare may have significantly less overhead (depending on how many arrays you're making).
Unfortunately, few vendors of standalone SAN RAID controllers seem to include RAID-6 (at least in their baseline products; it may be available in high-end gear or as an extra-cost option). Software implementations are more common, with Solaris ZFS's 'raidz2' and Linux's software RAID being two of them.
Sidebar: RAID-5 efficiency figures
For my own interest:
| disks | RAID-5 overhead | RAID-6 overhead |
| 2 | 50% | |
| 3 | 33% | 67% |
| 4 | 25% | 50% |
| 5 | 20% | 40% |
| 6 | 17% | 33% |
| 7 | 14% | 29% |
| 8 | 12.5% | 25% |
| 9 | 11% | 22% |
| 10 | 10% | 20% |
| 11 | 9% | 18% |
| 12 | 8% | 16% |
| 13 | 7.7% | 15% |
| 14 | 7% | 14% |
| 15 | 6.7% | 13% |
Common sizes for standalone SATA RAID controllers are 12 disks (2U) or 15 disks (3U).
Why we need our SAN RAID controllers to support logical drives
Not all SAN RAID controllers support the idea of logical disks, where you can make a bunch of drives into a storage pool and then carve the storage pool up into software-managed bits; some prefer the simpler and more straightforward approach of just exporting the pool itself and staying out of the space management business.
Unfortunately, we need logical drives. The problem (and the reason) is that logical drives are the only way to sensibly dynamically split the space on a RAID controller between multiple fileservers. This is because logical drives are the only way to split out relatively small amounts of the controller's disk space with any efficiency and ease.
If you can only split whole disks out to fileservers and you want protection against single drive failures, the minimum allocation to a fileserver is two disks (RAID-1, a 50% overhead). If you want less wasted space, you get to make significantly larger allocations. In addition, adding or removing disks from a RAID set is an expensive operation that requires shuffling a lot of data around. When it can be done at all (which is not guaranteed, especially for shrinking a RAID set), it's not going to be fast.
By contrast, logical drives can split off relatively small amounts of disk space, and they do it easily. Because they are carving free space out of a pool of it, no data has to be shuffled around; because they are using the pool, they are as efficient as the pool itself.
As a result, if you want to be able to shuffle unused disk space in your pool of space on the SAN between fileservers without having to plan it well in advance, you need the SAN RAID controllers to support logical drives. RAID controllers without logical drives are only really good for situations where you can statically allocate things at setup time, especially when you are not splitting up the space very much (otherwise the overhead eats you alive).