Why you should let ZFS handle the RAID stuff

September 11, 2009

In a comment on my last entry, Matt Simmons asked why you'd let ZFS handle RAID level issues instead of just handling them in your storage backend (SAN or otherwise). Having ZFS do this is the recommended practice, for a number of reasons (some of which the ZFS FAQs will tell you about).

Here's the reasons for letting ZFS handle the RAID stuff that I know about (or at least can think of right now):

  • ZFS can reliably repair corrupted blocks in mirrors, because it can tell which copy of a block is bad and which is good.
  • ZFS should be able to recover better from subtle RAID-5/6 corruption. A backend RAID setup can fix an outright read error just as well as ZFS can, but I believe that ZFS can also fix things if a sector gets corrupted instead of simply reporting read errors.

  • ZFS reconstructs things faster and better after a disk failure than a SAN backend can.

  • ZFS will probably run faster if it's scheduling IO relatively close to the physical disks. This especially applies if the physical disks are actually shared between pools behind ZFS's back (as is typical if you define a big RAID-5 or RAID-6 array and then carve it up into LUNs, each of which you turn into a separate pool).

  • some write loads on RAID-5 or RAID-6 may run faster.

  • without its own redundancy (across different backends), ZFS will panic your system if one of your backends is temporarily unavailable for more than a relatively short amount of time.

Having ZFS handle RAID level issues has some drawbacks, though. They includes at least:

  • ZFS's versions of RAID-5 and RAID-6 have significantly lower random read rates.

  • You have to count on ZFS to get spares right.

  • various things hang if some of your disks go away in the wrong way, even though your pools are still working.

  • ZFS is much less prepared for long term storage management than serious (and seriously expensive) 'enterprise' SAN systems, many of which can do really quite sophisticated rearrangements of your SAN arrays.

I would hope that storage backends have better status monitoring and reporting tools than ZFS currently does (because ZFS's are fairly bad), but I am not at all sure that that is the case.

Comments on this page:

From at 2009-09-11 04:17:43:

Stop getting referenced everywhere Matt! :)

From at 2009-09-11 07:06:13:

Sorry, I just can't help it. I'm curious about things, and it gets me into trouble!

Seriously though, this is really interesting. Again, forgive my ignorance, but I don't work in a Sun shop, but does this mean that enterprise SUN people typically buy JBODs, or do they use servers with built-in storage more frequently, like this lucky gentleman?

I guess with my shop, the mindset is "buy enterprise storage so that the handling of the RAID and other particulars can be abstracted away".

And I appreciate your patience and responses to my questions. I know I ask a lot of questions. I'm just trying to wrap my head around it :-) Thanks!

Matt Simmons

By cks at 2009-09-11 12:51:07:

That's a good question, but unfortunately I don't have any answers; we're nowhere near being enterprise-level with our Sun stuff (or indeed anything), and I'm not in touch with that world.

I believe that I've read about a number of fairly large scale ZFS deployments, and they typically use JBODs or Thumper-level machines. I wouldn't be surprised if this was the common pattern for people who've decided to seriously go for ZFS.

From at 2009-09-11 13:08:30:

If you want Enterprise storage from Sun, using Sun technologies, get a Try & Buy for a J7000 series box. If you are unfamiliar with Analytics, you should really check out the VM simulator...


There are lots of other resources (videos, blogs by the Fishworks guys, etc) to look into it as well. Useful if only for drool-response and to see what's really possible when smart people wield DTrace.

From at 2009-09-20 02:55:47:

A good write-up. At least I felt like home with your arguments, having always been rather skeptical of RAID.

Maybe it is worth point out the recent article on LWN as well:


Written on 11 September 2009.
« What I know about how ZFS actually handles spare disks
My opinions on when you should let ZFS handle RAID stuff »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Sep 11 01:20:22 2009
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.