ZFS's issues with long term storage management
Unfortunately, one of ZFS's current issues is that it does not have any direct support for the sort of data migration you need to do for long term storage management. There are three possible approaches to this:
- direct migration support: not present.
- growing a pool with the new storage then shrinking it by removing the old storage: while ZFS can grow pools it can't shrink them.
- add the new storage as a mirror and then detach the old storage: you can't mirror raidz or raidz2 pools.
The way you are supposed to use ZFS is to feed it raw disks and make raidz or raidz2 pools with them, but this leaves you without good migration options. Replacing individual disks as if they had failed and been replaced is not good enough; it takes out your redundancy for the duration of the rebuild, plus you can never change the number of disks that the storage pool is using.
If you let your backend handle the RAID bits and feed ZFS logical disks instead of physical ones, you can fake migration through mirroring (since you can mirror and then unmirror disks that are not part of a raidz or raidz2 pool). But doing this gives up some of ZFS's performance and reliability advantages and you don't get RAID 6 unless your backend supports it, which not all do.
(Shrinking ZFS pools is theoretically a planned future feature, but I can't make plans that count on it until it actually gets implemented and appears in Solaris.)
The advantage of a SAN
It's simple: the advantage of a SAN is that it decouples your storage space from your servers.
This means that you can deal with each side independently; you can add storage or move storage without having to change the server hardware, and you can replace (and add) servers without having to do anything to your disks. In turn this means that you can use common garden variety servers instead of big monster servers that are designed to hold loads of disks, which means it is much easier to maintain and replace them.
(You may also be able to use relatively commodity disk enclosures, instead of monster huge specialized ones, although this depends on your SAN.)
This decoupling also simplifies failover and recovering from hardware problems in general, because it means that a server failing doesn't automatically take down some disk space and vice versa. (Merely using a SAN does not necessarily make your environment more reliable, though.)
Painless long term storage management without disturbing users
One of the things I have become conscious of recently is that managing your storage over the long term without lots of pain and without disturbing your users requires an additional set of features that you may not realize you need. (Partly I came to this realization by blithely proposing a design for our next generation of storage without really thinking about the issues, and then having people gently point out the problems I had missed.)
(By 'painless' I mean that it does not take a lot of sysadmin effort to deal with, so you do not have people babysitting dump and restore runs all weekend. To avoid disturbing users you can't change around your filesystem layout or have to unmount and remount NFS filesystems or CIFS shares or whatever; ideally they shouldn't notice that anything has happened.)
There are two major sorts of changes that happen over the long term: you replace old hardware, especially old disks, with newer and bigger stuff, and you add more storage, often in the process of replacing hardware. (Over the long term you are going to replace everything, including the physical servers themselves.)
So you need migration and growth: you need to be able to migrate data from old storage to new storage, and once this is done you need to be able to grow things to use the extra space on the new storage. It is reassuring to also be able to mirror data between your old storage and your new storage and run them both at the same time; this gives you a live test of the new storage before you are completely committed to it.
(In some sorts of migration systems you get mirroring for free, because you migrate data by creating a mirror and then discarding the old side of it.)
These features have to be provided by your storage system, because you cannot make them painless and transparent without the storage system's cooperation. As a consequence of this, you need to be able to make both the new and the old storage visible to the storage system on a single machine at the same time.
And of course all of this has to be transparent, which means that you have to be able to do it while your systems are live, with users banging on the filesystems and so on.