Some thoughts on SAN long-term storage migration
In theory one of the advantages having of a SAN instead of simple disk servers is relatively painless storage migration over the long term, where given a suitable setup and suitable software you can more or less transparently migrate data from old storage backends to new ones without any particular user-visible downtime. In practice I've come around to the idea that we may never be able to do this in our fileserver environment and that in general it takes a particular sort of environment to make it work.
Put simply, the disks you get today are generally going to be significantly different than the disks of four or five years ago, in both capacity and perhaps performance (if you go to SSDs). These differences may well cause you to want to reshape how your storage is laid out to do things like consolidate to fewer spindles (or to spread out to even more to preserve IOPs while growing space). So in order to have transparent storage migration you need not just a SAN but frontend storage software that can do this sort of potentially drastic rearrangement of storage. Replacing individual disks with bigger disks is something that almost every storage system can do, but that's the simple case. It's less common to have support for transformations like moving from two smaller disks to one bigger disk or moving from a few big disks to more smaller disks (as might happen if you went from HDs to SSDs).
Put another way, you often want to design different storage setups today than four or five years ago even if you keep the same top level technology (eg ZFS). Given this, a transparent migration either requires some way to transmogrify the storage setups from five years ago into the storage setups of today or just living with a five year old setup (which is often less than desirable).
While our hand was forced by ZFS this time around, this is certainly one thing that probably would have biased us towards a non-transparent migration anyways. Moving from 750 GB and 1TB disks to 2TB disks caused us to roughly double our standard chunk size, but politics mean we can't give people double the space for free and anyways some people have space split across multiple ZFS pools (on different fileservers) that we'd like to consolidate into one. Doing the necessary reshaping transparently would take features that ZFS just doesn't have. I suspect that we'll run into similar issues in four or so years when we next migrate to another set of hardware and disks; the disks of four or five years from now are hopefully going to be significantly different from today.
Our migration from our previous generation DiskSuite based fileservers to our ZFS ones was obviously forced to be non-transparent because we were moving to ZFS, but in retrospect it also had the same issue; we moved from 35 GB 'chunks' to much larger ones and that would have again requires support for major storage reshaping even if we'd stuck with DiskSuite.
(And in general we've been lucky that ZFS has remained a viable option for us across this many years. If Sun had not released ZFS and Solaris as open source we'd probably be migrating away from ZFS now, just as we migrated away from DiskSuite last time. This ties into the arrogance of trying to plan for long term storage management in the face of all of the uncertainty in the technology world.)