While I wrote up the general state of affairs with ZFS and 'advanced format' disks I've never described how this affects us in specific. The short version is that we are not in as much trouble as we might otherwise be because we're running ancient and somewhat under-functional software. You are in the maximal amount of trouble if your version of ZFS will refuse to add 4K sector disks to old pools and you have no way to lie to ZFS (or the kernel in general) about what the physical sector size of your disks is. Our situation is mostly the reverse of this.

First, our version of Solaris (Solaris 10 update 8 plus some patches) turns out to be so old that it doesn't even know about physical sector size as distinct from logical sector size. This is good in that it will not even notice that it's mixing dissimilar disks but bad in that we now have no way of creating new pools or vdevs with ashift=12. Second, our ISCSI target software doesn't export information about the physical sector size of disks that it's making visible so even if our version of Solaris was aware of 4K disks, it wouldn't see any. The upshot of this is that we can freely add 4K disks to our existing pools. The performance impact of this is not currently clear to me, partly because our environment is somewhat peculiar in ways that make me think we'll experience less impact than normal people in this situation. The bad news is that my initial testing on streaming IO shows a visible difference in write performance, although not a huge one (I need to put together a good random write test before I can have opinions on that).

In the short term, we'll survive if we have to replace 512b disks with 4K disks; some things may run slower but so far it doesn't look like they will be catastrophically slow. In the long term we need to replace the entire fileserver infrastructure and migrate all of the data to new pools created with ashift=12. We'd like to do it before we have to buy too many 4K disks as replacement disks for existing pools.

(We always knew we had to replace the existing hardware and update the software someday, but it used to be less urgent and we expected that we could keep the pools intact and thus do the upgrade with minimal user impact. Our current vague timeline is now to do this sometime in 2014, depending on when we can get money for hardware and so on.)

PS: ZFS continues to be our best option for a replacement fileserver infrastructure, although we don't know what OS it'll be running on. Linux btrfs is the only other possible competitor and it's nowhere near ready yet. Our budget is unlikely to allow us to purchase any canned appliance-like solution.