Our situation with ZFS and 4 Kb physical sector disks
While I wrote up the general state of affairs with ZFS and 'advanced format' disks I've never described how this affects us in specific. The short version is that we are not in as much trouble as we might otherwise be because we're running ancient and somewhat under-functional software. You are in the maximal amount of trouble if your version of ZFS will refuse to add 4K sector disks to old pools and you have no way to lie to ZFS (or the kernel in general) about what the physical sector size of your disks is. Our situation is mostly the reverse of this.
First, our version of Solaris (Solaris 10 update 8 plus some patches)
turns out to be so old that it doesn't even know about physical sector
size as distinct from logical sector size. This is good in that it will
not even notice that it's mixing dissimilar disks but bad in that we now
have no way of creating new pools or vdevs with
our ISCSI target software
doesn't export information about the physical sector size of disks that
it's making visible so even if our version of Solaris was aware of 4K
disks, it wouldn't see any.
The upshot of this is that we can freely add 4K disks to our existing
pools. The performance impact of this is not currently clear to me,
our environment is somewhat peculiar in ways that
make me think we'll experience less impact than normal people in this
situation. The bad news is that my initial testing on streaming IO shows
a visible difference in write performance, although not a huge one (I
need to put together a good random write test before I can have opinions
In the short term, we'll survive if we have to replace 512b disks with
4K disks; some things may run slower but so far it doesn't look like
they will be catastrophically slow. In the long term we need to replace
the entire fileserver infrastructure and migrate all of the data to new
pools created with
ashift=12. We'd like to do it before we have to buy
too many 4K disks as replacement disks for existing pools.
(We always knew we had to replace the existing hardware and update the software someday, but it used to be less urgent and we expected that we could keep the pools intact and thus do the upgrade with minimal user impact. Our current vague timeline is now to do this sometime in 2014, depending on when we can get money for hardware and so on.)
PS: ZFS continues to be our best option for a replacement fileserver infrastructure, although we don't know what OS it'll be running on. Linux btrfs is the only other possible competitor and it's nowhere near ready yet. Our budget is unlikely to allow us to purchase any canned appliance-like solution.