Our situation with ZFS and 4 Kb physical sector disks

May 27, 2013

While I wrote up the general state of affairs with ZFS and 'advanced format' disks I've never described how this affects us in specific. The short version is that we are not in as much trouble as we might otherwise be because we're running ancient and somewhat under-functional software. You are in the maximal amount of trouble if your version of ZFS will refuse to add 4K sector disks to old pools and you have no way to lie to ZFS (or the kernel in general) about what the physical sector size of your disks is. Our situation is mostly the reverse of this.

First, our version of Solaris (Solaris 10 update 8 plus some patches) turns out to be so old that it doesn't even know about physical sector size as distinct from logical sector size. This is good in that it will not even notice that it's mixing dissimilar disks but bad in that we now have no way of creating new pools or vdevs with ashift=12. Second, our ISCSI target software doesn't export information about the physical sector size of disks that it's making visible so even if our version of Solaris was aware of 4K disks, it wouldn't see any. The upshot of this is that we can freely add 4K disks to our existing pools. The performance impact of this is not currently clear to me, partly because our environment is somewhat peculiar in ways that make me think we'll experience less impact than normal people in this situation. The bad news is that my initial testing on streaming IO shows a visible difference in write performance, although not a huge one (I need to put together a good random write test before I can have opinions on that).

In the short term, we'll survive if we have to replace 512b disks with 4K disks; some things may run slower but so far it doesn't look like they will be catastrophically slow. In the long term we need to replace the entire fileserver infrastructure and migrate all of the data to new pools created with ashift=12. We'd like to do it before we have to buy too many 4K disks as replacement disks for existing pools.

(We always knew we had to replace the existing hardware and update the software someday, but it used to be less urgent and we expected that we could keep the pools intact and thus do the upgrade with minimal user impact. Our current vague timeline is now to do this sometime in 2014, depending on when we can get money for hardware and so on.)

PS: ZFS continues to be our best option for a replacement fileserver infrastructure, although we don't know what OS it'll be running on. Linux btrfs is the only other possible competitor and it's nowhere near ready yet. Our budget is unlikely to allow us to purchase any canned appliance-like solution.


Comments on this page:

From 198.102.62.250 at 2013-05-27 11:39:07:

ZFS on Linux seems to be maturing quite nicely -- though I haven't tried it in some time.

What about OpenIndiana/Illumos or moving to Solaris 11?

By cks at 2013-05-27 15:24:07:

Our current leading candidates are some Illumos distribution and ZFS on Linux. ZFS on FreeBSD is currently not viable because of a bad iSCSI initiator implementation; Solaris 11 is more or less ruled out because of cost and lack of source code access. But all of this is pretty preliminary because everything is evolving fairly rapidly and we're probably not making a decision until early next year. By that time things may be very different.

(There's some work being done on the FreeBSD iSCSI initiator so I'm hopeful that it will be a viable option by the time we have to make a real decision.)

Written on 27 May 2013.
« Empirically, modern mailing list services are spam senders
Understanding SQL placeholders »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon May 27 00:43:15 2013
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.