2012-09-14
What determines how much work a ZFS resilver has to do
We lost another fileserver iSCSI backend the other day for the first time in a long while, and this has gotten me thinking about how expensive a ZFS resilver is. What I really mean by that is some way to start guessing at the amount of time that a resilver will take (or at least the relative amount of time compared to another potential resilver).
In a simple world, the time to resilver would be proportional to the amount of data on the vdev being resilvered. ZFS doesn't work this way. Because ZFS scrubs and resilvers are nonlinear, a ZFS resilver has to look at all of the pool's metadata in order to find all of the data it needs to copy (for mirroring) or recreate (for raidzN). So I think the answer is that a resilver takes time proportional to the size of the pool's metadata (including directories) plus the amount of data on the vdev or vdevs that are being resilvered. Scanning pool metadata is probably going to be largely seek bound; scanning and copying the data itself will hopefully be more linear.
Unfortunately I don't believe that our version of Solaris keeps track of the metadata space usage for ZFS pools (and I don't think you can deduce a good number for it from other information). You can get overall space usage but not information on how much of that space is directories, inodes, and so on versus actual file contents. However if you just want a relative comparison you can assume that two pools have the same relative metadata/data ratio and then directly compare pool sizes.
Where this approach hits the rocks for me is how to scale the relative contributions of the size of the vdevs being resilvered and the size of the theoretical pool metadata (or alternately how to factor in the size of the resilvering vdevs). Without doing much analysis I think that you want to take the total pool size, subtract the size of the vdev(s) being resilvered, scale down by some factor, and then add the size of the resilvering vdevs back in again.
(I think you need to specifically consider the size of the resilvering vdev(s). If pool A and pool B are both 500 GB but the vdev to be resilvered is 20 GB in pool A and 200 GB in pool B, pool A seems likely to finish resilvering first.)
2012-09-02
Solaris 11 is still closed source (and Oracle is careless with words)
In a comment on my original entry on Solaris 11 being closed source, a commentator pointed out what looked like source code for Solaris 11 on Oracle's site, here. Flush with optimism, I eagerly downloaded the two large zip files, unpacked them, and dove in to take a look at things like the Solaris 11 version of ZFS.
I will spare you the bother of doing this and skip to the punchline. The Oracle web page carefully says that it is for 'source code for open source software components'. What they really mean, at least for Solaris, is third party open source software. The zipfiles contain nothing more than the source code for things like Perl, Python, all of the Gnome desktop code, and so on. There is no Solaris kernel source, no DTrace, no anything that belongs to Oracle itself.
If you want to, you can argue that Oracle's omission of the crucial words 'third party' from the description of stuff on this web page is technically not misleading. After all, Solaris was never open source; what was open source was OpenSolaris, which was close to but not identical to Solaris itself, and there is no mention of OpenSolaris on that web page. It is mere optimism for people to see 'open source software components' and 'Solaris' on the same page and jump to the unwarranted conclusion that Oracle is including updated OpenSolaris code that corresponds to Solaris 11.
In practice, this page is quite misleading. I think that most people seeing it are likely to make the same assumption that my commentator made, that the 'Solaris 11' stuff actually includes real Solaris 11 or OpenSolaris source code. I have no idea if Oracle was deliberately misleading when they wrote the text for this web page, or if it just happened by accident and because the people doing it didn't care enough to think about how outside people would read the text. Either case would be perfectly typical of Oracle and their attitude towards the open source world.
Really, it doesn't matter which is true. We still don't have any OpenSolaris or Solaris code update; it is still closed source.
(If you want to slice things finely you can argue that this page's omission of 'third party' in its description plus the omission of OpenSolaris source code is a clear message that Oracle no longer considers Solaris to be open source. While I expect that the conclusion is right, I don't think this page's text had that much thought and care put into it. I also suspect that Oracle is avoiding saying anything definite about Solaris' closed source status; to put it one way, there isn't much upside to Oracle making a definite statement about it.)