Wandering Thoughts archives

What we keep track of for ZFS pools

One of the attractions of ZFS pools is that they are pretty self-documenting. A ZFS pool automatically captures a great deal of the basic information that you need to deal with it, such as the filesystems it contains and NFS export settings. But this isn't quite all of the information that you need for long term management and for disaster recovery, and in our new fileserver environment we've opted to duplicate some of the information outside of ZFS as well.

(Keeping track of filesystems, export permissions, and so on is important in a SAN environment where you can expect to move filesystems between physical servers, for example for disaster recovery. Having a SAN that lets other machines get at the actual data doesn't help if the information needed to manage the data is locked up on a single machine.)

We have two primary additional sources of information on our ZFS pools. First, we have a master configuration file for all of our filesystems that lists their name, the pool they are part of, and any non-default options on them; for us this is quotas, reservations, and any special NFS export permissions. (We have used a syntax that lets us easily say 'add this netgroup to the exports', because this is our common case; writing 'rw+=cluster2' is much better than repeating the entire long share options with a minor change, even if it requires custom software.)

Second, we periodically harvest information about all pools on each fileserver and save it to a central location. The information includes:

• pool size and pool quota, if any
• the LUNs that the pool is using
• the filesystems that it's currently configured with, and their actual quota and reservation settings.

(We also create a reverse index from LUNs to what they are being used for; among other uses, this is good for picking out unused LUNs to use when adding new pools or expanding existing ones.)

This pool information is partly for disaster recovery and partly to have all of the relatively current data about all pools and LUNs in one spot, instead of having to log in to a fileserver and poke around with sometimes obscure ZFS commands.

We don't directly save a mapping between ZFS pools and what virtual fileserver they belong to, because we have adopted a pool naming convention that puts the virtual fileserver in the pool name itself. Thus, in a disaster recovery situation all we have to do is scan the 'zpool import' list for pools with a particular prefix to know what pools belong where.

(The fileserver information is indirectly contained in the pool information dump in normal circumstances, as each machine's data winds up in a separate file, and the filesystem configuration file does say what fileserver each filesystem is supposed to be on, so you can sort of reverse engineer it from there.)