I mentioned recently that the one remaining filesystem we need to migrate from our remaining OmniOS fileserver to our Linux fileservers is our central administrative filesystem. This filesystem is a challenge to move because everything depends on it and a number of systems are tangled into it (for example, our fileservers all rsync a partial copy from its master fileserver, which means that they need to know and agree which is the master fileserver and the master fileserver had better not try to replicate a copy from itself). With so many things depending on this filesystem and it so tangled into our systems, we clearly needed to carefully consider and plan the migration.

Of course, this is not the first time we've moved this filesystem between ZFS fileservers; we did it before when we moved from our first generation Solaris fileservers to our our OmniOS fileservers. Since I'm a big fan of checklists and I have theoretically learned a lesson about explicitly saving a copy after things are over, I was pretty sure that I had written up a checklist for the first migration and then kept it. Somewhere. In my large, assorted collection of files, in several directories, under some name.

As you might expect from how I wrote that, my initial attempts to find a copy of our 2015-ish checklist for this migration did not go well. I looked in our worklog system, where ideally it should have been mailed in after the migration was done, and I hunted around in several areas where I would have expected to keep a personal copy. Nothing came up in the searches that I attempted, and I found myself wondering if we had even done a checklist (as crazy as that seemed). If we had done one, it seemed we had lost it since then.

Today, freshly back from vacation, I resorted to a last ditch attempt at brute force. I used 'zpool history' to determine that we had probably migrated the filesystem in mid-February of 2015, and then I went back to the archives of both our worklog system and our sysadmin mailing list (where we coordinate and communicate with each other), and at least scanned through everything from the start of that February. This finally turned up an email with the checklist (in our sysadmin mailing list archives, which fortunately we keep), and once I had that, I could re-search all of my files for a tell-tale phrase from it. And there the checklist was, in a file called 'work/Old/cssite-mail-migration'.

(It had that name because back in 2015 we migrated a bunch of administrative filesystems all at once at the end of things, including our central /var/mail. This time around, we migrated /var/mail very early because we knew 10G instead of 1G would make a real difference for it.)

Sadly, I'm not sure right now how I could have done much better than this round-about way. Explicitly sending the checklist to our worklog system would have helped a bit, but even then I would have had to stumble on the right search terms to find it. Both taxonomy and searching are hard (human) problems; with my searching, I was basically playing a guessing game about what specific terms, commands, or whatever would have been used in the checklist, and evidently I guessed wrong. One possible improvement might be to make a storage directory specifically for checklists, which would at least narrow my searching down (there are a lot of things in 'work/Old').

(Things like the name of the filesystem and 'migration' are not useful, because it shows up in every filesystem migration we do since it's where we put central data about NFS mounts and so on.)