Wandering Thoughts archives

2008-01-16

Why sysadmins should keep a lab notebook

Yesterday, a coworker and I were working on a performance issue we're having with our new SAN RAID controller. We had a hypothesis about what might provoke the problem, so we sat down, fired up some tests, and watched our logs; nothing showed up. Later on in the day, we saw some odd indications in other logs and wanted to see if they correlated with the tests we'd done, but you can already guess the punchline: we hadn't recorded when we started and stopped the tests.

This wasn't because we were stupid idiots (although you may disagree); it happened because we were focused on what we were looking for at the time of the experiment, which was going to give us a yes or no answer right away.

The important thing about a lab notebook is not so much the physical object; it is the discipline of writing everything down, even if you don't think you need it at the time. Keeping a record, even a simple one, means that you do not have to rely on fallible memory and guesswork when you later want to look back to summarize what experiments you've done (especially the unsuccessful ones, especially the fine details), or what exactly you did in the process of fixing the mysterious problem, and so on.

(This is especially important for problem fixes, because humans have a great habit of assuming that the last thing we did has to be what worked and then brushing everything else out of the way. And we can do this without even consciously realizing what we're doing.)

As my experience handily demonstrates, keeping a lab notebook is especially important during problems and crises, when you have no real idea what is going on, what to do next, and how to fix things. When we know the least is exactly the time when we need to record the most, because we just don't know what's going to turn out to be important in the end (and we are prone to overconfident, hopeful guessing).

(As a side benefit, scrawling grumpy remarks in your lab notebook can be a good stress relief that does not involve ranting at your coworkers.)

sysadmin/SysadminLabNotebook written at 23:51:41; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.