Chris's Wiki :: blog/sysadmin/SysadminLabNotebook

Why sysadmins should keep a lab notebook

January 16, 2008

Yesterday, a coworker and I were working on a performance issue we're having with our new SAN RAID controller. We had a hypothesis about what might provoke the problem, so we sat down, fired up some tests, and watched our logs; nothing showed up. Later on in the day, we saw some odd indications in other logs and wanted to see if they correlated with the tests we'd done, but you can already guess the punchline: we hadn't recorded when we started and stopped the tests.

This wasn't because we were stupid idiots (although you may disagree); it happened because we were focused on what we were looking for at the time of the experiment, which was going to give us a yes or no answer right away.

The important thing about a lab notebook is not so much the physical object; it is the discipline of writing everything down, even if you don't think you need it at the time. Keeping a record, even a simple one, means that you do not have to rely on fallible memory and guesswork when you later want to look back to summarize what experiments you've done (especially the unsuccessful ones, especially the fine details), or what exactly you did in the process of fixing the mysterious problem, and so on.

(This is especially important for problem fixes, because humans have a great habit of assuming that the last thing we did has to be what worked and then brushing everything else out of the way. And we can do this without even consciously realizing what we're doing.)

As my experience handily demonstrates, keeping a lab notebook is especially important during problems and crises, when you have no real idea what is going on, what to do next, and how to fix things. When we know the least is exactly the time when we need to record the most, because we just don't know what's going to turn out to be important in the end (and we are prone to overconfident, hopeful guessing).

(As a side benefit, scrawling grumpy remarks in your lab notebook can be a good stress relief that does not involve ranting at your coworkers.)

Comments on this page:

From 70.79.141.48 at 2008-01-17 09:25:02:

I follow Thomas' cycle system from Time Management for System Administrators for time management. I don't quite follow his system. I use an PAA (personal analog assistant) -- it's a composition book. When you open the book, the right page is my planner, the left is for any notes I need to take -- phone numbers, serial numbers, directions or anything else I may need away from a computer.

The point is that I always have this with me. If I'm in the lan room and there's an error message on a crashed machine, I'll write it down. If I need to record times of events, I'll also jot it down.

- Gary

From 83.145.204.27 at 2008-01-17 15:29:11:

Amen.

As long as I remember, I have used a simple diary. On every machine I admin this text file is located in the given admin-user's home directory in capital letters. In lack of a full-blown version control system, it even worked well for one machine that had an co-admin.

17.1.2008

Enabled ACPI PnP features. Booted fine, but disable if problems occur.

options PCI_INTR_FIXUP # PCI interrupt routing via ACPI options ACPI_ACTIVATE_DEV # If set, activate inactive devices options ACPICA_PEDANTIC # force strict conformance to the Spec.

drear.

By cks at 2008-01-18 00:00:49:

I think that this starts to blur lab notebooks together with what I'll call changelogs, and my views on this got long enough that I put them in a new entry, LabbooksVsChangelogs.

From 71.116.163.210 at 2011-05-19 03:02:17:

Do you just use an ordinary notebook? Or separate pieces of paper (maybe in a looseleaf notebook)? (I was going to ask if you used paper or a wiki or notes on tickets or something else, but you answered that in the lab notebook vs. changelog entry.)

I often jot down notes while working on a problem, and have been thinking about trying to be more systematic about it. But I often find myself working on several things at the same time, and with a bound notebook, I'd either have to allocate some number of pages (and possibly end up with extra pages or not enough) or mix things together. Thus far, I've managed to move from using small pieces of paper to letter-size pages, and keeping one topic per sheet. Where it seems worthwhile, I'll move information from the notes into my documentation wiki after things are settled.

I'm also somewhat tempted by using something like org-mode to keep notes. One of the big advantages of an electronic format, it seems to me, is that you can copy and paste command lines directly into your notes. Writing out long Unix pipelines gets old pretty quickly.

-- Claire

By cks at 2011-05-19 10:37:38:

What has wound up happening is that I take notes electronically whenever I can (I just dump them into plain ASCII files), and have a basic notebook for when I can't make electronic notes. My handwriting is slow enough and bad enough that I would not enjoy the experience of trying to write a lot on paper.

(And I think that for things like command lines where you want to be absolutely sure that they are just what you actually ran, you should cut & paste instead of rewriting/retyping.)

In the paper notebook I just date each page and add some note about what it's about. This is okay because it's not used very much. The electronic files tend to be called something like '2011-05-19-explosion-issue', with both the date that things started on and some hint of what the issue is.

(Once I have a file for an issue it gets all subsequent stuff for that issue, even stuff from another day. I generally annotate my notes when the day changes.)

Written on 16 January 2008.

«	What applications are actually crucial at a university

Lab notebooks are not changelogs