Documentation needs testing
August 19, 2006
One of the under-appreciated areas of writing systems documentation is testing it. And I don't mean proofreading it (although that's necessary too); I mean making sure that it is things like comprehensible, clear, correct, and complete.
Testing is vital, because documentation that is incomplete or incorrect is often worse than no documentation at all. Documentation creates confidence, which means that bad documentation lets people go wrong with confidence, turning a small mess into a much bigger one. ('But the instructions said to newfs /dev/sda1...')
Unfortunately, testing documentation is hard and time consuming. You don't really know if it's good until someone else follows it and things work, and this can be hard to arrange, especially in small or specialized environments. (This is a hidden benefit of part-time student labour at a university; it provides you with a steady stream of guinea pigs to test your documentation on.)
And of course for testing documentation on procedures, you need to try out the procedure in an environment where you can afford to fail. Which often means spare time and spare equipment, which can also be hard to get. (Testing overview and orientation documentation may be even harder; how do you test that someone learned enough from it to be useful?)
In the absence of good guinea pigs, your co-workers are the best testers you have. Unfortunately, it's usually difficult to get people to really take the time to go over the documentation carefully (unless people already get all this). After all, your co-workers either already know most of this stuff or don't need to, and they have other work to do.
(I've always liked to get pre-readers and the like, but it took a recent discussion of documentation to make me understand why and to see this issue.)
Sidebar: why you need other people
Documentation has to be tested by someone else for the same reason you can't thoroughly test your own programs; you're too close to it. You really need someone who's ignorant enough that they can't just fill in the blanks and the unclear bits on their own, without noticing.
(And part of the problem is that after a while, your initially ignorant person learns enough to start filling in the blanks on their own, and you need another guinea pig.)
Serious disaster recovery tests are the extreme example of this, where you send an accounting clerk out to the backup site with your instruction binder and a pile of backup tapes and see what happens.
Written on 19 August 2006.
* * *