2013-12-25
Procedures are not documentation
In the introduction to this good and interesting Sysadvent entry on automatically testing your web security I hit a sentence that raised my hackles right up:
Writing automated tests for your code is one of those things that, once you have gotten into it, you never want to see code without tests ever again. Why write pages and pages of documentation about how something should work when you can write tests to show exactly how something does work? [...]
(Emphasis mine.)
I cannot put this strongly enough: WRONG. Procedures (including tests, checklists, and even configuration management) are not documentation and cannot substitute for it. You really do want to have actual documentation (although good procedures can reduce how much of it you need).
Procedures more or less tell you the 'what' of your systems, but they do not give you any background behind that 'what'. Contrary to the implication in the sentence, they often can't really tell you how things work (just that they do). They can't give you the 'why' of the design and set up of your system, the rationale and logic behind it; knowing this is crucial information for making good modifications to the system later. They don't distinguish between core features and requirements and things that are just accidents of the current implementation. Procedures also can't tell you why you aren't doing something, which can be quite useful to know.
Tests are especially prone to this problem because tests by their nature must be specific and they're totally mute about the why of the test. For example, if you're doing a 'does it work' test of a web server and looking for specific HTML output is that specific HTML output a requirement or simply a recognition signal, so that it could be any other recognizable chunk of HTML on that page?
(Also, a test suite is almost always an imperfect reflection of reality, which means that if you attempt to reverse engineer reality from your test suite you are guaranteed to get something wrong.)
Note that these are not new issues. Programmers have an entire set of practices around writing good tests, detecting bad tests, understanding tests, and so on.
(Documentation surrounding procedures can tell you this information, if it exists and is maintained, but that's different. That's actual documentation. Some people will immediately say that you should never write tests or other procedures without at least some surrounding documentation to give at least basic context and I'll fully agree with them, but that's not quite the position being advocated here.)