Wandering Thoughts archives

The danger of having system programmers around, illustrated

My little issue the other day with DTrace makes a nice illustration of a variant of programmer laziness. As I sort of alluded to in the entry, although DTrace's function tracing limitation is documented, I didn't actually find it by reading the documentation. Instead I found it the hard and indirect way, by being a system programmer.

When I ran into a kernel function that DTrace couldn't trace, I wound up:

• using the Solaris kernel debugger to verify that the function was actually present in the kernel, and then disassembling some of the functions that called it to check that the function hadn't been inlined into its callers (which in retrospect was a silly thing to check).
• checking several other static functions to see if DTrace perhaps couldn't trace any static function. (It can.)
• searching high and low (with find and grep) to see if DTrace had some hardcoded database or configuration file of what kernel functions it would trace. (It doesn't; it uses existing symbol tables, and per above they include all static functions.)
• eventually I stumbled over the kernel code for DTrace's function tracing, skimmed through it, and wound up reading a nice comment that explained the whole situation.

Only after all the dust settled did I read the documentation and lo and behind, there it was documented for me (twice).

(This is far from the first time that I've done stuff like this. For example, I've read through Linux kernel source in order to work out when and where a specific errno result was generated. More than once.)

Now, if you are the right sort of system programmer, all of this makes perfect sense. The problem with documentation, as any sysadmin will frequently tell you, is that it is often incomplete, unreliable, or in fact outright wrong. When you are a system programmer (or have that mindset), the easy way to answer questions is not to waste time reading the fine manual, it is to proceed straight to authoritative things like reading kernel source and disassembling code. If you can see how the system actually works, you don't have to trust the manual, and you only have to be burned a couple of times by bad or incomplete documentation before this seems like a good idea.

Which is exactly the danger of having system programmers around: we're perfectly capable of going very overboard before we do basic things like checking documentation, because we think of things like disassembling code and reading kernel source as the fast way to do things. Sometimes we're right, but sometimes we're very wrong.

(It's like people with hammers, where you know that sooner or later every solution is going to involve nails, or at least hitting things.)