One of the consequences of yesterday's SAN backend failure is that I found a bug in our spares handling code. It was the kind of bug where the program aborts, and when that happens we started getting email with Python exception backtraces that ended in what struck me as a very odd exception:

KeyError: False

Normally KeyErrors make a kind of sense, in that you see what you're looking up and the question is either why you wound up using that key or why that key isn't present when you thought it would be. KeyError: False was so far out of left field that it left me lost until I finally stared at my code long enough to spot the problem.

It turns out that there is a world of difference between the following two (summarized) list comprehensions:

[... if x['vdev_state' == 'resilvering']]
[... if x['vdev_state'] == 'resilvering']

Yes indeed, looking up the results of a (constant) string equality comparison is rather different from looking up the value of a key and then comparing it with a string, even though the difference between the two is only a repositioned ']' (and the first doesn't look obviously wrong when you're skimming). For a start, one works and the other complains that you've never put False in your property dict.

(The classical short edit distance mistake is of course C's '=' for '==' error, which is not possible in Python.)

One awkward thing that this reveals is that I clearly never actually tested this code (whether with unit tests or just trying to make sure that it works). But that's another entry that reveals my ignorance of how to do TDD right.