Wandering Thoughts archives

Testing versus extensibility

One of the test driven development mantras that I've heard is 'if you haven't tested it, it doesn't work'. Which leaves me with a question that I don't know the answer to: how do you test an extendible format?

Suppose that you have a data file format (for example, an XML dialect) that allows more or less arbitrary extensions to be embedded in it, and also that you have some agreement on how your program is supposed to handle extension elements that aren't known to it (ignore them, silently pass them through, or whatever). How do you test that your program really can handle random extensions to the base format?

(Clearly this is one of the times when 'write the minimum amount of code to pass the test' is not what you should do.)

I suspect that the pragmatic answer is that you write test cases for all of the various ways and places that extensions can appear, and probably extra unit tests for boundary conditions in your code, and declare that if it works with your test cases, it should work for any random format extension. (Then if you find a counter-example in the field, you add more tests.)

This must be much more of an issue for test suites, which are both much more black box and which can run serious risks of people coding to the tests instead of the specification. Possibly one can exhaustively generate enough random extensions (covering all of the special cases that the specification allows) so that the easiest way to write a program that can pass the test suite is to have it handle extensions right in general.

Undo is sometimes not good enough

There is a tendency in programming to consider any spiffy new general feature, such as a good infinite undo system, as the way to deal with many of your program's problems. This is often a mistake, and undo makes a good illustration of why.

(It's easy to see how this happens; you have this powerful system that you've invested a lot of effort into, so it's natural to apply it on as much as possible. And programmers really like general solutions, because they have an appealing clarity and simplicity.)

Consider a non-destructive photo editor with a good undo implementation but where there is no way to turn a photo modification off; instead, you are supposed to just undo it if you don't like it. Now imagine that you are working on a photo: you make a modification (say darkening the shadows a bunch), make a whole bunch of further modifications, and then decide that maybe dark shadows aren't the right thing after all and you'd like to see how your picture looks with normal shadows.

Well, now you have a problem. Yes, you can undo the dark shadows (the system has infinite undo), but in order to get there you will have to reverse all of your other modifications. Among other things this is self-defeating, because you can't see what the picture looks like with your other modifications but with normal shadows.

(Technically you can; you just have to revert to normal shadows, remember and carefully re-do all of your other modifications, and then at the end apply the dark shadows modification. Right now you should be having version control flashbacks.)

The problem with undo here is that it is inherently time-based, and thus it is imposing an artificial ordering on something that is unordered or close to it (as the user sees it). This doesn't make time-based undo bad (sometimes the user really will be thinking that way), but it does make it insufficient. Undo cannot be the answer to all of your problems, even though it's a nice general feature.