Wandering Thoughts archives

The importance of test suites for standards

A while back I noted that very few of the web's standards have a test suite and that this could be a problem. You might reasonably ask why this matters, especially since so few standards in general have a test suite.

My answer is that having an official test suite for the standard does a lot of things:

• it lets you know if the implementation you've created actually conforms. Without this you're left with various sorts of ad-hoc tests that may be hard to set up and run (eg, do you interoperate with as many of the other implementations as possible in as many situations as possible).

• it means that everyone has the same idea of conformance and what the correct behavior is. Ideally this includes odd and unconventional behaviors, because the people who created the test suite looked for areas in the standard that could be missed or misunderstood and added tests to cover them.

• a test suite forces standard writers to be unambiguous about what the standard says. When people write tests, they also have to come up with what the results of the tests should be.

• the process of creating a test suite exercises the standard and thus helps to insure that it doesn't have subtle contradictions and that it is complete. These issues will also be discovered by attempting to implement the standard, but the advantage of a test suite is that it discovers these issues before the standard is frozen.

• the process of creating the test suite also makes sure that the standard's authors understand at least some of the implications of the standard's requirements before the standard is finalized. This is not as good as requiring an implementation, but it will at least find some of the problems.

All of these are good and praiseworthy things, but there's another way to look at the situation. The reality is that every standard needs a test suite and is going to get at least one. The only question is whether the 'test suite' will be written independently by every sane implementor, using whatever potentially mistaken ideas of the proper standard-compliant behavior that the implementor gathered from reading the standard a few times, or if the test suite will be created by people who know exactly what the standard requires because they wrote it.

(Every sane implementor needs a test suite because they need to test whether their implementation actually works right.)

(Yes, all of this is obvious and well known. I just feel like writing it down, partly to fix it in my mind.)