Why I do unit tests from inside my modules, not outside them

December 9, 2014

In reading about how to do unit testing, one of the divisions I've run into is between people who believe that you should unit test your code strictly through its external API boundaries and people who will unit test code 'inside' the module itself, taking advantage of internal features and so on. The usual arguments I've seen for doing unit tests from outside the module are that your API working is what people really care about and this avoids coupling your tests too closely to your implementation, so that you don't have the friction of needing to revise tests if you revise the internals. I don't follow this view; I write my unit tests inside my modules, although of course I test the public API as much as possible.

The primary reason why I want to test from the inside is that this gives me much richer and more direct access to the internal operation of my code. To me, a good set of unit tests involves strongly testing hypotheses about how the code behaves. It is not enough to show that it works for some cases and then call it a day; I want to also poke the dark corners and the error cases. The problem with going through the public API for this is that it is an indirect way of testing things down in the depths of my code. In order to reach down far enough, I must put together a carefully contrived scenario that I know reaches through the public API to reach the actual code I want to test (and in the specific way I want to test it). This is extra work, it's often hard and requires extremely artificial setups, and it still leaves my tests closely coupled to the actual implementation of my module code. Forcing myself to work through the API alone is basically testing theater.

(It's also somewhat dangerous because the coupling of my tests to the module's implementation is now far less obvious. If I change the module implementation without changing the tests, the tests may well still keep passing but they'll no longer be testing what I think they are. Oops.)

Testing from inside the module avoids all of this. I can directly test that internal components of my code work correctly without having to contrive peculiar and fragile scenarios that reach them through the public API. Direct testing of components also lets me immediately zero in on the problem if one of them fails a test, instead of forcing me to work backwards from a cascade of high level API test failures to find the common factor and realize that oh, yeah, a low level routine probably isn't working right. If I change the implementation and my tests break, that's okay; in a way I want them to break so that I can revise them to test what's important about the new implementation.

(I also believe that directly testing internal components is likely to lead to cleaner module code due to needing less magic testing interfaces exposed or semi-exposed in my APIs. If this leads to dirtier testing code, that's fine with me. I strongly believe that my module's public API should not have anything that is primarily there to let me test the code.)

Comments on this page:

By tilboerner at 2014-12-09 04:44:18:

I've found that, in general, testing "from the outside" forces me to adopt a much cleaner architecture, just in order to make the code easily testable. Vice versa, if I find it hard to set up tests, or if I get that uneasy feeling that I'm "cheating" by using internal knowledge, I take it as a strong hint that there's room for improvement in some aspect; API design and coupling are hot candidates. The bugs I run into can usually be exposed by a test simulating some external constraint (in other words: a previoulsy missing API test case).

However, I think there are cases where it can be very helpful to rely on some piece of insider knowledge for a test, or even to directly test an internal component. (Lucky if you can.) Ususally, if I do the former, I need to follow up with the latter, in order to fix the knowledge in a test case and get a heads up when this premise changes. I try to avoid it, though, and given the choice, I'd rather change the original code.

By Dan Wiebe at 2014-12-16 07:47:55:

I used to be a big proponent of "testing through the side door" as well, for most of the reasons you mention.

These days, though, I tend to decompose and decompose until I get units that are easily tested through the front door.

For example, in the Babysitter kata, I might be driving the rate calculator and need to write the time-mapping function, which will be buried in a private method and hard to get at with a test.

So I say to myself, "You know, the root of the problem here is that rate calculation and time mapping are really separate responsibilities. They ought to be handled by different units." I'll put the time-mapping code in its own function or tiny single-method class, and suddenly it's cake to test through the front door.

Do I inject the time mapper into the rate calculator at runtime, or do I just have the rate calculator call or construct it directly? Judgment call. If it's complicated or mocking it will allow me to drive edge-case handling in the client, I'll inject it; but in the case of the Babysitter time mapper, probably I won't bother with injection.

Written on 09 December 2014.
« Why I don't believe in generic TLS terminator programs
How to delay your fileserver replacement project by six months or so »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue Dec 9 00:35:03 2014
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.