Wandering Thoughts archives

Why I do unit tests from inside my modules, not outside them

In reading about how to do unit testing, one of the divisions I've run into is between people who believe that you should unit test your code strictly through its external API boundaries and people who will unit test code 'inside' the module itself, taking advantage of internal features and so on. The usual arguments I've seen for doing unit tests from outside the module are that your API working is what people really care about and this avoids coupling your tests too closely to your implementation, so that you don't have the friction of needing to revise tests if you revise the internals. I don't follow this view; I write my unit tests inside my modules, although of course I test the public API as much as possible.

The primary reason why I want to test from the inside is that this gives me much richer and more direct access to the internal operation of my code. To me, a good set of unit tests involves strongly testing hypotheses about how the code behaves. It is not enough to show that it works for some cases and then call it a day; I want to also poke the dark corners and the error cases. The problem with going through the public API for this is that it is an indirect way of testing things down in the depths of my code. In order to reach down far enough, I must put together a carefully contrived scenario that I know reaches through the public API to reach the actual code I want to test (and in the specific way I want to test it). This is extra work, it's often hard and requires extremely artificial setups, and it still leaves my tests closely coupled to the actual implementation of my module code. Forcing myself to work through the API alone is basically testing theater.

(It's also somewhat dangerous because the coupling of my tests to the module's implementation is now far less obvious. If I change the module implementation without changing the tests, the tests may well still keep passing but they'll no longer be testing what I think they are. Oops.)

Testing from inside the module avoids all of this. I can directly test that internal components of my code work correctly without having to contrive peculiar and fragile scenarios that reach them through the public API. Direct testing of components also lets me immediately zero in on the problem if one of them fails a test, instead of forcing me to work backwards from a cascade of high level API test failures to find the common factor and realize that oh, yeah, a low level routine probably isn't working right. If I change the implementation and my tests break, that's okay; in a way I want them to break so that I can revise them to test what's important about the new implementation.

(I also believe that directly testing internal components is likely to lead to cleaner module code due to needing less magic testing interfaces exposed or semi-exposed in my APIs. If this leads to dirtier testing code, that's fine with me. I strongly believe that my module's public API should not have anything that is primarily there to let me test the code.)

Using go get alone is a bad way to keep track of interesting packages

When I was just starting with Go, I kept running into interesting Go packages that I wanted to keep track of and maybe use someday. 'No problem', I thought, 'I'll just go get them so I have them sitting around and maybe I'll look at them too'.

Please allow yourself to learn from my painful experience here and don't do this. Specifically, don't rely on 'go get' as your only way to keep track of packages you want to keep an eye on, because in practice doing so is a great way to forget what those packages are. There's no harm in go get'ing packages you want to have handy to look through, but do something in addition to keep track of what packages you're interested in and why.

At first, there was nothing wrong with what I was doing. I could easily look through the packages and even if I didn't, they sat there in $GOPATH/src so I could keep track of them. Okay, they were about three levels down from $GOPATH/src itself, but no big deal. Then I started getting interested in Go programs like vegeta, Go Package Store, and delve, plus I was installing and using more mundane programs like goimports and golint. The problem with all of these is that they have dependencies of their own, and all of these dependencies wind up in $GOPATH/src too. Pretty soon my Go source area was a dense thicket of source trees that intermingled programs, packages I was interested in in their own right, and dependencies of these first two.

After using Go seriously for not very long I've wound up with far too many packages and repos in $GOPATH/src to keep any sort of track of, and especially to remember off the top of my head which packages I was interested in. Since I was relying purely on go get to keep track of interesting Go packages, I have now essentially lost track of most of them. The interesting packages I wanted to keep around because I might use them have become lost in the noise of the dependencies, because I can't tell one from the other without going through all 50+ of the repos to read their READMEs.

As you might guess, I'd be much better off if I'd kept an explicit list of the packages I found interesting in some form. A text file of URLs would be fine; adding notes about what they did and why I thought they were interesting would be better. That would make it trivial to sort out the wheat from the chaff that's just there because of dependencies.

(These days I've switched to doing this for new interesting packages I run across, but there's some number of packages from older times that are lost somewhere in the depths of $GOPATH/src.)

PS: This can happen with programs too, but at least there tends to be less in $GOPATH/bin than in $GOPATH/src so it's easier to keep track of them. But if you have an ever growing $GOPATH/bin with an increasing amount of programs you don't actually care about, there's the problem again.

Lisp and data structures: one reason it hasn't attracted me

I've written before about some small scale issues with reading languages that use Lisp style syntax, but I don't think I've said what I did the other day on Twitter, which is that the syntax of how Lisp languages are written is probably the primary reason that I slide right off any real interest in them. I like the ideas and concepts of Lisp style languages, the features certainly sound neat, and I often use all of these in other languages when I can, but actual Lisp syntax languages have been a big 'nope' for a long time.

(I once wrote some moderately complex Emacs Lisp modules, so I'm not coming from a position of complete ignorance on Lisp. Although my ELisp code didn't exactly make use of advanced Lisp features.)

I don't know exactly why I really don't like Lisp syntax and find it such a turn-off, but I had an insight on Twitter. One of the things about the syntax of S-expressions is that they very clearly are a data structure. Specifically, they are a list. In effect this gives lists (yes, I know, they're really cons cells) a privileged position in the language. Lisp is lists; you cannot have S-expressions without them. Other languages are more neutral on what they consider to be fundamental data structures; there is very little in the syntax of, say, C that privileges any particular data structure over another.

(Languages like Pyhton privilege a few data structures by giving them explicit syntax for initializers, but that's about it. The rest is in the language environment, which is subject to change.)

Lisp is very clearly in love with lists. If it's terribly in love with lists, it doesn't feel as if it can be fully in love with other data structures; whether or not it's actually true, it feels like other data structures are going to be second class citizens. And this matters to how I feel about the language, because lists are often not the data structure I want to use. Even being second class in just syntax matters, because syntactic sugar matters.

(In case it's not clear, I do somewhat regret that Lisp and I have never clicked. Many very smart people love Lisp a lot and so it's clear that there are very good things there.)

States in a state machine aren't your only representation of state

I think in terms of state machines a lot; they're one of my standard approaches to problems and I wind up using them quite a bit. I've recently been planning out a multi-threaded program that has to coordinate back and forth between threads as they manipulate the state of host authentication. At first I had a simple set of states, then I realized that these simple states only covered the main flow of events and needed to be more and more complicated, and then I had a blinding realization:

Not all state needs to be represented as state machine states.

When you have a state machine it is not so much tempting as obvious to represent every variation in the state of your entities as another state machine state. But if you do this, then like me you may wind up with an explosion of states, many of which are extremely similar to each other and more or less handled the same way. This isn't necessary. Instead, it's perfectly sensible to represent certain things as flags, additional or detailed status fields, or the like. If you want to mark something as going to be deleted once it's unused, there's no need to add new states to represent this if you can just add a flag. If you have three or four different ways for something to fail and they all lead to basically the same processing, well, you don't need three or four different states for 'failed X way'; you can have just one 'failed' state and then another field with the details of why.

Off the top of my head now, I think that states are best for things that have a different flow of processing (ideally a significantly different flow). The more both the origin state and the processing of two 'states' resembles each other, the less they need to be separate states and the more the difference can be captured in a different variable or field (and then handled in the code with only some ifs).

(On the other hand, if two different states were handled the same way but came from different origin states and transitioned to different destination states, I think I'd definitely keep them as separate states and just share the common code somehow. This would preserve the clarity of state flow in the system. Although if two separate states needed exactly the same handling in the code, I might think I was overlooking something about the states in general.)

A drawback in how DWiki parses its wikitext

In my initial installment on how DWiki parses its wikitext I said that one important thing DWiki does is that it has two separate parsers:

[...] One parser handles embedded formatting in running text (things like fonts, links, and so on) and the other one handles all of the line oriented block level structures like paragraphs, headers, blockquotes, lists, etc. What makes it work is that the block level parser doesn't parse running text immediately for multi-line things like paragraphs; [...]

This sounds great and in general it is perfectly fine, but it does turn out to impose one restriction on your wiki dialect: it doesn't support block-level constructs that require looking ahead into running text. To work right, this requires that all block level constructs can be recognized before you have to start parsing running text, which means that they have to all come at the start of the line.

This doesn't sound like a particularly onerous restriction on your wikitext dialect, but it actually causes DWiki heartburn in one spot. In my wikitext dialect, definition lists are written as:

- first the <dt> text: And post-colon is the <dd> text.

This looks like a perfectly natural way to write a definition list entry, but phrased this way it requires block level parsing to look ahead into the line to recognize and find the ':' that separates the <dt> text from the <dd> text. Now suppose that you want to have a link to an outside website in the <dt> text, which of course is going to contain a ':' in the URL. Oops. Similar issues come up if you just want a : in the <dt> text for some reason. As a result DWiki's parsing of definition lists basically disallows a lot of stuff in the <dt> text, which has led me to not use them very much.

(The other problem with this definition is that it restricts the <dt> text to a single line.)

I think that this may also cause problems for natural looking tables. Most of the ways of writing natural tables are going to rely on interior whitespace to create visible columns and thus demonstrate that this is a table. Looking ahead in what would otherwise be running text to spot runs of whitespace is less dangerous than trying to find a character in a line, but it still breaks this pure separation.

(I didn't think of this issue when I wrote my first entry with its enthusiastic praise; sadly it's a corner case that's easy to forget about most of the time.)

Sidebar: DWiki's table parsing also sort of breaks the rule

DWiki doesn't need to look ahead in running text to know that it's processing a table, but it does have to peek into the running text to find column dividers. This is at least impure, but so far I think it's less annoying than the definition list case; in practice the column dividers don't seem to naturally occur in my table text so far. Still, it's not an easy problem and I'd like a better solution.

(One approach is to be able to tell the running text parser to stop if it runs into a certain character sequence in unquoted text. I think that this works best if you have an incremental parser for running text that can be fed input, parse it as much as possible, and then suspend itself to wait for more.)

In Go I've given up and I'm now using standard packages

In my Go programming, I've come around to an attitude that I'll summarize as 'there's no point in fighting city hall'. What this means is that I'm now consciously using standard packages that I don't particularly like just because they are the standard packages.

I'm on record as disliking the standard flag package, for example, and while I still believe in my reasons for this I've decided that it's simply not worth going out of my way over it. The flag package works and it's there. Similarly, I don't think that the log package is necessarily a great solution for emitting messages from Unix style command line utilities but in my latest Go program I used it anyways. It was there and it wasn't worth the effort to code warn() and die() functions and so on.

Besides, using flag and log is standard Go practice so it's going to be both familiar to and expected by anyone who might look at my code someday. There's a definite social benefit to doing things the standard way for anything that I put out in public, much like most everyone uses gofmt on their code.

In theory I could find and use some alternate getopt package (these days the go to place to find one would be godoc.org). In practice I find using external packages too much of a hassle unless I really need them. This is an odd thing to say about Go, considering that it makes them so easy and accessible, but depending on external packages comes with a whole set of hassles and concerns right now. I've seen a bit too much breakage to want that headache without a good reason.

(This may not be a rational view for Go programming, given that Go deliberately makes using people's packages so easy. Perhaps I should throw myself into using lots of packages just to get acclimatized to it. And in practice I suspect most packages don't break or vanish.)

PS: note that this is different from the people who say you should eg use the testing package for your testing because you don't really need anything more than what it provides and stick with the standard library's HTTP stuff rather than getting a framework. As mentioned, I still think that flag is not the right answer; it's just not wrong enough to be worth fighting city hall over.

Sidebar: Doing standard Unix error and warning messages with log

Here's what I do:

log.SetPrefix("<progname>: ")
log.SetFlags(0)

If I was doing this better I would derive the program name from os.Args[0] instead of hard-coding it, but if I did that I'd have to worry about various special cases and no, I'm being lazy here.

My experience doing relatively low level X stuff in Go

Today I wound up needing a program that spoke the current Firefox remote control protocol instead of the old -remote based protocol that Firefox Nightly just removed. I had my choice between either adding a bunch of buffer mangling to a very old C program that already did basically all of the X stuff necessary or trying to do low-level X things from a Go program. The latter seemed much more interesting and so it's what I did.

(The old protocol was pretty simple but the new one involves a bunch of annoying buffer packing.)

Remote controlling Firefox is done through X properties, which is a relatively low level part of the X protocol (well below the usual level of GUIs and toolkits like GTK and Qt). You aren't making windows or drawing anything; instead you're grubbing around in window trees and getting obscure events from other people's windows. Fortunately Go has low level bindings for X in the form of Andrew Gallant's X Go Binding and his xgbutil packages for them (note that the XGB documentation you really want to read is for xgb/xproto). Use of these can be a little bit obscure so it very much helped me to read several examples (for both xgb and xgbutil).

All told the whole experience was pretty painless. Most of the stumbling blocks I ran into were because I don't really know X programming and because I was effectively translating from an older X API (Xlib) that my original C program was using to XCB, which is what XGB's API is based on. This involved a certain amount of working out what old functions that the old code was calling actually did and then figuring out how to translate them into XGB and xgbutil stuff (mostly the latter, because xgbutil puts a nice veneer over a lot of painstaking protocol bits).

(I was especially pleased that my Go code for the annoying buffer packing worked the first time. It was also pretty easy and obvious to write.)

One of the nice little things about using Go for this is that XGB turns out to be a pure Go binding, which means it can be freely cross compiled. So now I can theoretically do Firefox remote control from essentially any machine I remotely log into around here. Someday I may have a use for this, perhaps for some annoying system management program that insists on spawning something to show me links.

(Cross machine remote control matters to me because I read my email on a remote machine with a graphical program, and of course I want to click on links there and have them open in my workstation's main Firefox.)

Interested parties who want either a functional and reasonably commented example of doing this sort of stuff in Go or a program to do lightweight remote control of Unix Firefox can take a look at the ffox-remote repo. As a bonus I have written down in comments what I now know about the actual Firefox remote control protocol itself.

Thinking about how to create flexible aggregations from causes

Every so often I have programming puzzles that I find frustrating, not so much because I can't solve them as such but because I feel that there must already be a solution for them if I could both formulate the problem right and then use that formulation to search existing collections of algorithms and such. Today's issue is a concrete problem I am running into with NFS activity monitoring.

Suppose that you have a collection of specific event counters, where you know that userid W on machine X working against filesystem Y (in ZFS pool Z) did N NFS operations per second. My goal is to show aggregate information about the top sources of operations on the server, where a source might be one machine, one user, one filesystem, one pool, or some combination of these. This gives me two problems.

The first problem is efficiently going 'upwards' to sum together various specific event counters into more general categories (with the most general one being 'all NFS operations'). This feels like I want some sort of clever tree or inverted tree data structure, but I could just do it by brute force since I will probably not be dealing with too many specific event counters at any one time (from combinations we can see that each 4-element specific initial event maps to 16 categories; this is amenable to brute force on modern machines).

The second problem is going back 'down' from a category sum to the most specific cause possible for it so that we can report only that. The easiest way to explain this is with an example; if we have (user W, machine X, fs Y, pool Z) with 1000 operations and W was the only user to do things from that machine or on that filesystem, we don't want a report that lists every permutation of the machine and filesystem (eg '1000 from X', '1000 against Y', '1000 from X against Y', etc). Instead we want to report only that 1000 events came from user W on machine X doing things to filesystem Y.

If I wind up with a real tree, this smells like a case of replacing nodes that have only one child with their child (with some special cases around the edges). If I wind up with some other data structure, well, I'll have to figure it out then. And a good approach for this might well influence what data structure I want to use for the first problem.

If all of this sounds like I haven't even started trying to write some code to explore this problem, that would be the correct impression. One of my coding hangups is that I like to have at least some idea of how to solve a problem before I start trying to tackle it; this is especially the case if my choice of language isn't settled and I might want to use a different solution depending on the language I wind up in.

(There are at least three candidate languages for what I want to do here, including Go if I need raw speed to make a brute force approach feasible.)

Don't split up error messages in your source code

Every so often, developers come up with really clever ways to frustrate system administrators and other people who want to go look at their code to diagnose problems. The one that I ran into today looks like this:

if (rval != IDM_STATUS_SUCCESS) {
	cmn_err(CE_NOTE, "iscsi connection(%u) unable to "
	    "connect to target %s", icp->conn_oid,
	    icp->conn_sess->sess_name);
	idm_conn_rele(icp->conn_ic);
}

In the name of keeping the source lines under 80 characters wide, the developer here has split the error message into two parts, using modern C's constant string concatenation to have the compiler put them back together.

Perhaps it is not obvious why this is at least really annoying. Suppose that you start with the following error message in your logs:

iscsi connection(60) unable to connect to target <tgtname>

You (the bystander, who is not a developer) would like find the code that produces this error message, so that you can understand the surrounding context. If this error message was on one line in the code, it would be very easy to search for; even if you need to wild-card some stuff with grep, the core string 'unable to connect to target' ought to be both relatively unique and easy to find. But because the message has been split onto multiple source lines, it's not; your initial search will fail. In fact a lot of substrings will fail to find the correct source of this message (eg 'unable to connect'). You're left to search for various substrings of the message, hoping both that they are unique enough that you are not going to be drowned in hits and that you have correctly guessed how the developer decided to split things up or parameterize their message.

(I don't blame developers for parameterizing their messages, but it does make searching for them in the code much harder. Clearly some parts of this message are generated on the fly, but are 'connect' or 'target' among them instead of being constant part of the message? You don't know and have to guess. 'Unable to <X> to <Y> <Z>' is not necessarily an irrational message format string, or you equally might guess 'unable to <X> to target <Z>'.)

The developers doing this are not making life impossible for people, of course. But they are making it harder and I wish they wouldn't. It is worth long lines to be able to find things in source code with common tools.

(Messages aren't the only example of this, of course, just the one that got to me today.)

Go is mostly easy to cross-compile (with notes)

One of the things I like about Go is that it's generally very easy to cross-compile from one OS to another; for instance, I routinely build (64-bit) Solaris binaries from my 64-bit Linux instead of having to maintain a Solaris or OmniOS Go compilation environment (and with it all of the associated things I'd need to get my source code there, like a version of git and so on). However when I dug into the full story in order to write this entry, I discovered that there are some gaps and important details.

So let's start with basic cross compilation, which is the easy and usual bit. This is well covered by eg Dave Cheny's introduction. The really fast version looks like this (I'm going to assume a 64-bit Linux host):

cd /some/where
hg clone https://code.google.com/p/go
cd go/src
./all.bash
export GO386=sse2
GOOS=linux GOARCH=386 ./make.bash --no-clean
GOOS=solaris GOARCH=amd64 ./make.bash --no-clean
GOOS=freebsd GOARCH=386 ./make.bash --no-clean

(See Installing Go from source for a full discussion of these environment variables.)

With this done, we can build some program for multiple architectures (and deploy the result to them with just eg scp):

cd $HOME/src/call
go build -o call
GOARCH=386 go build -o call32
GOOS=solaris GOARCH=amd64 go build -o call.solaris

(Add additional architectures to taste.)

This generally works. I've done it for quite some time with good success; I don't think I've ever had such a cross-compiled binary not work right, including binaries that do network things. But, as they say, there is a fly in the ointment and these cross-compiled binaries are not quite equivalent to true natively compiled Go binaries.

Go cross-compilation has one potentially important limit: on some platforms, Linux included, true native Go binaries that use some packages are dynamically linked into the C runtime shared library and some associated shared libraries through Cgo (see also). On Linux I believe that this is necessary to use the true native implementation of anything that uses NSS; this includes hostname lookup, username and UID lookup, and group lookups. I further believe that this is because the native versions of this use dynamically loaded C shared libraries that are loaded by the internals of GNU libc.

Unfortunately, Cgo does not cross-compile (even if you happen to have a working C cross compiler environment on your host, as far as I know). So if you cross-compile Go programs to such targets, the binaries run but they have to emulate the native approach and the result is not guaranteed to give you identical results. Sometimes it won't work at all; for example os/user is unimplemented if you cross-compile to Linux (and all username or UID lookups will fail).

(One discussion of this is in Alan Shreve's article, which was a very useful source for writing this entry.)

Initially I thought this was no big deal for me but it turns out that it potentially is, because compiling for 32-bit Linux on 64-bit Linux is still cross-compiling (as is going the other way, from 32-bit host to 64-bit target). If you build your Go environment on, say, a 64-bit Ubuntu machine and cross-compile binaries for your 32-bit Ubuntu machines, you're affected by this. The sign of this happening is that ldd will report that you have a static executable instead of a dynamic one. For example, on 64-bit Linux:

; ldd call32 call64
call32:
        not a dynamic executable
call64:
        linux-vdso.so.1 =>  (0x00007ffff2957000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3be5111000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f3be4d53000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3be537a000)

If you have both 64-bit and 32-bit Linux and you want to build true native binaries on both, at least as far as the standard packages go, you have to follow the approach from Alan Shreve's article. For me, this goes like the following (assuming that you want your 64-bit Linux machine to be the native version, which you may not):

1. erase everything from $GOROOT/bin and $GOROOT/pkg
2. run 'cd src; ./make.bash' on the 32-bit machine
3. rename $GOROOT/pkg/linux_386 to some other name to preserve it
4. build everything on your 64-bit machine, including the 32-bit cross-compile environment.
5. delete the newly created $GOROOT/pkg/linux_386 directory hierarchy and restore the native-built version you saved in step 3.

If you're building from source using the exact same version from the Mercurial repository it appears that you can extend this to copying the pkg/$GOOS_$GOARCH directory between systems. I've tested copying both 32-bit Linux and 64-bit Solaris and it worked for me (and the resulting binaries ran correctly in quick testing). This means that you need to build Go itself on various systems but you can get away with doing all of your compilation and cross-compilation only on the most convenient system for you.

(I suspect but don't know that if you have any Cgo-using packages you can copy $GOPATH/pkg/$GOOS_$GOARCH around from system to system to get functioning native versions of necessary packages. Try it and see.)

Even with this road bump the pragmatic bottom line is that Go cross-compilation is easy, useful, and is probably going to work for your Go programs. It's certainly easy enough that you should give it a try just to see if it works for you.