Wandering Thoughts archives

One of my test based development imperfections: use driven testing

Recently I've been coding what I call a 'sinkhole SMTP server' in Go (a SMTP server that doesn't do anything messages except perhaps save them to disk). Over the process of this I've once again gotten to watch one of my (bad) habits with test driven development in action, something that I will call 'use driven testing'.

A SMTP server can conceptually be divided into at least two layers. One layer is a general layer that handles the SMTP protocol; it reads SMTP commands from the network, makes sure they are actually commands, enforces command ordering requirements, handles a bunch of fiddly stuff, and so on. A second layer sits atop this and handles higher level policy issues like what email addresses are acceptable (as source or destination addresses) and what to do with received email messages. The bottom layer is generic (you could in theory reuse it in any number of different SMTP servers) while the top layer is generally specific to your needs.

I started out writing the bottom layer as a Go package. Go has reasonably good support for testing, so I wrote tests for this layer's functionality of parsing SMTP command lines and handling the basic flow of the protocol. In other words, I did relatively conventional test focused development; I wrote code and then wrote tests to make sure it work, and sometimes I mutated the code some to make it easier to test. But at a certain point the general base SMTP layer passed the point where I could start writing the top layer. At this point I switched to writing the top layer and mutating the base layer as necessary to make a better API or to make things work. I didn't write any new base layer tests for the base layer's added functionality and I didn't write tests for the top layer; testing for the top layer consisted of actually using it. This is the switch to what I'm calling 'use driven testing', where actually using my code is how I test it.

This is flawed and imperfect but it's hard for me to see how to get out of it. Writing top layer code, changing the bottom layer to match, and then going on to write bottom layer tests seems like make work or wasted work. I have to have the top layer code and the bottom layer tests basically duplicate that work without telling me much extra. Of course this is wrong; writing tests will tell me not just if something works now but whether it keeps on working. But it's hard to feel motivated to do the extra work and it's also hard to shape an API for both of convenient testing and the convenience of higher layer stuff.

(There's also the related question of how much stuff in the higher layer I want to test and what the best way to test it is. I think that Go will let me write tests for code in the main package that makes up your program, but I haven't actually verified that.)

Okay, let me admit something else about this: writing live code is a lot more fun than writing tests. When I write top layer code, my program does something real for me. When I write more tests, yay, more tests (which may break and have to be redone if I restructure what my actual productive code does). It's very hard to avoid the fun and do drudgery, especially when I'm doing this entirely for fun. At the same time I wind up feeling guilty for having minimal tests and chunks of code that are only tested through use by the higher level.

Complicating this is that some of the functionality I wound up putting in the lower layer is not straightforward to test. For example, how do I test that TLS negotiation actually works or that network IO is (optionally) being written at an artificially slow rate of one character every tenth of a second? There are probably clever ways but they're not obvious to me, and it's hard to feel hugely motivated when I can test these using the live program by inspection or by using swaks.

(I have considered the merits of automatedly hooking the Go SMTP client up to my server and verifying that it, for example, sees the expected SMTP extensions. Maybe this is actually the right answer.)

I don't have any answers here, just stuff that I'm thinking about aloud. Although perhaps my use driven testing is not completely crazy and at some point I should just accept that high level tests of functionality are fine (even if some of them are manual).

PS: part of the pain here is that testing the output of a SMTP server is kind of a pain in the rear. It's easy enough to test the literal output in response to a series of commands but that's both verbose and it blows up any time you change the messages the server sends (which discourages changing those messages, which to me is bad). Doing better requires building some moderately complex testing infrastructure to extract, say, the sequence of SMTP response codes that you expect so you don't care about the literal text.

In Go, sometimes a nil is not a nil

Today I ran into what turns out to be a Go FAQ. First, as a little Go quiz, see if you can tell what this program should print before you run it in the Go playground (I've put the program in a sidebar in case it ever goes away on the playground). The core of the program is this code:

type fake struct { io.Writer }

func fred (logger io.Writer) {
   if logger != nil {
      logger.Write([]byte("..."))
   }
}

func main() {
   var lp *fake
   fred(nil)
   fred(lp)
}

Since Go variables are explicitly created with their zero values, which in the case of pointers such as lp is nil, you would expect this code to run (but do nothing). In fact it crashes on the second call to fred(). What is happening is that sometimes in Go what started out as a nil value and will look like a nil value if you print it straightforwardly is not in fact a nil value. In a nutshell, Go distinguishes between nil interface values and nil values of concrete types that are converted to interfaces. Only the former is really nil and so will compare equal to a plain nil, as fred() attempts to do here.

(As a corollary of this, your concrete methods on (f *fake) can get called with a nil f value. It may be a nil pointer, but it's a typed nil pointer and so it can have methods. It can even have methods through interfaces, as is the case here.)

For the situation I found myself in, the way to deal with this is to change the setup procedure. The real program set up fake conditionally, something like:

var l *sLogger

if smtplog != nil {
   l = &sLogger
   l.prefix = logpref
   l.writer = bufio.NewWriterSize(smtplog, 4096)
}
convo = smtpd.NewConvo(conn, l)

This passes a nil of the concrete type '* sLogger' to something that expected an io.Writer, causing interface conversion and hiding the nil. To get around it we can just add a level of indirection with an io.Writer variable that must be explicitly set:

var l2 io.Writer

if smtplog != nil {
   l := &sLogger
   l.prefix = logpref
   l.writer = ....
   l2 = l
}
convo = smtpd.NewConvo(conn, l2)

If we are not initializing our special logging, l2 stays a pure and real io.Writer nil and will be detected as such down in the depths of the code in the smtpd package.

(You can do a similar trick by pulling the setup out into a function that has a return type of io.Writer and explicitly returns a nil in the case of no logging. But you have to return the interface type; if you give your setup function a return type of '* sLogger' you'll have the same problem all over again.)

It's a matter of taste if you want to keep nil guard code in your sLogger method functions. In the end I decided that I didn't want to; if I get this sort of initialization wrong in the future in this code, I want a crash so I can fix it.

The other lesson I've learned from this is that if I'm printing values for debugging purposes where I might run into this issue I don't want to use %v as the format specifier, I want to use %#v. The former will print a plain and misleading '<nil>' for both the nil interface and nil concrete type cases, while the latter will print '<nil>' for the former and something like '(*main.fake)(nil)' for the latter.

Sidebar: the test program

package main

import (
    "fmt"
    "io"
)

type fake struct {
    io.Writer
}

func fred(logger io.Writer) {
    if logger != nil {
        logger.Write([]byte("a test\n"))
    }
}

func main() {
    // this is born <nil>
    var t *fake

    fred(nil)
    fmt.Printf("passed 1\n")
    fred(t)
    fmt.Printf("passed 2\n")
}

Some things for enumerated constants in Go

The standard way to do C style enums in Go is with iota and a typed set of constants. This looks like this:

type SmtpCmd int

const (
    EHLO SmtpCmd = iota
    MAILFROM
    RCPTTO
    ....
)

This set of constants is perfectly fine in one sense, but it has a non-obvious drawback (at least for someone coming from a Python world): you can't tell an uninitialized instance of SmtpCmd from an EHLO value. Like all Go types, SmtpCmd has a zero value and since it's ultimately an integer the zero value is, well, 0. iota starts numbering from 0 so EHLO is also 0. Having your zero value overlap with a valid value that you can see normally has the possibility of hiding errors like uninitialized fields or unset return values. Instead of being glaringly obvious it will just look like your code is somehow incorrectly determining that you have an EHLO.

The simple fix for this is to introduce a dummy first value:

const (
    unsetCmd SmtpCmd = iota
    EHLO
    ....
)

(We make unsetCmd start with a lower case letter so that it won't be visible from outside the package. This may or may not be a good idea depending on whether outside people generate instances of SmtpCmd or just look at them.)

The other thing you can do in Go with a typed set of constants is to pretty-print them for the purposes of the %v formatting operator. All this takes is a String() method function defined for your type. Doing a minimal version that annotates the string value with type information is simple but perhaps not completely useful:

func (v SmtpCmd) String() string {
    return fmt.Sprintf("<SmtpCmd %d>", v)
}

More sophisticated versions are possible if you have either a source of number to name mapping information or just build one yourself. You probably don't need it to be strikingly efficient because you probably won't be using %v on your enumerated types very often.

(One thing you can do in even a simple version is specifically annotate unsetCmd and perhaps anything that's out of range for your actual defined constants.)

Sidebar: A potential trick with String() methods

In many cases being able to use %v on your package's values is most useful when debugging test failures. In addition, printing useful information about them may involve crusty code that you don't really like sitting around in the main package source.

As far as I can tell, you can actually put the String() method definitions in your *_test.go files and have them work. The *_test.go packages are an internal part of your package when built for tests, but I believe that stuff defined in them is not part of your package's exported functionality and the crufty and internals specific pretty-printing code can live with the rest of your testing code.

(Of course this may cause a certain amount of confusion someday, when your package internal tests print out nice looking and informative %v values for things while in your outside code they just gets eg basic numbers. This is your call to make. I honestly don't know which side I come down on right now, although I get a half-pass because I'm not writing a package I expect anyone else to ever use.)

Why Java is a compiled language and Python is not

A commentator on Charles Marsh's article Why are there so many Pythons asked an interesting question:

[...] If both Java and Python produces Bytecodes which in-turn is run by VM's why Java is called as Compiled language and Python is called Interpreted language? [...]

One comment's answer was 'marketing', which in a sense is correct; one reason we call Java a compiled language is that that's what Sun called it from the start. Another comment noted that Java has an explicit compilation phase that is separate from having the JVM execute your Java program by interpreting the bytecodes. All of this points us towards what I feel is the real answer:

In Java, bytecode is a first class object. In Python it's an internal implementation detail.

You've always been able to find specific documentation on the JVM and its bytecodes; as far as I know they were released along side the first definitions of Java the language. JVM bytecode has never been marked 'for internal use only' and in fact it's explicitly been a stable, public representation of Java programs. For example you don't download the source code for Java applets into your browser, you download the JVM bytecode (and in general JVM bytecode is the common delivery method of Java code even on servers). And the JVM guarantees that this will work regardless of what sort of machine you have (and in theory guarantees to make it work securely even in the face of maliciously generated bytecode).

This public visibility, this treatment of bytecode as a real part of the language environment with its own documentation and guarantees and specifications, makes JVM bytecode a first class object in the overall Java ecosystem. In turn this makes it accurate to say that Java programs are (usually) compiled to JVM bytecode and then executed by an interpreter of that bytecode. This is especially so when the language's implementation makes these two explicitly separate steps with a whole collection of artifacts that are this compiled bytecode.

In Python there is nothing like this; CPython's use of bytecode is just an internal detail of the interpreter implementation. CPython bytecode is not part of Python semantics, it is not really documented, and you are not really supposed to generate your own. There's no guarantee of any cross-version compatibility, as CPython bytecode is expected to be consumed by the same interpreter that generated it. And so on. CPython bytecode is an interesting curiosity, not a first class part of (C)Python.

(It's technically accurate to say that CPython compiles Python source to bytecodes and then interprets these bytecodes. But this is a narrow technicality, not how people really see CPython working, and can be said about any number of languages that we normally think of as interpreted.)

The advantages of editors over database programs for modifying your data

Yesterday I mentioned that one advantages of using plain text files is that you could modify your data with an editor instead of having to use some sort of database program (whether a SQL shell, some shell scripting on top of that, or custom code you write). One reason this is an advantage is that someone else has already written your editor of choice and you already know how to use it, but another part of this is that editors are often genuinely better environments to modify your data in.

There are at least two aspects of this. The first is that any decent editor has not just undo but multi-level undo that crosses over file save boundaries. In most editors you can revert your entire set of modifications all the way up until you quit out of the editor while at the same time making your modifications visible to other programs (by saving the file or files). Actual database programs generally can't match this level of support for 'oh oops, better revert that' moments, partly by design.

(Most editors are implicitly built around being the sole modifier of the files you're working on. Databases are generally built around assuming multiple people are modifying things on an ongoing basis.)

The second is that editors make your changes fully visible and also show you the surrounding context (or at least a screen's worth). You literally see your data sitting there on the screen. This makes it much easier to notice undesirable changes and other accidents, especially large scale ones; if you mangled your data, you have a chance to see that. Did you change a field you didn't want to or accidentally delete half your data? You'll probably spot that. By contrast, working through a database access program can literally make your accidents invisible since you often don't see the post-change or post-addition records.

The two aspects combined are great for avoiding accidents and fixing them when they happen during modifications (provided that you notice at the time). This goes especially well with version control, not just because version control lets you revert things but also because it can explicitly show you what you changed. Databases, well, databases don't do diff very much without a lot of outside assistance.

Why I don't use relational databases in practice

I started out planning to write an entry I was going to title 'when I reach for a relational database'. The problem with this implicit question is that in practice the answer is 'when the system I'm building on basically insists on it'. Otherwise I basically don't use one, even when the problem more or less naturally fits into a relational world. On a purely logical basis, this is crazy; using a relational database for relational things is much easier than the alternatives and SQLite makes this trivial.

In thinking about it, I've come around to the idea that my problem with relational databases is that using a database makes my data relatively opaque and as a result complicates my life. The easy option for a relational database is an SQL based database, probably SQLite, and all of those store my data in unreadable and uneditable binary files. In order to get at my data I need to use the database server's tools (or write code). The direct consequence of this is that I can't inspect and go through my data by firing up less or grep or any number of other things and I can't (if I want to) put things under version control in any easy way.

The slightly more indirect consequence is that I need to write more tools. When my data is in plain text files I can modify my data with an editor or other simple Unix programs; add something, delete something, modify something, shuffle things around, it's all something that gets done with existing Unix programs and maybe some thin scripts. When my data is in a SQLite database I have to manipulate the data through SQLite in some way, either in more complicated shell scripts that drive the SQLite shell or through actual little utility programs that I write. In practice the latter is much safer and so what I'm likely to wind up with. Either way it's more work and more dedicated tools that I have to keep track of in various ways.

For me all of this adds friction to using a real relational database, enough friction that I'll endure some level of pain from hacking things together with plain text files instead. This is not completely rational, but evidently I put a high premium on being able to see things with less and edit them with vi.

(Note that some people have violent objections to relational databases, either in general or for uses where they aren't 'needed' in some sense. I'm not one of them; I don't have any strong feelings that they're inelegant or consume too many resources or whatever.)