Wandering Thoughts archives

Some notes on Go's expvar package

I recently decided to add accessible statistics to my sinkhole SMTP server by using the expvar package. In the process of experimenting with this, I've wound up with a collection of notes that I want to write down while I remember them.

First up is my discovery about expvar.Var, namely that it wants you to return properly quoted JSON instead of just a string. You can read the entry for more details.

The expvar package exposes a lot of 'New<whatever>' functions to give you new variables of various types. The important thing to bear in mind about these is that they bundle together both the actual variable and its expvar name. This means that they will always be separate top level JSON objects; you can't put them inside an application-wide or package-wide JSON dictionary or the like.

(This is the difference between having a 'memstats' JSON dictionary with elements like 'Alloc' and 'BySize', and having separate 'memstats-Alloc', 'memstats-BySize', and so on JSON things.)

In my view, you generally want to namespace all your variables to make them easier to deal with. To do namespacing, you don't use 'New<whatever>', except for one top level map (that will become a JSON dictionary); instead you create instances of whatever you need and then manually embed them in your namespace map. Here, have an example:

var counter1, counter2 expvar.Int
var status expvar.String

m := expvar.NewMap("myapp")
m.Set("counter1", &counter1)
m.Set("counter2", &counter2)
m.Set("status", &status)

Most expvar types can be used this way without explicit initialization. The exception is expvar.Map, which requires you to call its .Init() before you do anything else with it:

var submap expvar.Map
submap.Init()
submap.Set("somevar", &somevar)
submap.Set("somevar2", &somevar2)
m.Set("subthing", &submap)

The other way of setting up variables or in fact entire hierarchies of information is to bulk-export them through a function by using expvar.Func. Often such a function will be handing over an existing structure that you maintain. For example:

type iMap struct {
   sync.Mutex
   ips   map[string]int
   stats struct {
      Size, Adds, Lookups, Dels int
   }
}

func (i *iMap) Stats() interface{} {
   i.Lock()
   defer i.Unlock()
   i.stats.Size = len(i.ips)
   return i.stats
}

var tlserrs iMap

Then, to create this as a variable in your application's namespace map:

m.Set("tlserrors", expvar.Func(tlserrs.Stats))

One important note here is that the conversion to JSON is done using json.Marshal and is subject to its rules about field visibility. I believe that this means all fields need to be exported, as they are in this example (or at least all fields that you want to be visible as expvar variables). This is not a requirement for things that you register explicitly or set up through expvar.New<whatever>; as in my examples above, they can be non-exported variables and usually are.

(This is how the standard expvar cmdline and meminfo variables are implemented.)

Sidebar: Locking and statistics

My first version of the code I wrote here had a subtle bug. The original version before I added statistics sensibly used sync.RWMutex and took only a read lock during lookups; a write lock was only necessary when adding or deleting entries (a rarer activity). But when I added statistics variables using non-atomic operations, lookups were suddenly doing writes and so now actually needed to be protected with a write lock. Which meant everything needed a write lock, which is why the code here uses a plain sync.Mutex instead.

For subtle reasons beyond the scope of this sidebar, I don't think you can get around all concurrency races here by using expvar types (which are all normally safe against multiple updates); as far as I can see, the .Stats() function itself has races.

(Note that in general, global statistics are always a contention point if you have high concurrency.)

A surprise to watch out for with Go's expvar package (in expvar.Var)

The standard expvar package is a handy thing for easily exposing counters, values, and so on in a way that can be queried from outside your running program. As you might expect, it ultimately works through an interface type, expvar.Var. This interface is very simple:

type Var interface {
    String() string
}

If you see this definition, your eyes may light up with familiarity (as mine did), because this is exactly the extremely standard fmt.Stringer interface, where everything that has a .String() method can be handled by a lot of things. So of course you might well write code like this:

m := expvar.NewMap("myapp")
// the time.Time type has a String()
// method, so this will totally work.
m.Set("startTime", time.Now())

If you do this, everything will work right up to the point where programs that parse the JSON returned by the /debug/vars endpoint start failing with weird errors. If you look at the raw JSON, what you will see is something like this:

[...], "startTime": 2016-04-01 17:49:09.385829528 -0400 EDT, "anothervar": "something", [...]

In case you don't see the problem (as I didn't for some time), the string value for "startTime" doesn't have quotes around it, which makes it very invalid JSON. Go's current JSON parser starts trying to interpret the starting '2016-' bit as a number, then runs into the '-' and complains about it.

What is happening is that the expvar.Var String() interface method is misnamed; it should really be called something like JSON(). What the Var.String() method is actually required to do is produce the JSON representation of the Var as a string; for strings, this requires them to be quoted. A normal Stringer .String() method doesn't do this quoting, of course, because it would get in the way. The two interpretations of .String() are not really compatible, but there is no way to tell them apart and Go's implicitly satisfied interfaces will let you substitute one for the other (as I did when I tried to use time.Time as a Var).

So the takeaway here is that just because something has a .String() doesn't mean you can use it as an expvar.Var; in fact, you probably can't. Anything that's designed to be used as an expvar.Var will specifically say so in its documentation (or at least it should). Anything that has a .String() but doesn't mention expvar should be assumed to be satisfying the far more common fmt.Stringer interface instead.

(I don't have any clever solution for this. And I think the Go 1 API compatibility guarantee will keep this from changing, as expvar.Var was in its current form in the initial Go 1 release.)