I recently decided to add accessible statistics to my sinkhole SMTP server by using the expvar package. In the process of experimenting with this, I've wound up with a collection of notes that I want to write down while I remember them.

First up is my discovery about expvar.Var, namely that it wants you to return properly quoted JSON instead of just a string. You can read the entry for more details.

The expvar package exposes a lot of 'New<whatever>' functions to give you new variables of various types. The important thing to bear in mind about these is that they bundle together both the actual variable and its expvar name. This means that they will always be separate top level JSON objects; you can't put them inside an application-wide or package-wide JSON dictionary or the like.

(This is the difference between having a 'memstats' JSON dictionary with elements like 'Alloc' and 'BySize', and having separate 'memstats-Alloc', 'memstats-BySize', and so on JSON things.)

In my view, you generally want to namespace all your variables to make them easier to deal with. To do namespacing, you don't use 'New<whatever>', except for one top level map (that will become a JSON dictionary); instead you create instances of whatever you need and then manually embed them in your namespace map. Here, have an example:

var counter1, counter2 expvar.Int
var status expvar.String

m := expvar.NewMap("myapp")
m.Set("counter1", &counter1)
m.Set("counter2", &counter2)
m.Set("status", &status)

Most expvar types can be used this way without explicit initialization. The exception is expvar.Map, which requires you to call its .Init() before you do anything else with it:

var submap expvar.Map
submap.Init()
submap.Set("somevar", &somevar)
submap.Set("somevar2", &somevar2)
m.Set("subthing", &submap)

The other way of setting up variables or in fact entire hierarchies of information is to bulk-export them through a function by using expvar.Func. Often such a function will be handing over an existing structure that you maintain. For example:

type iMap struct {
   sync.Mutex
   ips   map[string]int
   stats struct {
      Size, Adds, Lookups, Dels int
   }
}

func (i *iMap) Stats() interface{} {
   i.Lock()
   defer i.Unlock()
   i.stats.Size = len(i.ips)
   return i.stats
}

var tlserrs iMap

Then, to create this as a variable in your application's namespace map:

m.Set("tlserrors", expvar.Func(tlserrs.Stats))

One important note here is that the conversion to JSON is done using json.Marshal and is subject to its rules about field visibility. I believe that this means all fields need to be exported, as they are in this example (or at least all fields that you want to be visible as expvar variables). This is not a requirement for things that you register explicitly or set up through expvar.New<whatever>; as in my examples above, they can be non-exported variables and usually are.

(This is how the standard expvar cmdline and meminfo variables are implemented.)

Sidebar: Locking and statistics

My first version of the code I wrote here had a subtle bug. The original version before I added statistics sensibly used sync.RWMutex and took only a read lock during lookups; a write lock was only necessary when adding or deleting entries (a rarer activity). But when I added statistics variables using non-atomic operations, lookups were suddenly doing writes and so now actually needed to be protected with a write lock. Which meant everything needed a write lock, which is why the code here uses a plain sync.Mutex instead.

For subtle reasons beyond the scope of this sidebar, I don't think you can get around all concurrency races here by using expvar types (which are all normally safe against multiple updates); as far as I can see, the .Stats() function itself has races.

(Note that in general, global statistics are always a contention point if you have high concurrency.)