For reasons beyond the scope of this blog entry, I have been recently playing around adding some Prometheus metrics to a Go program through the official Prometheus Go client packages. The particular metrics I wanted to add were going to have labels:

scripts_requests_total { script="ntpdate" } 58
scripts_requests_total { script="sntp" }    112

How to do this is not entirely clear in the Prometheus client package documentation. You start by creating, say, a CounterVec, then some magic happens involving methods with names like With and CurryWith, and you finally have metrics that you can set, increment, and so on. After some head scratching I have finally figured out a mental model of this, and now I'm going to write it down before I forget it.

When you create a metric with labels by using, say, NewCounterVec(), what you really create is a template for the actual metrics you will wind up with. Your template specifies what the names of the labels are, but obviously it doesn't (and can't) specify what values the labels have for any particular metric. In order to actually get a specific metric, you must fill in values for all of the labels in some way, which creates a specific metric from your template. With a specific metric in hand, you can now manipulate it to, for example, count things. If you're working with metrics that have labels, you always have to perform this specialization step, even if you're only ever going to generate a single metric (for example, a 'scripts_build_info' metric where the point is the label values).

The Go package offers two ways of doing this specialization. First, you can create a Labels map that maps all of the label names to specific values, and then use .With() or .GetMetricWith(). Second, you can simply use .WithLabelValues() or .GetWithLabelValues() to list off all of the label values in the same order as you specified the labels themselves when you created the *Vec metric template. Which one you use depends on which is more convenient.

These metrics templates can also be partially specified, filling in some but not all label values. This is done with the .CurryWith() and .MustCurryWith() methods, which return a 'narrower' *Vec metric template. I can imagine several situations where this might be useful, but since I haven't used this in code yet I'm not going to write out my speculations here.

(I suspect that Prometheus client packages for other languages follow a similar model, but I haven't looked at them yet.)

Sidebar: An unfortunate limitation of promhttp

The Prometheus Go client packages include promhttp, which can generate metrics related to the HTTP requests that your program handles (and also to any HTTP requests it makes, with a separate set of functions). Unfortunately the instrumentation it provides doesn't have any way to customize the labels of the metrics on a per-request basis, and re-implementing what it's doing requires duplicating a bunch of un-exported HTTP middleware functionality that it contains.

For a non-hypothetical example, promhttp goes through a large amount of work to be able to capture the HTTP reply's status code and other response information. When I started looking through that code, I decided that the HTTP status wasn't quite important enough to put in my own metrics.

(The problem here is that a concrete instance of the http.ResponseWriter may support additional interfaces, like http.Flusher or http.CloseNotifier, and this support may be important to either your HTTP server code or things that your code calls. It's easy to implement a plain http.ResponseWriter in middleware, but then downstream code loses access to these additional interfaces.)