A note on using the Go Prometheus client package to exposed labeled metrics
For reasons beyond the scope of this blog entry, I have been recently playing around adding some Prometheus metrics to a Go program through the official Prometheus Go client packages. The particular metrics I wanted to add were going to have labels:
scripts_requests_total { script="ntpdate" } 58 scripts_requests_total { script="sntp" } 112
How to do this is not entirely clear in the Prometheus client package
documentation. You start by creating, say, a CounterVec
, then
some magic happens involving methods with names like With
and
CurryWith
, and you finally have metrics that you can set, increment,
and so on. After some head scratching I have finally figured out a
mental model of this, and now I'm going to write it down before I
forget it.
When you create a metric with labels by using, say, NewCounterVec()
,
what you really create is a template for the actual metrics you
will wind up with. Your template specifies what the names of the labels
are, but obviously it doesn't (and can't) specify what values the
labels have for any particular metric. In order to actually get a
specific metric, you must fill in values for all of the labels in
some way, which creates a specific metric from your template. With
a specific metric in hand, you can now manipulate it to, for example,
count things. If you're working with metrics that have labels, you
always have to perform this specialization step, even if you're
only ever going to generate a single metric (for example, a
'scripts_build_info
' metric where the point is the label values).
The Go package offers two ways of doing this specialization. First,
you can create a Labels
map that maps all of the label names to specific values, and then
use .With()
or .GetMetricWith()
. Second, you can simply use
.WithLabelValues()
or .GetWithLabelValues()
to list off all of
the label values in the same order as you specified the labels
themselves when you created the *Vec
metric template. Which one you
use depends on which is more convenient.
These metrics templates can also be partially specified, filling
in some but not all label values. This is done with the .CurryWith()
and .MustCurryWith()
methods, which return a 'narrower' *Vec
metric template. I can imagine several situations where this might
be useful, but since I haven't used this in code yet I'm not going
to write out my speculations here.
(I suspect that Prometheus client packages for other languages follow a similar model, but I haven't looked at them yet.)
Sidebar: An unfortunate limitation of promhttp
The Prometheus Go client packages include promhttp
, which can
generate metrics related to the HTTP requests that your program
handles (and also to any HTTP requests it makes, with a separate
set of functions). Unfortunately the instrumentation it provides
doesn't have any way to customize the labels of the metrics on a
per-request basis, and re-implementing what it's doing requires
duplicating a bunch of un-exported HTTP middleware functionality
that it contains.
For a non-hypothetical example, promhttp goes through a large amount of work to be able to capture the HTTP reply's status code and other response information. When I started looking through that code, I decided that the HTTP status wasn't quite important enough to put in my own metrics.
(The problem here is that a concrete instance of the
http.ResponseWriter
may support additional interfaces, like http.Flusher
or http.CloseNotifier
, and this support
may be important to either your HTTP server code or things that
your code calls. It's easy to implement a plain http.ResponseWriter
in middleware, but then downstream code loses access to these
additional interfaces.)
|
|