Wandering Thoughts archives

A note on using the Go Prometheus client package to exposed labeled metrics

For reasons beyond the scope of this blog entry, I have been recently playing around adding some Prometheus metrics to a Go program through the official Prometheus Go client packages. The particular metrics I wanted to add were going to have labels:

scripts_requests_total { script="ntpdate" } 58
scripts_requests_total { script="sntp" }    112

How to do this is not entirely clear in the Prometheus client package documentation. You start by creating, say, a CounterVec, then some magic happens involving methods with names like With and CurryWith, and you finally have metrics that you can set, increment, and so on. After some head scratching I have finally figured out a mental model of this, and now I'm going to write it down before I forget it.

When you create a metric with labels by using, say, NewCounterVec(), what you really create is a template for the actual metrics you will wind up with. Your template specifies what the names of the labels are, but obviously it doesn't (and can't) specify what values the labels have for any particular metric. In order to actually get a specific metric, you must fill in values for all of the labels in some way, which creates a specific metric from your template. With a specific metric in hand, you can now manipulate it to, for example, count things. If you're working with metrics that have labels, you always have to perform this specialization step, even if you're only ever going to generate a single metric (for example, a 'scripts_build_info' metric where the point is the label values).

The Go package offers two ways of doing this specialization. First, you can create a Labels map that maps all of the label names to specific values, and then use .With() or .GetMetricWith(). Second, you can simply use .WithLabelValues() or .GetWithLabelValues() to list off all of the label values in the same order as you specified the labels themselves when you created the *Vec metric template. Which one you use depends on which is more convenient.

These metrics templates can also be partially specified, filling in some but not all label values. This is done with the .CurryWith() and .MustCurryWith() methods, which return a 'narrower' *Vec metric template. I can imagine several situations where this might be useful, but since I haven't used this in code yet I'm not going to write out my speculations here.

(I suspect that Prometheus client packages for other languages follow a similar model, but I haven't looked at them yet.)

Sidebar: An unfortunate limitation of promhttp

The Prometheus Go client packages include promhttp, which can generate metrics related to the HTTP requests that your program handles (and also to any HTTP requests it makes, with a separate set of functions). Unfortunately the instrumentation it provides doesn't have any way to customize the labels of the metrics on a per-request basis, and re-implementing what it's doing requires duplicating a bunch of un-exported HTTP middleware functionality that it contains.

For a non-hypothetical example, promhttp goes through a large amount of work to be able to capture the HTTP reply's status code and other response information. When I started looking through that code, I decided that the HTTP status wasn't quite important enough to put in my own metrics.

(The problem here is that a concrete instance of the http.ResponseWriter may support additional interfaces, like http.Flusher or http.CloseNotifier, and this support may be important to either your HTTP server code or things that your code calls. It's easy to implement a plain http.ResponseWriter in middleware, but then downstream code loses access to these additional interfaces.)

What I want out of my window manager

One answer to what I want out of my window manager is 'fvwm'. It's my current window manager and I'm not likely to switch to anything else because I'm perfectly satisfied with it. But that's not a good answer, because fvwm has a lot of features and I'm not using them all. As with everyone who uses a highly customizable thing, my important subset of fvwm is probably not quite the same as anyone else's important subset of it.

(I'm thinking about what I want out of my window manager because Wayland is coming someday, and that means I'm almost certainly going to need a new window manager at some time in, say, the next ten years.)

I can't tell for sure what's important to me, because I'm sort of a fish in water when it comes to fvwm and my fvwm configuration; I've been using it exclusively for so long that I'm not certain what I'd really miss if I moved and what's unusual. With that said, I think that the (somewhat) unusual features that I want go like this (on top of a straightforward 'floating layout' window manager):

• Something like FvwmIconMan, which is central to how I manage terminal windows (which I tend to have a lot of).

• The ability to iconify windows to icons on the root window and then place those icons in specific locations where they'll stay. I also want to be able to record the location of those icons and reposition them back, because I do that. Putting iconified windows in specific places is how I currently manage my plethora of Firefox windows, including keeping track of what I'm going to read soon. As usual, icons need to have both an icon and a little title string.

  (Perhaps I should figure out a better way to handle Firefox windows, one that involves less clutter. I have some thoughts there, although that's for another entry. But even with Firefox handled, there are various other windows I keep around in iconified form.)

• Multiple virtual desktops or screens, with some sort of pager to show me a schematic view of what is on what screen or desktop and to let me switch between them by the mouse. I also need key bindings to flip around between screens. It has to be possible to easily move windows (in normal or iconified form) from screen to screen, including from the command line, and I should be able to set it so that some icons or windows are always present (ie, they float from screen to screen).

• Window title bars that can be either present or absent, because some of my windows have them and some don't. I'd like the ability to customize what buttons a window titlebar has and what they do, but it's not really important; I could live with everything with a titlebar having a standard set.

• User-defined menus that can be brought up with a wide variety of keys, because I have a lot of menus that are bound to a lot of different keys. My fvwm menus are one of my two major ways of launching programs, and I count on having a lot of different key bindings to make them accessible without having to go through multiple menu levels.

• User-defined key bindings, including key bindings that still work when the keyboard focus is on a window. Key bindings need to be able to invoke both window manager functions (like raising and lowering windows) and to run user programs, especially dmenu.
• User-defined bindings for mouse buttons, because I use a bunch of them.

• Minimal or no clutter apart from things that I specifically want. I don't want the window manager insisting that certain interface elements must exist, such as a taskbar.

• What fvwm calls 'focus follows mouse', where the keyboard focus is on the last window the mouse was in even if the mouse is then moved out to be over the root window. I don't want click to focus for various reasons and I now find strict mouse focus to be too limiting.

Fvwm allows me great power over customizing the fonts used, the exact width of window borders, and so on, but for the most part it's not something I care deeply about if the window manager does a competent job and makes good choices in general. It's convenient if the window manager has a command interface for testing and applying small configuration changes, like FvwmConsole; restarting the window manager when you have a lot of windows is kind of a pain.

(As you might guess from my priorities, my fvwm configuration file is almost entirely menu configurations, key and mouse button bindings, and making FvwmIconMan and fvwm's pager work right. I have in the past tried tricky things, but at this point I'm no longer really using any of them. All of my vaguely recent changes have been around keyboard bindings for things like moving windows and changing sound volume.)

PS: Command line interface and control of the window manager would be pretty handy. I may not use FvwmCommand very often, but I like that it's there. And I do use the Perl API for my hack.

Sidebar: Fvwm's virtual screens versus virtual desktops

Fvwm has both virtual screens and virtual desktops, and draws a distinction between them that is covered in the relevant section of its manpage. I use fvwm's virtual screens but not its desktops, and in practice I treat every virtual screen as a separate thing. It can sometimes be convenient that a window can spill over from virtual screen to virtual screen, since it often gives me a way of grabbing the corner of an extra-large window. On the other hand, it's also irritating when a window winds up protruding into another virtual screen.

All of this is leading up to saying that I wouldn't particularly object to a window manager that had only what fvwm would call virtual desktops, without virtual screens. This is good because I think that most modern window managers have adopted that model for their virtual things.