Another way to do easy configuration for lots of Prometheus Blackbox checks
Early on in our use of Prometheus, I wrote up a scheme for easy configuration of lots of Blackbox checks where I encoded the name of the Blackbox module to use in the names of the targets you configured, and then extracted them with relabeling. The result gave you target names that looked like:
- ssh_banner,somehost:22 - http_2xx,http://somewhere/url
This encodes the Blackbox module before the comma and the actual Blackbox target after it (you can use any suitable separator; I picked comma for how it looks).
This works, but I've learned that there is another approach that is more natural and perhaps clearer, namely adding explicit additional labels to your targets and then using those labels in relabeling to determine things like the Blackbox module or even the target to check.
Let's start with the basics (since I didn't know this for a while),
which is that a Prometheus '
targets' section of statically
configured targets can have additional labels specified. The
ostensible purpose of this (covered in the documentation)
is to attach additional labels to all metrics scraped from the
- targets: - 18.104.22.168:53 - 22.214.171.124:53 labels: - type: external
(My initial use of this was to explicitly label some of the hosts we check as off-network hosts, because check failure for them is different from failure for our local machines.)
However, as covered in this prometheus-users message from Ben
these additional labels are available at the start of the scrape,
and so you can use relabeling to turn them into things like what
Blackbox module to use. For example, suppose you add a '
ssh_banner' label to a set of targets that you want checked with
that Blackbox module, and then have a relabeling configuration like
# Set the target from the address, # as usual - source_labels: [__address__] target_label: __param_target # Set the Blackbox module from # the 'module:' label - source_labels: [module] target_label: __param_module # And now point the address to a # local Blackbox as usual. - target_label: __address__ replacement: 127.0.0.1:9115
(As a disclaimer, I haven't actually tested this snippet.)
I see advantages and disadvantages to this approach. One advantage
is that it's likely to be more clearer and normal. People are (or
should be) used to attaching extra labels to static targets, and
it's clearly documented, so the only magic and mystery is how your
module label takes effect. While I like my original
syntax, it's clearly more magical and unusual; you're going to have
to read the relabeling configuration to understand what's going on
and how to write additional things.
One drawback is that it pretty much forces you to group checks by module instead of by target. With my scheme, you can list several checks for a host together:
- targets: [...] - ssh_banner,host:22 - smtp_banner,host:25 - http_2xx,http://host/url
With an explicit label-based approach to selecting the module, each
of these has to be in a separate static configuration section because
they each need a different
module label. On the other hand, this
pushes you toward listing all of your checks for a given Blackbox
module in one spot.
A place where this can be an active drawback is if you need to vary
additional labels for groups of targets, especially across modules.
For instance, if you want to attach a '
dc' label to all Blackbox
metrics from a group of hosts, you now need to split up those per
module sections (with a '
module' label) into multiple sections,
one for each combination of
dc. This could easily
get pretty verbose (although it might not matter if you're
automatically generating this from external configuration information).
I probably won't be changing our configuration from my current trick to this more straightforward approach, but I'm going to bear it in mind for future use. Partly this is because our setup already exists and works, and partly it's because we use some additional labels now and I want to preserve our freedom to easily use more in the future.