2018-10-11
Some notes on Prometheus's Blackbox exporter
To make a long story short, I'm currently enthusiastically experimenting with Prometheus. As part of this I'm trying out Prometheus's support for 'black box' status and health checks, where you test services and so on from the outside (instead of the 'white box' approach of extracting health metrics from them directly). The Prometheus people don't seem to be too enthusiastic about black box metrics, so it's perhaps not surprising that the official Prometheus blackbox exporter is somewhat underdocumented and hard to understand.
The three important components in setting up blackbox metrics are targets, modules, and probers. A prober is the low level mechanism for making a check, such as making a HTTP request or a TCP connection; the very limited set of probers is built into the code of the blackbox exporter (and Prometheus is probably unenthused about adding more). A module specifies a collection of parameters for a specific prober that are used together to check a target. More than one module may use the same prober, presumably with different parameters. Modules are specified in the blackbox exporter's configuration file. Finally, a target is whatever you are checking with a module and its prober, and it comes from your Prometheus configuration.
The names of probers are set because they are built into the code of the blackbox exporter. The names of modules are arbitrary; you may call them whatever you want and find convenient. Although the official examples give modules names that are related to their prober (such as http_2xx and imap_starttls), this doesn't matter and doesn't influence the prober's behavior, such as what port the TCP prober connects to. This was quite puzzling to me for a long time because it was far from obvious where the TCP prober got the port to connect to from (and it isn't documented).
When Prometheus makes a blackbox check, the blackbox exporter is passed the module and the target in the URL of the request:
http://localhost:9115/probe?target=TARGET&module=MODULE
The target is the only per-target parameter that is passed in to the blackbox exporter, so everything that the prober allows you to vary or specify on a per-target basis is encoded into it (and all other prober configuration comes from the specific module you use). How things are encoded in the target and what you can put there depends on the specific prober.
For the icmp
prober, the target is a host name or an IP address.
No further per-target things can be provided, and the module
parameters are sort of minimal too.
For the dns
prober, the target is a host name or IP address of
the DNS server to query, plus the port, formatted as 'host:port'
(and so normally 'host:53'). What DNS query to make is set in the
module's parameters, as is what reply to expect, whether to use
TCP or UDP, and so on. There is no way to specify these as part of
the target, so if you want to query different DNS names, you need
different modules. This is not particularly scalable if you want
to query the same DNS server for several names, but then I suspect
that the Prometheus people would tell you to write a script for
that sort of thing.
(It turns out that if you leave off ':port', it defaults to 53, but I had to read the code to find this out.)
For the http
prober, the target is the full URL to be requested
(or, as an undocumented feature, you can leave off the 'http://'
at the start of the URL). What to require from the result is
configured through the module's parameters, as is various aspects
of TLS. As with DNS probes, if you want to check that some URLs
return a 2xx status and some URLs redirect, you will need two
separate modules. The http
prober automatically choses HTTP or
HTTPS based on the scheme of the target, as you'd expect, which
is why a single http
prober based module can be used for URLs
from either scheme. Under normal circumstances, using HTTPS URLs
automatically verifies the certificate chain through whatever system
certificate store Go is using on your machine.
For the tcp
prober, the target is the host:port to connect to.
As we should expect by now, everything else is configured through
the module's parameters, including everything to do with TLS;
this means that unlike HTTP, you need different modules for
checking non-TLS connections and TLS connections. The tcp
prober
lets the module control whether or not to do TLS on connection
(normally with server certificate verification), and you can set
up a little chat dialog to test the service you're connecting to
(complete with switching to TLS at some suitable point in the
dialog). Contrary to the documentation, the expect:
strings in
the chat dialog are regular expressions, not plain strings.
Much of Prometheus's official blackbox checking would be more flexible if you could pass optional additional parameters outside of the target; the obvious case is DNS checks.
Sidebar: Understanding blackbox relabeling
The Prometheus configuration examples for the blackbox exporter
contain a great deal of magic use of relabel_configs
. Perhaps
what it is doing and what is required is obvious to experienced
Prometheus people, but in any case I am not one right now.
The standard example from the README is:
metrics_path: /probe params: module: [AMODULE] static_configs: - targets: - ATARGET relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: 127.0.0.1:9115
The __address__
label starts out being set to ATARGET, our
blackbox exporter target, because that is what we told Prometheus;
if we were using an ordinary exporter, this would be the host and
port that Prometheus was going to scrape. Since the blackbox exporter
is special, we must instead turn it into the target=
parameter
of the URL we will scrape, which we do by relabeling it to
__param_target
. We also save it into the instance
label,
which will propagate through to the final metrics that come from
this (so that we can later find things by our targets). Finally,
we set the __address__
label to the actual host:port to scrape
from, because if we didn't Prometheus wouldn't even talk to the
blackbox exporter.
We also need a module=
URL parameter. Here, that comes from the
module:
in the params
section; during relabeling it is
__param_module
, and it will be set to AMODULE.