== Some notes on Prometheus's Blackbox exporter To make a long story short, I'm currently enthusiastically experimenting with [[Prometheus https://prometheus.io/]]. As part of this I'm trying out Prometheus's support for 'black box' status and health checks, where you test services and so on from the outside (instead of the 'white box' approach of extracting health metrics from them directly). The Prometheus people don't seem to be too enthusiastic about black box metrics, so it's perhaps not surprising that [[the official Prometheus blackbox exporter https://github.com/prometheus/blackbox_exporter]] is somewhat underdocumented and hard to understand. The three important components in setting up blackbox metrics are *targets*, *modules*, and *probers*. A *prober* is the low level mechanism for making a check, such as making a HTTP request or a TCP connection; the very limited set of probers is built into the code of the blackbox exporter (and Prometheus is probably unenthused about adding more). A *module* specifies a collection of parameters for a specific *prober* that are used together to check a *target*. More than one module may use the same *prober*, presumably with different parameters. Modules are specified in the blackbox exporter's configuration file. Finally, a *target* is whatever you are checking with a *module* and its *prober*, and it comes from your Prometheus configuration. The names of *probers* are set because they are built into the code of the blackbox exporter. The names of *modules* are arbitrary; you may call them whatever you want and find convenient. Although the official examples give *modules* names that are related to their *prober* (such as ``http_2xx'' and ``imap_starttls''), this doesn't matter and doesn't influence the *prober*'s behavior, such as what port the TCP prober connects to. This was quite puzzling to me for a long time because it was far from obvious where the TCP *prober* got the port to connect to from (and it isn't documented). When Prometheus makes a blackbox check, the blackbox exporter is passed the *module* and the *target* in the URL of the request: > !http://localhost:9115/probe?target=~~TARGET~~&module=~~MODULE~~ The *target* is the only per-target parameter that is passed in to the blackbox exporter, so everything that the *prober* allows you to vary or specify on a per-target basis is encoded into it (and all other *prober* configuration comes from the specific *module* you use). How things are encoded in the *target* and what you can put there depends on the specific *prober*. For the _icmp_ prober, the *target* is a host name or an IP address. No further per-target things can be provided, and the *module* parameters are sort of minimal too. For the _dns_ prober, the *target* is a host name or IP address of the DNS server to query, plus the port, formatted as 'host:port' (and so normally 'host:53'). ~~What DNS query to make is set in the module's parameters~~, as is what reply to expect, whether to use TCP or UDP, and so on. There is no way to specify these as part of the target, so if you want to query different DNS names, you need different *modules*. This is not particularly scalable if you want to query the same DNS server for several names, but then I suspect that the Prometheus people would tell you to write a script for that sort of thing. (It turns out that if you leave off ':port', it defaults to 53, but I had to read the code to find this out.) For the _http_ prober, the *target* is the full URL to be requested (or, as an undocumented feature, you can leave off the '!http://' at the start of the URL). What to require from the result is configured through the *module*'s parameters, as is various aspects of TLS. As with DNS probes, if you want to check that some URLs return a 2xx status and some URLs redirect, you will need two separate modules. The _http_ prober automatically choses HTTP or HTTPS based on the scheme of the *target*, as you'd expect, which is why a single _http_ prober based *module* can be used for URLs from either scheme. Under normal circumstances, using HTTPS URLs automatically verifies the certificate chain through whatever system certificate store Go is using on your machine. For the _tcp_ prober, the *target* is the host:port to connect to. As we should expect by now, everything else is configured through the *module*'s parameters, including everything to do with TLS; this means that unlike HTTP, you need different *modules* for checking non-TLS connections and TLS connections. The _tcp_ prober lets the module control whether or not to do TLS on connection (normally with server certificate verification), and you can set up a little chat dialog to test the service you're connecting to (complete with switching to TLS at some suitable point in the dialog). Contrary to the documentation, the _expect:_ strings in the chat dialog are regular expressions, not plain strings. Much of Prometheus's official blackbox checking would be more flexible if you could pass optional additional parameters outside of the *target*; the obvious case is DNS checks. === Sidebar: Understanding blackbox relabeling The Prometheus configuration examples for the blackbox exporter contain a great deal of magic use of ((relabel_configs)). Perhaps what it is doing and what is required is obvious to experienced Prometheus people, but in any case I am not one right now. The standard example from [[the README https://github.com/prometheus/blackbox_exporter]] is: > metrics_path: /probe > params: > module: [AMODULE] > static_configs: > - targets: > - ATARGET > relabel_configs: > - source_labels: [__address__] > target_label: __param_target > - source_labels: [__param_target] > target_label: instance > - target_label: __address__ > replacement: 127.0.0.1:9115 The ((__address__)) label starts out being set to ATARGET, our blackbox exporter *target*, because that is what we told Prometheus; if we were using an ordinary exporter, this would be the host and port that Prometheus was going to scrape. Since the blackbox exporter is special, we must instead turn it into the ((target=)) parameter of the URL we will scrape, which we do by relabeling it to ((__param_target)). We also save it into the ((instance)) label, which will propagate through to the final metrics that come from this (so that we can later find things by our *target*s). Finally, we set the ((__address__)) label to the actual host:port to scrape from, because if we didn't Prometheus wouldn't even talk to the blackbox exporter. We also need a ((module=)) URL parameter. Here, that comes from the _module:_ in the _params_ section; during relabeling it is ((__param_module)), and it will be set to AMODULE.