It's a bit risky to give people access to your Prometheus Blackbox exporter

May 28, 2022

A lot of the time you run the Prometheus Blackbox exporter on the same machine as Prometheus itself, so you can make it listen only on 'localhost' and block outside people connecting to it. But suppose that you need to run Blackbox on a different machine, for example to give Prometheus visibility into a network segment the Prometheus host can't normally reach. How potentially risky is it if you leave Blackbox accessible to everyone on the network? Unfortunately, the answer is that it's potentially somewhat risky to do this.

The most obvious and straightforward risk is that Blackbox lets people use any configured module to probe any host, either by host name or by IP address. Some probes are relatively harmless (such as ICMP pings) or relatively constrained (Blackbox DNS probes have to be configured in Blackbox itself with the specific DNS query to make). However, a typically configured Blackbox will have modules that allow people to make TCP connections or HTTP (or HTTPS) GET requests to arbitrary hosts, ports, and URLs. This may include both internal and external hosts, depending on what the machine Blackbox is on can reach.

(These modules are in the standard configuration file and anyway you may need them for the checks you need this Blackbox instance to perform.)

Potential attackers don't need to guess what you've called your Blackbox modules and what they do, either; Blackbox exposes its full configuration file to anyone who can talk to it. This means by extension that any sensitive information in your Blackbox configuration file is visible to everyone who can talk to Blackbox. Blackbox also exposes a log (with details) of recent probes it has done, including the module, the target, and the debug log from the probe.

Both debug logs and the returned metrics include a lot of useful but potentially dangerous details. For example, the returned metrics will tell you if Blackbox was able to resolve a hostname, and the debug log will tell you what IP address it was resolved to. You can get a debug log from any probe that you make manually by adding a 'debug=true' query parameter, saving you from having to look at the Blackbox page and pick your query out from all the rest. This means that access to Blackbox can be used to determine what hostnames resolve and what IP addresses they resolve to.

For DNS probes, often the debug logs contain the response from the DNS server (even if it's not what the module expects). For TCP port probes that expect some specific response, the debug logs will contain one or perhaps more lines that were received from the remote server. Fortunately, the debug logs for HTTP probes don't (currently) contain any text from the reply (although they do have the status code), even for modules where you're checking that the response contains something in specific.

Blackbox currently doesn't contain any built in IP access restrictions, or any feature to restrict access to subsets of its functionality; if you can talk to Blackbox for probes, you can talk to it for everything. Blackbox does support guarding access with HTTP Basic Authentication, although this may make things like debugging problems with its setup harder. As far as I know, you cannot exempt some IPs (like 'localhost') from HTTP Basic Authentication, and the documentation specifically says that if you configure authentication it applies to all of the URLs it supports, including its metrics endpoint.

PS: All of the features that Blackbox exposes are perfectly sensible in a trusted environment. Having detailed information in the debug logs for a specific probe attempt can be quite useful, for example.

Written on 28 May 2022.
« Modal dialogs and other things that steal keyboard focus are dangerous
My wish for per-port IP access controls in systemd .service units »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sat May 28 21:26:40 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.