Why Prometheus exporters really need fixed TCP ports
It started with this Fediverse post from Scott Laird:
Prometheus maintains a list of port numbers for exporters, so exporters don't conflict with each other. They allocated the range 9100-9999 when the project started. 900 exporters should be enough for everyone?
Yeah, that range is now 100% allocated.
This sparked a discussion that touched on having exporters randomly pick their TCP port when they start and being set up in Prometheus through some service discovery mechanism, including low-tech ones where some external program updated files that Prometheus was reading, as if you'd manually edited them. Reading this sparked a realization for me that you almost certainly don't want to do this.
One of the default labels that Prometheus adds to all metrics scraped
from exporters is the '
instance' label (see Jobs and Instances), which tells
you what specific instance of an exporter things come from. This
label normally includes the instance's port number. There are good
operational reasons to have this label and to include the port
number; for example, it lets you trace back odd things to their
real, assured source instead of having to guess.
However, in Prometheus two metrics are only part of the same time
series if they have the same label set, ie the same labels with the
same values. If even one label changes its value, you have two
different time series, and then various Prometheus functions and
features won't work out the way you want. So when your exporter
restarts, picks a new random port, and your service discovery updates
Prometheus, the '
instance' label will change from (say) '184.108.40.206:5670'
to '220.127.116.11:8670' and all of the metrics being pulled from that
exporter are now a new set of time series that are not contiguous
with the old, pre-restart time series from it. Speaking from personal
experience of doing this to myself at a much smaller and less
frequent scale (cf), you probably
don't want that.
(You also have potential label cardinality issues, although you hopefully aren't restarting exporters all that often.)
You don't technically need your exporters to have a constant port (although that will make your life easier). But you definitely do want every instance of an exporter to pick one port when it's started up for the first time and then stick with it for the lifetime of that host (or container or whatever).