Wandering Thoughts archives

Why Prometheus exporters really need fixed TCP ports

It started with this Fediverse post from Scott Laird:

Prometheus maintains a list of port numbers for exporters, so exporters don't conflict with each other. They allocated the range 9100-9999 when the project started. 900 exporters should be enough for everyone?

Yeah, that range is now 100% allocated.

This sparked a discussion that touched on having exporters randomly pick their TCP port when they start and being set up in Prometheus through some service discovery mechanism, including low-tech ones where some external program updated files that Prometheus was reading, as if you'd manually edited them. Reading this sparked a realization for me that you almost certainly don't want to do this.

One of the default labels that Prometheus adds to all metrics scraped from exporters is the 'instance' label (see Jobs and Instances), which tells you what specific instance of an exporter things come from. This label normally includes the instance's port number. There are good operational reasons to have this label and to include the port number; for example, it lets you trace back odd things to their real, assured source instead of having to guess.

However, in Prometheus two metrics are only part of the same time series if they have the same label set, ie the same labels with the same values. If even one label changes its value, you have two different time series, and then various Prometheus functions and features won't work out the way you want. So when your exporter restarts, picks a new random port, and your service discovery updates Prometheus, the 'instance' label will change from (say) '1.2.3.4:5670' to '1.2.3.4:8670' and all of the metrics being pulled from that exporter are now a new set of time series that are not contiguous with the old, pre-restart time series from it. Speaking from personal experience of doing this to myself at a much smaller and less frequent scale (cf), you probably don't want that.

(You also have potential label cardinality issues, although you hopefully aren't restarting exporters all that often.)

You don't technically need your exporters to have a constant port (although that will make your life easier). But you definitely do want every instance of an exporter to pick one port when it's started up for the first time and then stick with it for the lifetime of that host (or container or whatever).