Wandering Thoughts archives

An easy optimization for restricted multi-metric queries in Prometheus

Here's something that I've repeatedly had to learn and remember the hard way when I was building Prometheus queries for our dashboards, especially for graphs. Suppose that you have a PromQL query that involves multiple metrics, for instance you want to know the number of inodes used on a filesystem:

node_filesystem_files - node_filesystem_files_free

(The reason that the Prometheus host agent doesn't directly export a 'number of inodes used' metric is that statfs(2) doesn't provide that information on common Unixes; it provides only 'total inodes' and 'inodes free'. The host agent is being honest here. I could say a lot of things about statfs(2), but this is not the entry for it.)

Now, suppose that you have a lot of servers and a lot of filesystems and that you're actually only interested in the number of inodes used for a few of them. For example, you have a Grafana dashboard that displays the information for the root filesystem on a single host. A perfectly sensible way to write the query for this dashboard is:

node_filesystem_files{ host="$host", mountpoint="/" } - node_filesystem_files_free

Unfortunately, current versions of Prometheus (2.8.0 as I write this entry) miss the obvious way of optimizing this query when they execute it. Instead of propagating the label restrictions from the left hand side query to the right hand side as well, the PromQL engine will get all of the metrics for node_filesystem_files_free, across all of your servers and filesystems, and then throw out all but the single one that matches the left hand side.

As a result, any time you have a multi-metric query that is matching labels across the metrics and one or more of the metrics is restricted with label matches, you can usually improve things by replicating the restriction into the other metrics. This goes for arithmetic operators, boolean operators, and 'and', but obviously doesn't apply for 'or' or 'unless'. This improvement doesn't just boost performance; under some circumstances, it can make the difference between a query that gets a Prometheus error about loading too many metrics points and a query that works.

(I find this a bit unfortunate, in that the natural and more maintainable way to write the query is not always the workable way. The performance impact of the less efficient version I can usually live with, but I really don't want my graphs and queries falling over with 'too many metrics points' when I extend the time ranges far enough.)