Remembering that Prometheus expressions act as filters
In conventional languages, comparisons like '
>' and other boolean
operations like '
and' give you implicit or explicit boolean
results. Sometimes this is a pseudo-boolean result; in Python if
you say '
A and B', you famously get either
False or the value
B as the end result (instead of
True). However, PromQL
doesn't work this way. As I keep having to remember over and over,
in Prometheus, comparisons and other boolean operators are
In PromQL, when you write '
some_metric > 10', what happens is
that first Prometheus generates a full instant vector for
some_metric, with all of the metric points and their labels and
their values, and then it filters out any metric point in the instant
vector where the value isn't larger than 10. What you have left is
a smaller instant vector, but all of the values of the metric points
in it are their original ones.
The same thing happens with '
and'. When you write '
and other_metric', the
other_metric is used only as a filter;
metric points from
some_metric are only included in the result
set if there is the same set of labels in the
instant vector. This means that the values of
irrelevant and do not propagate into the result.
The large scale effect of this is that the values that tend to propagate through your rule expression are whatever started out as the first metric you looked at (or whatever arithmetic you perform on them). Sometimes, especially in alert rules, this can bias you toward putting one condition in front of the other. For instance, suppose that you want to trigger an alert when the one-minute load average is above 20 and the five-minute load average is above 5, and you write the alert rule as:
expr: (node_load5 > 5) and (node_load1 > 20)
The value available in the alert rule and your alert messages is
the value of
is what you started out the rule with. If you find the value of
node_load1 more useful in your alert messages, you'll want to
flip the order of these two clauses around.
As the PromQL documentation covers, you can turn comparison operations
from filters into pseudo-booleans by using '
bool', as in
some_metric > bool 10'. As far as I know, there is no way to
do this with '
and', which always functions as a filter, although
you can at least select what labels have to match (or what labels
PS: For some reason I keep forgetting that '
unless' can use '
on' and '
ignoring' to select what labels
you care about. What you can't do with them, though, is propagate
some labels from the right side into the result; if you need that,
you have to use '
group_left' or '
group_right' and figure
out how to re-frame your operation so that it involves a comparison,
and' and company don't work with grouping.
(I was going to confidently write an entry echoing something that I said on the Prometheus users mailing list recently, but when when I checked the documentation and performed some tests, it turned out I was wrong about an important aspect of it. So this entry is rather smaller in scope, and is written mostly to get this straight in my head since I keep forgetting the details of it.)