rate()
versus irate()
in Prometheus (and Grafana)
Prometheus's PromQL
query language has two quite similar functions for calculating the
rate of things (well, of counters), rate()
and irate()
.
When I was starting out writing PromQL things, I found people singing
the praises of each of them and sometimes suggesting that you avoid
the other as misleading (eg,
and).
In particular, it's often said that you should use irate()
lest
you miss brief activity spikes, which is both true and not true
depending on how exactly you use irate()
. To explain that, I need
to start with what these two functions do and go on to the corollaries.
rate()
is the simpler function to describe. Ignoring things like
counter resets, rate()
gives you the per second average rate
of change over your range interval by using the first and the last
metric point in it (whatever they are, and whatever their timestamps
are). Since it is the average over the entire range, it necessarily
smooths out any sudden spikes; the only things that matter are the
start and the end values and the time range between them.
As for irate()
, I'll quote straight from the documentation:
[
irate()
] calculates the persecond instant rate of increase of the time series in the range vector. This is based on the last two data points.
(Emphasis mine.)
In other words, irate()
is the per second rate of change at the
end of your range interval. Everything else in your range interval
is ignored (if there are more than two data points in it).
There are some immediate corollaries that follow from this. First,
there's no point in giving irate()
a particularly large range
interval; all you need is one that's a large enough to insure that
it has two points in it. You'll need this to be somewhat larger
than twice your scrape interval, but there's little point in going
further than three or four times that. Second, irate()
is not
going to be very useful on metrics that change less often than you
scrape them, because some of the time the last two points that
irate()
will use are the same metric update, just scraped twice,
and irate()
will therefor report no change.
(Consider, for example, a metric that you expose through the node
exporter's textfile
collector and generate once a minute from cron, while your Prometheus
configuration scrapes the node exporter itself once every fifteen
seconds. Three out of every four metric points collected by Prometheus
are actually the same thing; only every fourth one represents a
genuine new data point. Similar things can happen with metrics
scraped from Pushgateway.)
Obviously, the difference between rate()
and irate()
basically
vanish when the range interval gets sufficiently small. If your
range interval for rate()
only includes two metric points, it's
just the same as irate()
. However, it's easier to make irate()
reliable at small range intervals; if you use rate()
, it may be
chancy to insure that your range interval always has two and only
two points. irate()
automatically arranges that for you by how
it works.
Often when we use either rate()
or irate()
, we want to graph
the result. Graphing means moving through time with query steps and that means we get into interactions
between the query step and both the range interval and the function
you're using. In particular, as the query step grows large enough,
irate()
will miss increasingly large amounts of changes. This
is because it is the instant rate of change at the end of your
range interval (using the two last metric points). When the query
steps include more than two points in each interval, you lose the
information from those extra points.
As an extreme example, imagine a query step of five minutes and a
metric that updates every thirty seconds. If you use irate()
,
you're only seeing the last minute out of every five minute slice
of time; you have no idea of what happened in the other four minutes,
including if there was an activity spike in them. If you use rate()
instead, you can at least have some visibility into the total changes
across those five minutes even if you don't capture any short term
activity spikes.
There are two unfortunate corollaries of this for graphing. First,
the range interval you likely want to use for rate()
depends
partly on your query step. For many graphs, there is no fixed one
size fits all range interval for rate()
. Depending on what you
want to see, you might want a rate()
range interval of either the
query step (which will give you the rate()
of disjoint sets of
metric points) or somewhat more than the query step (which will
give you somewhat overlapping metric points, and perhaps smooth
things out). The second is that whether you want to use rate()
or irate()
depends on whether your query step is small or large.
With a small query step and thus a high level of detail, you probably
want to use irate()
with a range interval that's large enough to
cover three or four metric points. But for large query steps and
only broad details, irate()
will throw away huge amounts of
information and you need to use rate()
instead.
This is where we come to Grafana. Current versions of Grafana offer
a $__interval
Grafana templating variable that
is the current query step, and you can plug that into a rate()
expression. However, Grafana offers no way to switch between rate()
and irate()
depending on the size of this interval, and it also
ties together the minimum interval and the minimum query step (as
I sort of grumbled about in this entry).
This makes Grafana more flexible than basic interactive Prometheus
usage, since you can make your rate()
range intervals autoadjust
to fit the query steps for your current graphs and their time ranges.
However you can't entirely get a single graph that will show you
both fine details (complete with momentary spikes) over small time
ranges and broad details over large time ranges.
(People who are exceedingly clever might be able to do something
with Grafana's $__interval_ms
and multiple metric calculations.)
For our purposes, I think we want to use rate()
instead of irate()
in our regular Grafana dashboard graphs because we're more likely
to be looking at broad overviews with fairly wide timescales. If
we're looking at something in fine detail, we can turn to Prometheus
and byhand irate()
based queries and graphs.
PS: Some of this is grumbling. For many practical purposes, a Grafana
graph with a rate()
range interval and query step that's down to
twice your Prometheus metric update interval is basically as good
as an irate()
based graph at your metric update interval. And if
you regularly need a Grafana dashboard for finegrained examination
of problems, you can build one that specifically uses irate()
in
places where it's actually useful.
Sidebar: The minimum range interval versus the minimum query step
Suppose that you have a continuously updating metric that Prometheus
scrapes every fifteen seconds. To do a rate()
or irate()
of
this, you need at least two metric points and thus a range interval
of thirty seconds (at least; in practice you need a somewhat larger
interval). However, you get a new metric value roughly every fifteen
seconds and thus a potentially new rate of change every fifteen
seconds as well (due to your new metric point). This means it's
reasonable to look at a graph with a query step of fifteen seconds
and an interval of thirty seconds.
(And you can't go below an interval of thirty seconds or so here,
because once your range durations only include one metric point,
both rate()
and irate()
stop returning anything and your graph
disappears.)

