Prometheus's delta()
function can be inferior to subtraction with offset
The PromQL delta()
function is used on gauges to, well, let's quota its help text:
delta(v range-vector)
calculates the difference between the first and last value of each time series element in a range vectorv
, returning an instant vector with the given deltas and equivalent labels. The delta is extrapolated to cover the full time range as specified in the range vector selector, so that it is possible to get a non-integer result even if the sample values are all integers.
Given this description, you would expect that 'delta(yourmetric[24h])
'
is preferable to the essentially functionally equivalent but more
verbose version using offset
:
yourmetric - yourmetric offset 24h
(Ignoring some hand waving about any delta extrapolation and so on.)
Unfortunately it is not. In some situations, the offset
based version
can work when the delta()
version fails.
The fundamental problem is unsurprisingly related to Prometheus's
lack of label based optimization,
and it is that using delta()
attempts to load all samples in
the entire range into memory, even though most of them will be
ignored and discarded. If your metric has a lot of metric points,
for example because it has relatively high metric cardinality (many different label
values), attempting to load all of the samples into memory can trip
Prometheus limits and cause the delta()
-based version to fail.
The offset
based version only ever loads metric points from two
times, so it will almost always work.
On the one hand, it's easy to see how Prometheus's implementation
of PromQL
could wind up doing this. It is natural to write general code that
loads range vectors and then have delta()
just call it generically
and ignore most of the result, especially since there are various
special cases. On the other hand, this is a very unfortunate
artificial limit that's probably eventually going to affect any
delta()
query that's made over a sufficiently large timescale.
(This issue doesn't affect rate()
and friends, at least in one
sense. Because rate()
and company have to check for resets over
the entire time range, they need to load and use all of the sample
points. You can't replace an increase()
with an offset
unless
you're willing to ignore any errors caused by counter resets. If
you're doing ad-hoc queries, you probably need to narrow down the
number of metric points you're trying to load by using labels and
so on. And if you really want to know, say, the average interface
bandwidth for a specific network interface over an entire year, you
may be plain out of luck until you put more RAM in your Prometheus
server and increase its query limits.)
Comments on this page:
|
|