Prometheus's delta() function can be inferior to subtraction with offset

March 24, 2019

The PromQL delta() function is used on gauges to, well, let's quota its help text:

delta(v range-vector) calculates the difference between the first and last value of each time series element in a range vector v, returning an instant vector with the given deltas and equivalent labels. The delta is extrapolated to cover the full time range as specified in the range vector selector, so that it is possible to get a non-integer result even if the sample values are all integers.

Given this description, you would expect that 'delta(yourmetric[24h])' is preferable to the essentially functionally equivalent but more verbose version using offset:

yourmetric - yourmetric offset 24h

(Ignoring some hand waving about any delta extrapolation and so on.)

Unfortunately it is not. In some situations, the offset based version can work when the delta() version fails.

The fundamental problem is unsurprisingly related to Prometheus's lack of label based optimization, and it is that using delta() attempts to load all samples in the entire range into memory, even though most of them will be ignored and discarded. If your metric has a lot of metric points, for example because it has relatively high metric cardinality (many different label values), attempting to load all of the samples into memory can trip Prometheus limits and cause the delta()-based version to fail. The offset based version only ever loads metric points from two times, so it will almost always work.

On the one hand, it's easy to see how Prometheus's implementation of PromQL could wind up doing this. It is natural to write general code that loads range vectors and then have delta() just call it generically and ignore most of the result, especially since there are various special cases. On the other hand, this is a very unfortunate artificial limit that's probably eventually going to affect any delta() query that's made over a sufficiently large timescale.

(This issue doesn't affect rate() and friends, at least in one sense. Because rate() and company have to check for resets over the entire time range, they need to load and use all of the sample points. You can't replace an increase() with an offset unless you're willing to ignore any errors caused by counter resets. If you're doing ad-hoc queries, you probably need to narrow down the number of metric points you're trying to load by using labels and so on. And if you really want to know, say, the average interface bandwidth for a specific network interface over an entire year, you may be plain out of luck until you put more RAM in your Prometheus server and increase its query limits.)


Comments on this page:

By Davor C. at 2019-04-24 19:05:54:

Also, delta extrapolates, which will look weird if you are calculating some an integral increase. And if the time interval is small (it's not here, I know), you can get time series in Grafana that jump around or have peaks that occasionally appear and then a minute later go away. offset will use the exact sample values at that point in time, so you'll get counters that behave as you'd expect.

Written on 24 March 2019.
« Link: What has your microcode done for you lately?
The mystery of my desktop that locks up when it gets too cold »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Mar 24 18:58:22 2019
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.