A Prometheus wish: easy ways to evaluate a PromQL query at a given time

December 30, 2020

To do most things in Prometheus, you use PromQL to make queries against Prometheus' internal time series database that holds all of the metrics that your system has collected over however long your retention time is. PromQL queries are always evaluated at a given time, with the default for simple queries being 'right now' (this is true even for queries that ask for, say, the average of something over the past five minutes; those past five minutes start at the query's evaluation time).

PromQL has support for evaluating an expression at a time relative to the current time (well, the query's time), in the form of the offset modifier. This is handy for comparing the state of things now to the state of things some time ago, and using offset can make your queries more efficient (although there can be gotchas with using offset instead of delta()).

What Prometheus and PromQL have limited support for is evaluating a PromQL query at a specific (absolute) time. This is partly a PromQL language issue and partly a tool and user interface issue. The underlying instant query HTTP API allows you to specify the time that the query is evaluated at (what is considered 'now' for it), so it's possible to make whole queries at non-default times. The current Prometheus UI for ad-hoc queries actually does have support for manually setting the query time, but it's not very obvious and it has some bugs (which will probably get fixed in the future).

There are some tools for making PromQL queries from the command line, such as promql-cli, but I don't know of any current one that lets you specify an instant query time in a human readable format. Making raw queries with curl will let you do this, but you have to work out the Unix timestamp you need from the date (which can be done with GNU date, by combining some features covered in some useful features of GNU date).

PromQL itself has no syntax for making queries at absolute times, unlike for relative times with offset. You might think that you can press offset into service anyway by computing the offset on the fly based on the difference between the current time (available via time()) and the target time, but unfortunately offset specifically requires a duration and you can't give it an expression. There are likely good reasons to limit offset this way, but it does make me wish for an 'at' modifier, even one that only took a Unix timestamp.

(What is frustrating about this is that of course the information is in there and the capacity to access it is intrinsic in Prometheus. It's just not well exposed or problem-free.)

Written on 30 December 2020.
« It feels like the broad Unix API is being used less these days
Some ways to do a Prometheus query as of a given time »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Dec 30 00:44:14 2020
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.