A Prometheus wish: easy ways to evaluate a PromQL query at a given time
To do most things in Prometheus, you use PromQL to make queries against Prometheus' internal time series database that holds all of the metrics that your system has collected over however long your retention time is. PromQL queries are always evaluated at a given time, with the default for simple queries being 'right now' (this is true even for queries that ask for, say, the average of something over the past five minutes; those past five minutes start at the query's evaluation time).
PromQL has support for evaluating an expression at a time relative
to the current time (well, the query's time), in the form of the
This is handy for comparing the state of things now to the state of
things some time ago, and using
offset can make your queries more
efficient (although there can be gotchas
offset instead of
What Prometheus and PromQL have limited support for is evaluating a PromQL query at a specific (absolute) time. This is partly a PromQL language issue and partly a tool and user interface issue. The underlying instant query HTTP API allows you to specify the time that the query is evaluated at (what is considered 'now' for it), so it's possible to make whole queries at non-default times. The current Prometheus UI for ad-hoc queries actually does have support for manually setting the query time, but it's not very obvious and it has some bugs (which will probably get fixed in the future).
There are some tools for making PromQL queries from the command
line, such as promql-cli,
but I don't know of any current one that lets you specify an instant
query time in a human readable format. Making raw queries with
curl will let you do this, but you have to work out the Unix
timestamp you need from the date (which can be done with GNU date,
by combining some features covered in some useful features of GNU
PromQL itself has no syntax for making queries at absolute times,
unlike for relative times with
offset. You might think that you
offset into service anyway by computing the offset on
the fly based on the difference between the current time (available
time()) and the target time, but unfortunately
specifically requires a duration and you can't give it an expression.
There are likely good reasons to limit
offset this way, but it
does make me wish for an '
at' modifier, even one that only took
a Unix timestamp.
(What is frustrating about this is that of course the information is in there and the capacity to access it is intrinsic in Prometheus. It's just not well exposed or problem-free.)