Wandering Thoughts archives

Some ways to do a Prometheus query as of a given time

Yesterday, I wrote about my wish for easier ways to evaluate a Prometheus query at a given time. Today I want to write down a few ways to do this for my future use, partly since I discovered one more way than I'd known when I went poking around after writing that entry.

Prometheus has an ad-hoc query view, which gives you answers either in a table or in a graph. The graph view has always supported adjusting the end time of the graph, in the same area above the graph where you change the time range and so on. In the table view you have the same option in the small 'Evaluation time' section. As of 2.23.0, this has a bug where if you directly edit the date or time (instead of picking from the calendar), your query may not be properly re-evaluated until you click outside the calendar, click on the time again to get the calendar to appear, and then click outside it again. If the evaluation time is wrong, clicking 'Execute' doesn't seem to fix it, which is irritating. But all of this is a bug and will probably go away at some point.

You can manually do instant queries with curl and as part of that you can specify the query time, as I wrote up in my notes on making Prometheus queries with curl. The normal form of this query is:

curl -s --data-urlencode 'query=up{job="alertmanager"}' --data-urlencode 'time=1556330052' http://localhost:9090/api/v1/query

The time is specified as a Unix timestamp. The most convenient way to get this is with some useful features of GNU date:

date -d 'Fri Apr 26 21:54:12 EDT 2019' +%s

It also turns out that promtool can be used to do this as part of 'promtool query instant', using the '--time' option (as of at least Prometheus 2.23.0):

promtool query instant --time 1609301510 http://localhost:9090/ node_load1

Unlike with direct manual queries with curl, you don't directly supply the URL of the query API, you instead supply the top level Prometheus URL. Promtool will put on the rest of the URL components as needed.

As documented in 'promtool query instant --help', --time takes either a Unix timestamp or a RFC 3339 time string. Such a time string can be produced by GNU Date, but to produce a RFC 3339 time string that promtool will accept, you want to use the '--iso-8601=seconds' command line option, not the option that GNU Date incorrectly claims is for RFC 3339 format. You may or may not want to use --utc.

(You can get JSON output by using 'promtool query instant -o json'.)

My current view is that using curl is better if you're going to process the result as JSON, while 'promtool query instant' is better if you just want to read the results. If I needed to use either for this regularly I would probably write a cover script, and I would probably have the cover script use 'date -d ...' on whatever time string I handed it for maximum convenience.

(Both curl and promtool have the annoyance for ordinary shell usage that you have to supply a probably gigantic URL to your Prometheus server. This URL is almost always constant but there's no way to provide a default value. promql-cli lets you provide a default Prometheus URL, but currently has no option for making an instant query at a time.)

A Prometheus wish: easy ways to evaluate a PromQL query at a given time

To do most things in Prometheus, you use PromQL to make queries against Prometheus' internal time series database that holds all of the metrics that your system has collected over however long your retention time is. PromQL queries are always evaluated at a given time, with the default for simple queries being 'right now' (this is true even for queries that ask for, say, the average of something over the past five minutes; those past five minutes start at the query's evaluation time).

PromQL has support for evaluating an expression at a time relative to the current time (well, the query's time), in the form of the offset modifier. This is handy for comparing the state of things now to the state of things some time ago, and using offset can make your queries more efficient (although there can be gotchas with using offset instead of delta()).

What Prometheus and PromQL have limited support for is evaluating a PromQL query at a specific (absolute) time. This is partly a PromQL language issue and partly a tool and user interface issue. The underlying instant query HTTP API allows you to specify the time that the query is evaluated at (what is considered 'now' for it), so it's possible to make whole queries at non-default times. The current Prometheus UI for ad-hoc queries actually does have support for manually setting the query time, but it's not very obvious and it has some bugs (which will probably get fixed in the future).

There are some tools for making PromQL queries from the command line, such as promql-cli, but I don't know of any current one that lets you specify an instant query time in a human readable format. Making raw queries with curl will let you do this, but you have to work out the Unix timestamp you need from the date (which can be done with GNU date, by combining some features covered in some useful features of GNU date).

PromQL itself has no syntax for making queries at absolute times, unlike for relative times with offset. You might think that you can press offset into service anyway by computing the offset on the fly based on the difference between the current time (available via time()) and the target time, but unfortunately offset specifically requires a duration and you can't give it an expression. There are likely good reasons to limit offset this way, but it does make me wish for an 'at' modifier, even one that only took a Unix timestamp.

(What is frustrating about this is that of course the information is in there and the capacity to access it is intrinsic in Prometheus. It's just not well exposed or problem-free.)