Yesterday, I wrote about my wish for easier ways to evaluate a Prometheus query at a given time. Today I want to write down a few ways to do this for my future use, partly since I discovered one more way than I'd known when I went poking around after writing that entry.

Prometheus has an ad-hoc query view, which gives you answers either in a table or in a graph. The graph view has always supported adjusting the end time of the graph, in the same area above the graph where you change the time range and so on. In the table view you have the same option in the small 'Evaluation time' section. As of 2.23.0, this has a bug where if you directly edit the date or time (instead of picking from the calendar), your query may not be properly re-evaluated until you click outside the calendar, click on the time again to get the calendar to appear, and then click outside it again. If the evaluation time is wrong, clicking 'Execute' doesn't seem to fix it, which is irritating. But all of this is a bug and will probably go away at some point.

You can manually do instant queries with curl and as part of that you can specify the query time, as I wrote up in my notes on making Prometheus queries with curl. The normal form of this query is:

curl -s --data-urlencode 'query=up{job="alertmanager"}' --data-urlencode 'time=1556330052' http://localhost:9090/api/v1/query

The time is specified as a Unix timestamp. The most convenient way to get this is with some useful features of GNU date:

date -d 'Fri Apr 26 21:54:12 EDT 2019' +%s

It also turns out that promtool can be used to do this as part of 'promtool query instant', using the '--time' option (as of at least Prometheus 2.23.0):

promtool query instant --time 1609301510 http://localhost:9090/ node_load1

Unlike with direct manual queries with curl, you don't directly supply the URL of the query API, you instead supply the top level Prometheus URL. Promtool will put on the rest of the URL components as needed.

As documented in 'promtool query instant --help', --time takes either a Unix timestamp or a RFC 3339 time string. Such a time string can be produced by GNU Date, but to produce a RFC 3339 time string that promtool will accept, you want to use the '--iso-8601=seconds' command line option, not the option that GNU Date incorrectly claims is for RFC 3339 format. You may or may not want to use --utc.

(You can get JSON output by using 'promtool query instant -o json'.)

My current view is that using curl is better if you're going to process the result as JSON, while 'promtool query instant' is better if you just want to read the results. If I needed to use either for this regularly I would probably write a cover script, and I would probably have the cover script use 'date -d ...' on whatever time string I handed it for maximum convenience.

(Both curl and promtool have the annoyance for ordinary shell usage that you have to supply a probably gigantic URL to your Prometheus server. This URL is almost always constant but there's no way to provide a default value. promql-cli lets you provide a default Prometheus URL, but currently has no option for making an instant query at a time.)