Every so often, I wind up wanting to directly extract some information from Prometheus without going through a Grafana dashboard or even the Prometheus web UI. At this point I have stubbed my toes on the same issues enough times (ie, more than once) that I'm going to write down some how-to information for my future reference.

In general, the Prometheus HTTP API itself is tersely documented in the obvious spot in the documentation, and see also Extracting raw samples from Prometheus. Since it returns JSON, you'll probably want to have some JSON processing program for command line usage; I'm fond of jq, which at this point should be in everyone's toolbox.

Making a query requires at least a PromQL expression. In simple cases where all you want is all metric points for a bare metric, you can give this directly on the command line for curl, eg:

curl 'localhost:9090/api/v1/query?query=up'

In more complex cases you'll have a PromQL query that requires encoding to be properly interpreted, and perhaps extra parameters on the query. Fortunately Prometheus accepts POST requests just like GET requests, and curl will do the work for us:

curl --data-urlencode 'query=up{job="alertmanager"}' --data-urlencode 'time=1556330052' localhost:9090/api/v1/query

Generally I want to use the -s flag with curl, to make it not spit out status information when I'm feeding its output to jq. I may or may not want to use -f to avoid server error pages. In scripts I want some sort of timeout, such as '-m 10'.

Some sources will provide curl command lines that also use -k. You don't want to use that flag unless you know what it means and you know that you need it and it's safe to use it.

(Possibly there is a clever way to get curl to URL-encode your query parameters in GET requests, but if so I didn't see it in a casual skim of the very large curl manpage. I don't think it matters for querying Prometheus, since Prometheus accepts either GET or POST requests here.)

PS: Extracting raw samples from Prometheus says that 'the remote read endpoint at api/v1/read [...] is more difficult to use [...]'. That would be a polite understatement. It turns out that Prometheus remote read requests and replies are snappy-compressed protobufs, following schemas that you can find here. You can apparently work with protobufs from the command line with curl, per this gist example, but I don't know how you'd handle the snappy compression side of things.

Sidebar: Sending data to Pushgateway with curl

Since I do this too, I might as well add my usual curl command for it here:

<generate metrics as text> |
   curl -X PUT -f -s --data-binary @- localhost:9091/metrics/job/...

I almost always use PUT here instead of the default of POST because of when and what metrics disappear on updates in Pushgateway.