Brief notes on making Prometheus instant queries with curl
Every so often, I wind up wanting to directly extract some information from Prometheus without going through a Grafana dashboard or even the Prometheus web UI. At this point I have stubbed my toes on the same issues enough times (ie, more than once) that I'm going to write down some how-to information for my future reference.
In general, the Prometheus HTTP API itself is tersely documented
in the obvious spot in the documentation, and
see also Extracting raw samples from Prometheus.
Since it returns JSON, you'll probably want to have some JSON
processing program for command line usage; I'm fond of jq
, which at this point should be in
everyone's toolbox.
Making a query requires at least a PromQL
expression. In simple cases where all you want is all metric points
for a bare metric, you can give this directly on the command line
for curl
, eg:
curl 'localhost:9090/api/v1/query?query=up'
In more complex cases you'll have a PromQL query that requires
encoding to be properly interpreted, and perhaps extra parameters
on the query. Fortunately Prometheus accepts POST
requests just
like GET
requests, and curl
will do the work for us:
curl --data-urlencode 'query=up{job="alertmanager"}' --data-urlencode 'time=1556330052' localhost:9090/api/v1/query
Generally I want to use the -s
flag with curl
, to make it not
spit out status information when I'm feeding its output to jq
.
I may or may not want to use -f
to avoid server error pages. In
scripts I want some sort of timeout, such as '-m 10
'.
Some sources will provide curl
command lines that also use -k
.
You don't want to use that flag unless you know what it means and
you know that you need it and it's safe to use it.
(Possibly there is a clever way to get curl
to URL-encode your
query parameters in GET
requests, but if so I didn't see it in
a casual skim of the very large curl
manpage. I don't think it matters
for querying Prometheus, since Prometheus accepts either GET
or
POST
requests here.)
PS: Extracting raw samples from Prometheus says that 'the remote
read endpoint at api/v1/read
[...] is more difficult to use [...]'.
That would be a polite understatement. It turns out that Prometheus
remote read requests and replies are snappy-compressed protobufs, following schemas
that you can find here.
You can apparently work with protobufs from the command line
with curl
, per this gist example, but I don't know
how you'd handle the snappy compression side of things.
Sidebar: Sending data to Pushgateway with curl
Since I do this too, I might as well add my usual curl
command
for it here:
<generate metrics as text> | curl -X PUT -f -s --data-binary @- localhost:9091/metrics/job/...
I almost always use PUT
here instead of the default of POST
because of when and what metrics disappear on updates in Pushgateway.
|
|