2021-01-08
How to extract raw time series data from Prometheus
Suppose, not entirely hypothetically, that you have some interesting metrics in your Prometheus system and you would like to get the raw time series data points out of Prometheus. For example, suppose that you are seeing DNS resolution failures for Google domains (except not from Google's own 8.8.8.8 resolver), and you want to know exactly when your Blackbox probes failed and succeeded over some time range of interest.
Prometheus obviously has this data in its underlying time series
database (TSDB). Unfortunately, Prometheus makes it hard to work
out when exactly things happened. The
straightforward way of looking at this data, such as graphing the
value of the probe_success metric for your probe, will round
off times and show you not the exact time when something happened
but the time of the query step after it.
This is true even if you use a command line tool like 'promtool
query range' or curl to make a range query and tell you the
results (but at least that way you don't have to read times off a
graph).
(The query results from Prometheus always have both a value and a 'timestamp'. The timestamp is not normally shown in the Prometheus console interface, but it's returned in the raw JSON data and most tools will show that.)
Fortunately there is a way out: if you make an instant query for simple metrics using a range vector selector, you get the raw TSDB data (which I've sort of seen before). That is, suppose that you make the following query:
probe_success { probe="dns_gmail_mx" } [10m]
You will get back a set of results that look like this (in the
text format of 'promtool query instant'):
probe_success{probe="dns_gmail_mx", ....} =>
1 @[1610165365.569]
1 @[1610165454.569]
1 @[1610165543.569]
1 @[1610165632.569]
1 @[1610165721.569]
1 @[1610165810.569]
1 @[1610165899.569]
These timestamps are straight from the underlying TSDB data; they aren't any sort of query step time or query evaluation time. Here they're 89 seconds apart because that's how often we perform this particular Blackbox check (and this is the metric series for only one of the DNS resolvers we check).
(You can verify this by checking the time of the most recent
metric point against what 'timestamp(...)' will give you.)
So if you want to dump the results for 9 am to 10 am (Eastern time) today for your DNS query checks, with exact timestamps and results, what you want is an instant query at the 10 am end time with a range vector time range that stretches back to the start time:
promtool query instant --time 1610118000 .... 'probe_success{ probe="dns_gmail_mx" } [60m]'
(You may want to get the result in JSON format and then reformat
it with jq.)