Grafana's problem with the order of dashboard panel legends and Prometheus

September 8, 2022

I tweeted something a bit obscure:

Although it's purely to deal with a missing Grafana feature, I do wish that Prometheus had some way to sort the returned query results by the value of some label. (I could hope for Grafana to add some way to deal with this, but good luck with that.)

It's common to build Grafana dashboards with graph panels (including bar graphs) that have multiple things in them. Generally when you do this, you include a legend with labels. The legend labels come from the query or queries you make, as covered in the query options for Prometheus as a datasource (the same is true for Loki as a datasource).

If each separate thing on the panel comes from a separate query, it's easy to put the legend labels in any order that you want, because the legend order comes from the order of the queries, and Grafana lets you freely re-order them. You might have this if you're graphing various memory metrics from a Linux server, for example, because the Prometheus host agent exposes each different thing from /proc/meminfo as a separate metric (and some things aren't exposed directly anyway, such as the amount of swap space used; you get only 'total swap' and 'free swap').

However, often you generate multiple time series from a single query and given them legend labels based on the value of some label from the metric; for example, all disk writes are together under one metric name, node_disk_written_bytes_total, and it has a 'device' label. In this case, the order of legend labels is the order that Prometheus returns all the time series in. This is not necessarily the order that you actually want (although I believe it's stable), and also isn't necessarily the same between different queries about the same things (so, for example, different queries about different disk metrics may return results in different order, resulting in graphs with different order of disks in the legend).

(I don't think Prometheus documents the order that query results are returned in. Experimentally, as of the current 2.38.0, Prometheus seems to order the labels alphabetically by name and then sort alphabetically by the resulting combination of label names and label values.)

Grafana provides no feature to change this order, although it could probably easily offer a 'sort by label name (ascending/descending)' option in the query options, or perhaps offer it as a general transformation. Prometheus currently provides no documented feature to control the order that time series are returned in based on label values, which isn't surprising since it mostly doesn't make any promises about the order in the first place.

Prometheus does provide sort() and sort_desc() to sort the time series by value. Although this can be useful to force an order, there can be three issues with it. The first is that it's generally not going to be consistent from panel to panel in the same dashboard. Second, the order can change when the dashboard refreshes (or when you explicitly look at a different time range). And third, I'm not sure how this interacts with actual graphs, as opposed to 'value over entire time interval' bar graphs, since the order may be different at every different step across the range.


Comments on this page:

By Andrew at 2022-09-09 00:10:23:

I dealt with this with Graphite, back when Grafana was a Graphite-only concern. I was moving my company over from original-flavor Graphite to carbonapi. Graphite-web returned metrics in a deterministic order, but carbonapi had a random order based partially on which metric queries ran fastest. Seeing the legends and colors on a dashboard full of graphs randomize themselves every 30 seconds was trippy and annoying. I ended up sending a patch to make carbonapi deterministic (and, as near as I could tell, the same as Graphite).

I do wish Prometheus would do the same but I think fixing that is beyond my skills. You're right that, now that Grafana has so many data sources, it should be nice to have a way to impose that ordering in Grafana itself instead of depending on a datasource to get it right.

By cks at 2022-09-09 13:13:20:

I think that it's almost impossible for the data source to always sort the way you want in Grafana, because the legend text is ultimately built in Grafana and don't necessarily come from one single label value (you can template it from multiple ones and extra text, at least for Prometheus and Loki). It would also be nice to be able to sort the legend text across multiple queries, which is something that would only be possible as a Grafana transform (not even a per-query sorting order).

Written on 08 September 2022.
« What systemd timer directives seem to be used in practice
How we monitor the temperature of our machine rooms »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Sep 8 22:30:59 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.