I wish Grafana dashboards and panels could have easy, natural comments

June 16, 2022

Recently I was looking at a panel in one of our Grafana dashboards and noticed that its PromQL queries used avg_over_time() when it (now) felt as if max_over_time() was what the panel should be using. It's been years since I created this panel and last touched it, and I definitely no longer remember what I was thinking at the time. Did I have a good reason that avg_over_time() was necessary, or did an average just feel more correct for the purpose of the panel at the time I created it?

We have Prometheus alert rules with similarly tangled PromQL expressions, but since Prometheus alert rules are (normally) configured in YAML text files that allow comments, most of our complicated and non-obvious alert rules have commentary about why they're that way, what options don't work, and so on. This commentary has periodically been extremely helpful for refreshing my mind about what on earth past me was thinking when he wrote the rule.

Setting up Grafana panels and dashboards is normally done through their web GUI, which doesn't really offer any good way of writing this sort of commentary. In theory the GUI could offer space and options for this; in practice, such things would probably be considered clutter by designers (and they would sort of be right). Writing comments isn't anywhere near as natural in a GUI environment as it is in text.

(There are ways of provisioning Grafana from text files, but they aren't the natural way to use Grafana. One sign of this is that Grafana natively stores your dashboards and panels in a SQLite database instead of a more externally editable format.)

I don't know if I'd written a comment about this particular PromQL query even if Grafana had comments; I might have considered it obvious at the time and not done so. But at least it would be more likely, and if I hadn't left a comment back then, I probably would now (now that I've stubbed my foot on this).

Basically, everything that you can configure in any complex way should have comments. Grafana dashboards and panels definitely count.

(I was looking at this panel for unhappy reasons.)

Comments on this page:

By dozzie at 2022-06-17 08:07:04:

If you think about it, Grafana is a GUI builder, like Borland Delphi, just dedicated to dashboards. If you approach from this perspective, it's clear how awful Grafana is at being a GUI builder, and it's also clear how it could work if you could define a dashboard as a program (like in Tcl/Tk, Java AWT/Swing, GTK+, Qt, and so on) or declaratively (XML definitions in gDesklets; and Grafana's JSON is not a good example of this, because it's just a dump of all options of used widgets and you're not really meant to use write JSON directly).

With this, it's no wonder that a good place for comments and remarks could be found, but Grafana Labs never made one (or never made one easily accessible).

By hdhoang at 2022-06-17 08:56:39:

we stuff such comment in panel info textbox & more links there too. it's an (i) icon on panels' top-left corner.

for promql specifically, grafana 8.5 accepts & highlights in-line #-comment correctly. But i'm not sure if it's accessible to viewers (even if through panel json dump). the info icon is more accessible. at least future editors can get it when they want to edit

By cks at 2022-06-17 16:17:10:

Because the text box/info box is accessible in general, I tend to use it to explain to viewers what the particular panel is showing and what its limitations are, rather than any sort of deep dive on how the panel is put together (which is only interesting to people editing it, ie me).

(The text box isn't also directly co-located with eg queries, which means I would have to go back and forth when commenting and updating comments as queries update.)

As far as Grafana being a GUI builder goes, I think being a GUI builder is important and perhaps essential to its success, because it lets people start with it easily. Honestly, if I'd had to define panels and dashboards in text format from the start (when I didn't know Grafana and didn't know what we wanted), I'm not sure I'd have persisted with it. But Grafana could have a reasonable text-based serialization or storage format that allowed for comments, such as YAML, so you could start out in the GUI builder and then transition to editing things through text.

(Such text based editing would make certain sorts of dashboard and panel building a lot easier, because there can be a lot of repetition and fine detail that's easier to express and check in text than through the GUI.)

By dozzie at 2022-06-18 09:52:21:

As far as Grafana being a GUI builder goes, I think being a GUI builder is important and perhaps essential to its success, because it lets people start with it easily.

Of course it's popular because it's a graphical GUI builder. My point was something different: it's a poor GUI builder, and how poor it is is clearly visible when you compare it with just about any other GUI builder, or even GUI toolkits.

Written on 16 June 2022.
« Understanding some peculiarities of per-cgroup memory usage accounting
What is our Python 2 endgame going to be? »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Jun 16 22:47:35 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.