Over on the Fediverse, I had a Grafana and Prometheus wish:

I wish Prometheus could tell Grafana what the scrape interval for a metric is, so Grafana could always set a sensible minimum interval for dashboard panels.

This thought is brought to you by changing entirely too many panels because I changed from 15s to 30s scrapes for our most verbose set of metrics (the ZFS exporter, which has many histograms).

In many cases, visualizing Prometheus metrics will require you to use rate() or maybe irate() (cf, but also). Generally this means that you'll wind up wanting a minimum query step of twice your scrape interval; otherwise, your rate() may not have two samples and may not give you any results (at least if you're using the query step as your range interval). Conveniently, Grafana allows you to configure a minimum interval for every panel, or even every query.

All of this is great and works fine right up until you do what I did, which is lengthen the scrape period and thus the necessary minimum query step for one set of metrics from our Prometheus data source. Since this is only one set of metrics, not all of them, I can't just set a larger minimum interval for the datasource as a whole (and generally you can't do this with Prometheus; in many configurations you'll have different scrape speeds for different exporters). This required me to go through all of the panels and change '30s' to '60s', which was just as tedious as you'd expect in a GUI.

As it turns out, there is a better, more future proof way that @halfa pointed out here and here, using an undocumented but apparently long-standing feature of Grafana dashboard variables. As currently documented, Grafana dashboard variables can only be used in a few places. However, it turns out that you can use 'interval' variables (and perhaps dashboard variables of other types) in the Query Options section of panel settings to set the 'Min Interval'. The example by @halfa used the syntax '>$variable', but I've tested with a plain '$variable' and it worked for me (albeit in Grafana 8.3.11, which we're still on for reasons beyond the scope of this entry).

So, what I can do to make my life easier is as follows. First, I create a dashboard variable, let's call it '$zfsinterval', that I make an 'interval' type variable, with one value of '60s', and I set it to be completely hidden. Then I can go through all of my panels that are based on metrics from the ZFS exporter and set them to have a Min Interval of '$zfsinterval', instead of any literal value. If I later change the scrape interval for the ZFS exporter back to 15s, or lengthen it to 45s, I only have to change one place in the dashboard configuration to update everything to the new minimum interval needed.

(You don't have to hide this dashboard variable, or give it only a single value if you're not hiding it, but generally there's not much point in exposing it. Maybe if you're scraping the same metric for some servers at a higher rate than for other servers, so the minimum interval you want is server specific.)