Chris's Wiki :: blog/sysadmin/Grafana9StatusOverTime Commentshttps://utcc.utoronto.ca/~cks/space/blog/sysadmin/Grafana9StatusOverTime?atomcommentsDWiki2022-12-20T03:43:37ZRecent comments in Chris's Wiki :: blog/sysadmin/Grafana9StatusOverTime.By Chris Siebenmann on /blog/sysadmin/Grafana9StatusOverTimetag:CSpace:blog/sysadmin/Grafana9StatusOverTime:c243f87573dadbb983031eca16f03e4b321d4992Chris Siebenmann<div class="wikitext"><p>This approach unfortunately isn't a good solution for us. We want to
see a count of how many instances of an alert have been firing, not
just a zero or active indicator, and we don't want the no-alert case
to show anything so that the points with alerts are easier to pick out.
In addition, setting 'No value' to 0 instead of the default blank and
fiddling with either value mappings or thresholds doesn't seem to make
the Status History panel render any more often than before; it's still
mostly blank (in the query I gave), even with a large minimum step
interval.</p>
<p>(It is obtaining query results, because it shows the labels you'd expect.)</p>
</div>2022-12-20T03:43:37ZBy Tyler Leeds on /blog/sysadmin/Grafana9StatusOverTimetag:CSpace:blog/sysadmin/Grafana9StatusOverTime:7453c722ba3a27066c14ecf9564635c635a2c53fTyler Leedshttps://www.linkedin.com/in/tylercleeds<div class="wikitext"><p>For the status history panel:</p>
<p>Try using this as your query:
<code>present_over_time((count(ALERTS{severity=~"[1234]"}) by (alertname))[$__rate_interval:$__rate_interval])</code></p>
<p>Set the minimum time interval to something sane (not 30s). Set the "No Value" value to 0, delete thresholds. Tell it to "merge equal consecutive values". </p>
<p>You can then set the value mappings so:
0 -> OK -> Green
1 -> Alert -> Red</p>
<p>Works pretty well. Still gets pretty bunched up if the panel is short and you have a bunch of alerts. We attach a severity label to all our alerts in alertmanager config, so you can filter by using a regex against the ALERTS metric.</p>
</div>2022-12-19T20:18:15Z