Chris's Wiki :: blog/__TopicPrometheus

Topic: Prometheus and Grafana

This collects most or all of the entries I've written on Prometheus and Grafana, in reverse chronological order. You can also see the overall index of entries (or the chronological index).

2023-03-19:	Easily adjusting the minimum interval on panels in Grafana dashboards
2023-03-13:	What I like using Grafana Loki for (and where I avoid it)
2023-02-21:	Grafana Loki doesn't compact log chunks and what this means for you
2023-02-07:	What I want in Prometheus (as a whole) is aggregating alert notifications
2023-02-05:	Some things on Prometheus's new feature to keep alerts firing for a while
2023-02-02:	A gotcha when making partial copies of Prometheus's database with rsync
2023-01-21:	How Prometheus makes good use of the HTTP Accept: header
2023-01-03:	Some thoughts on Prometheus Alertmanager's alert reminders
2022-12-21:	The Prometheus cardinality issues with systemd unit-related metrics
2022-12-14:	Your options for displaying status over time in Grafana 9
2022-12-11:	Prometheus Blackbox 0.23.0 has added a nice improvement to its DNS checks
2022-10-21:	The Prometheus `timestamp()` function can be used on expressions, sort of
2022-10-11:	When Promtail seems to make position checkpoints (as of v2.6.1)
2022-09-30:	Your Grafana Loki setup needs security and access control
2022-09-14:	Grafana Loki doesn't duplicate a central syslog server (or vice versa)
2022-09-12:	What's lost when running the Prometheus host agent as a non-root user on Linux
2022-09-08:	Grafana's problem with the order of dashboard panel legends and Prometheus
2022-09-06:	Machine room temperatures and the value of long Prometheus metrics history
2022-09-03:	Our Prometheus host metrics saved us from some painful experiences
2022-09-02:	An rsyslog(d) syslog forwarding setup for Grafana Loki (via Promtail)
2022-08-28:	Getting USB TEMPer2 temperature sensor readings into Prometheus (on Linux)
2022-08-22:	Some notes on Grafana annotations sourced from Prometheus metrics
2022-08-19:	I wish Prometheus had a table-driven label remapping feature
2022-08-12:	My adventure with URLs in a Grafana that's behind a reverse proxy
2022-08-08:	Two example Grafana Loki log queries to get things from ntpdate logs
2022-07-31:	Using Prometheus's recent '@ end()' PromQL feature to reduce graph noise
2022-07-26:	To be fully useful, Prometheus histograms want their cumulative sums
2022-07-21:	You can sensibly move or copy Prometheus's database with rsync
2022-07-19:	We won't be sending systemd logs to Grafana Loki in JSON format
2022-07-18:	Grafana Loki and what can go wrong with label cardinality
2022-06-16:	I wish Grafana dashboards and panels could have easy, natural comments
2022-06-06:	Doing a selective alert about a host's additional exporters in Prometheus
2022-06-05:	Checking a few metrics (time series) at once in Prometheus's query language
2022-05-28:	It's a bit risky to give people access to your Prometheus Blackbox exporter
2022-05-06:	Filtering Prometheus metrics with deliberately repeated labels
2022-03-20:	Prometheus: using gauge-like things as if they were counters
2022-03-09:	Linux disk names you can encounter in your Prometheus host metrics
2022-03-07:	The convenience of multi-purpose monitoring (in Prometheus)
2022-02-04:	Some notes on Grafana relative time ranges
2022-01-15:	You should do lint checks on your Prometheus alert (and recording) rules
2022-01-14:	Link: Histograms in Grafana (a howto)
2022-01-11:	Some things about Prometheus Alertmanager's notification metrics
2022-01-10:	The complexity of seeing if your Prometheus Alertmanager is truly healthy
2021-11-30:	Prometheus will make persistent connections to agents (scrape targets)
2021-09-30:	Moving averages (and rates) for metrics in Prometheus (and Grafana)
2021-09-11:	How many Prometheus metrics a typical host here generates
2021-09-04:	Adding a "host" label to all of your per-host Prometheus metrics
2021-09-03:	How I try out and test new versions of Grafana
2021-08-13:	Prometheus alerts and the idea of "deadbands" (or maybe hysteresis) (with an implementation)
2021-08-12:	Prometheus, Alertmanager, and maybe avoiding flapping alerts
2021-08-04:	When you do and don't get stuck query results from a down Prometheus
2021-06-29:	Monitoring the status of Linux network interfaces with Prometheus
2021-06-17:	In Prometheus queries, `on` and `ignoring` don't drop labels from the result
2021-05-27:	Some thoughts on having set up a personal Alertmanager instance
2021-05-22:	I don't know how much memory our Prometheus setup needs
2021-05-15:	The size of our Prometheus setup as of May 2021
2021-04-12:	Counting how many times something started or stopped failing in Prometheus
2021-04-04:	Some uses for Prometheus's `resets()` function
2021-03-31:	Understanding Prometheus' `changes()` function and what it can do for me
2021-03-14:	I wish Prometheus had some features to deal with 'missing' metrics
2021-03-13:	Prometheus and the case of the stuck metrics
2021-02-24:	How convenience in Prometheus labels for alerts led me into a quiet mistake How (and where) Prometheus alerts get their labels
2021-02-22:	How I set up testing alerts in our Prometheus environment
2021-01-10:	What timestamps you get back along with Prometheus query results
2021-01-08:	How to extract raw time series data from Prometheus
2020-12-30:	Some ways to do a Prometheus query as of a given time A Prometheus wish: easy ways to evaluate a PromQL query at a given time
2020-12-15:	How to make Grafana properly display a Unix timestamp In Prometheus, it's hard to work with when metric points happened
2020-12-02:	Prometheus 2.23.0 now lets you display graphs in local time
2020-11-17:	Grafana and the case of the infinite serial number
2020-10-31:	A gotcha with combining single-label and multi-label Prometheus metrics
2020-10-17:	A potential Prometheus issue for labeled metrics for infrequent events
2020-08-18:	The Prometheus host agent can disturb Linux CPU frequency measurements
2020-08-07:	How we choose our time intervals in our Grafana dashboards
2020-07-14:	Link: The Anatomy of a PromQL Query
2020-06-29:	How Prometheus Blackbox's TLS certificate metrics would have reacted to AddTrust's root expiry
2020-06-25:	What Prometheus Blackbox's TLS certificate expiry metrics are checking
2020-06-05:	Why we put alert start and end times in our Prometheus alert messages
2020-06-04:	Formatting alert start and end times in Prometheus Alertmanager messages
2020-05-22:	Working out how frequently your ICMP pings fail in Prometheus
2020-03-30:	Notes on Grafana 'value groups' for dashboard variables
2020-03-28:	The Prometheus host agent's CPU utilization metrics can be a bit weird
2020-03-19:	Make sure to keep useful labels in your Prometheus alert rules
2020-02-29:	OpenBSD versus Prometheus (and Go)
2020-02-27:	Some alert inhibition rules we use in Prometheus Alertmanager
2020-02-26:	The magic settings to make a bar graph in Grafana
2020-01-26:	How big our Prometheus setup is (as of January 2020)
2020-01-05:	Why I prefer the script exporter for exposing script metrics to Prometheus
2020-01-04:	Three ways to expose script-created metrics in Prometheus
2019-12-30:	The history and background of us using Prometheus
2019-12-29:	Prometheus and Grafana after a year (more or less)
2019-12-28:	Our setup of Prometheus and Grafana (as of the end of 2019)
2019-12-02:	You can have Grafana tables with multiple values for a single metric (with Prometheus) Calculating usage over time in Prometheus (and Grafana)
2019-11-30:	Counting the number of distinct labels in a Prometheus metric
2019-11-25:	In Prometheus, don't be afraid of high cardinality metrics if they're valuable enough
2019-10-08:	How we implement reboot notifications when our machines reboot in Prometheus
2019-09-17:	Finding metrics that are missing labels in Prometheus (for alert metrics)
2019-09-02:	Another way to do easy configuration for lots of Prometheus Blackbox checks
2019-08-26:	A lesson of (alert) scale we learned from a power failure
2019-07-28:	A note on using the Go Prometheus client package to exposed labeled metrics
2019-06-28:	Using Prometheus's statsd exporter to let scripts make metrics updates
2019-06-02:	Exploring the start time of Prometheus alerts via `ALERTS_FOR_STATE`
2019-05-20:	Understanding how to pull in labels from other metrics in Prometheus
2019-05-03:	Some implications of using `offset` instead of `delta()` in Prometheus
2019-04-28:	A gotcha with stale metrics and *_over_time() in Prometheus
2019-04-26:	Brief notes on making Prometheus instant queries with `curl`
2019-04-21:	My view on upgrading Prometheus (and Grafana) on an ongoing basis
2019-04-18:	A pattern for dealing with missing metrics in Prometheus in simple cases
2019-04-13:	Remembering that Prometheus expressions act as filters
2019-03-24:	Prometheus's `delta()` function can be inferior to subtraction with `offset`
2019-03-18:	Prometheus subqueries pick time points in a surprising way
2019-03-12:	An easy optimization for restricted multi-metric queries in Prometheus
2019-03-11:	Testing Prometheus alert conditions through subqueries
2019-03-10:	What the default query step is for Prometheus subqueries
2019-03-06:	Using Prometheus subqueries to look for spikes in rates
2019-02-27:	Using Prometheus subqueries to do calculations over time ranges
2019-02-17:	Some notes on heatmaps and histograms in Prometheus and Grafana
2019-01-23:	A little surprise with Prometheus scrape intervals, timeouts, and alerts
2018-12-14:	Why our Grafana URLs always require HTTP Basic Authentication
2018-12-12:	One situation where you absolutely can't use `irate()` in Prometheus
2018-12-03:	Linux disk IO stats in Prometheus
2018-11-25:	How we monitor our Prometheus setup itself
2018-11-20:	When Prometheus Alertmanager will tell you about resolved alerts
2018-11-11:	Easy configuration for lots of Prometheus Blackbox checks
2018-11-10:	Why Prometheus turns out not be our ideal alerting system
2018-11-09:	Getting CPU utilization breakdowns efficiently in Prometheus
2018-11-05:	`rate()` versus `irate()` in Prometheus (and Grafana)
2018-10-28:	How I'm visualizing health check history in Grafana
2018-10-22:	Using group_* vector matching in Prometheus for database lookups
2018-10-18:	Some things on delays and timings for Prometheus alerts
2018-10-17:	When metrics disappear on updates with Prometheus Pushgateway
2018-10-13:	Getting a CPU utilization breakdown in Prometheus's query language, PromQL How Prometheus's query steps (aka query resolution) work
2018-10-11:	Some notes on Prometheus's Blackbox exporter

This is a Category/PageManagement page.