How I set up a server for testing new Grafana versions and other things

May 27, 2023

I mentioned yesterday that you probably should have a server that you can test Grafana upgrades on, and having one is useful for experiments. There are a couple of ways to set up such a server, and as it happens our environment is built in such a way to make it especially easy. Although our Prometheus server and Grafana run on the same machine and so Grafana could access Prometheus as 'localhost:9090', when I set this up I decided that Grafana should instead access Prometheus through our reverse proxy Apache setup, using the server's public name.

(When I set this up, I think I had ideas of being able to watch for potential Grafana query problems by looking at the Apache log and seeing the queries it was sending to Prometheus. Then Grafana switched from querying Prometheus with GET and query parameters to using POST and a POST body, a change that's better for Grafana but which does limit what we can now get from Apache logs.)

This makes setting up a testing Grafana server basically trivial. We (I) install a random (virtual) machine, follow the steps to set up Apache and Grafana as if it were our production metrics server, and copy the production metrics server's current grafana.db to it (while Grafana isn't running). When I restart Grafana, it will come up with all of our dashboards and talking to our production Prometheus data source, since it's using the public name. This gives me a way to directly compare new versions of Grafana against the production version, including trying to update old panel types to new panel types and comparing the results.

(In our environment we have few enough changes to the production grafana.db that I can just copy the file around more or less any time I want to; I don't need to shut down the production Grafana to save a safe copy.)

This would still be relatively simple if I'd used 'localhost:9090' as the URL for our Prometheus data source instead of its public URL, since you can change the URL of a data source. I'd just have to remember to modify the copied database that way every time I copied or re-copied it. If our Prometheus server wasn't accessible at all off the machine (either on its own or through a reverse proxy Apache), I would probably enable my testing by ssh'ing from the test server to the production server to port forward 'localhost:9090'. I couldn't leave this running all the time, but there's generally no reason to leave the test Grafana server running unless I'm specifically interested in something.

(This is far easier to do with virtual machines than with physical ones, since these days starting up and shutting them down is as simple as 'virsh start <x>' and 'virsh shutdown <x>'.)

PS: Another trick you can play with a Grafana testing server is to keep multiple copies of your grafana.db around and swap them in and out of being the live one depending on what you want to do. For example, you might have one with a collection of test dashboards (to look into things like how to display status over time in the latest Grafana), and another that's a current copy of your production Grafana's database.

Written on 27 May 2023.
« In practice, Grafana has not been great at backward compatibility
My current editor usage (as of mid 2023) »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sat May 27 22:03:15 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.