2023-05-27
How I set up a server for testing new Grafana versions and other things
I mentioned yesterday that you probably should have a server that you can test Grafana upgrades on, and having one is useful for experiments. There are a couple of ways to set up such a server, and as it happens our environment is built in such a way to make it especially easy. Although our Prometheus server and Grafana run on the same machine and so Grafana could access Prometheus as 'localhost:9090', when I set this up I decided that Grafana should instead access Prometheus through our reverse proxy Apache setup, using the server's public name.
(When I set this up, I think I had ideas of being able to watch for potential Grafana query problems by looking at the Apache log and seeing the queries it was sending to Prometheus. Then Grafana switched from querying Prometheus with GET and query parameters to using POST and a POST body, a change that's better for Grafana but which does limit what we can now get from Apache logs.)
This makes setting up a testing Grafana server basically trivial.
We (I) install a random (virtual) machine, follow the steps to set
up Apache and Grafana as if it were our production metrics server,
and copy the production metrics server's current grafana.db to
it (while Grafana isn't running). When I restart Grafana, it will
come up with all of our dashboards and talking to our production
Prometheus data source, since it's using the public name. This
gives me a way to directly compare new versions of Grafana against
the production version, including trying to update old panel types
to new panel types and comparing the results.
(In our environment we have few enough changes to the production
grafana.db that I can just copy the file around more or less any
time I want to; I don't need to shut down the production Grafana
to save a safe copy.)
This would still be relatively simple if I'd used 'localhost:9090' as the URL for our Prometheus data source instead of its public URL, since you can change the URL of a data source. I'd just have to remember to modify the copied database that way every time I copied or re-copied it. If our Prometheus server wasn't accessible at all off the machine (either on its own or through a reverse proxy Apache), I would probably enable my testing by ssh'ing from the test server to the production server to port forward 'localhost:9090'. I couldn't leave this running all the time, but there's generally no reason to leave the test Grafana server running unless I'm specifically interested in something.
(This is far easier to do with virtual machines than with physical ones, since these days starting up and shutting them down is as simple as 'virsh start <x>' and 'virsh shutdown <x>'.)
PS: Another trick you can play with a Grafana testing server is to
keep multiple copies of your grafana.db around and swap them in
and out of being the live one depending on what you want to do. For
example, you might have one with a collection of test dashboards
(to look into things like how to display status over time in the
latest Grafana), and another that's a
current copy of your production Grafana's database.