Our Grafana and Loki installs have quietly become 'legacy software' here

May 28, 2025

At this point we've been running Grafana for quite some time (since late 2018), and (Grafana) Loki for rather less time and on a more ad-hoc and experimental basis. However, over time both have become 'legacy software' here, by which I mean that we (I) have frozen their versions and don't update them any more, and we (I) mostly or entirely don't touch their configurations any more (including, with Grafana, building or changing dashboards).

We froze our Grafana version due to backward compatibility issues. With Loki I could say that I ran out of enthusiasm for going through updates, but part of it was that Loki explicitly deprecated 'promtail' in favour of a more complex solution ('Alloy') that seemed to mostly neglect the one promtail feature we seriously cared about, namely reading logs from the systemd/journald complex. Another factor was it became increasingly obvious that Loki was not intended for our simple setup and future versions of Loki might well work even worse in it than our current version does.

Part of Grafana and Loki going without updates and becoming 'legacy' is that any future changes in them would be big changes. If we ever have to update our Grafana version, we'll likely have to rebuild a significant number of our current dashboards, because they use panels that aren't supported any more and the replacements have a quite different look and effect, requiring substantial dashboard changes for the dashboards to stay decently usable. With Loki, if the current version stopped working I'd probably either discard the idea entirely (which would make me a bit sad, as I've done useful things through Loki) or switch to something else that had similar functionality. Trying to navigate the rapids of updating to a current Loki is probably roughly as much work (and has roughly as much chance of requiring me to restart our log collection from scratch) as moving to another project.

(People keep mentioning VictoriaLogs (and I know people have had good experiences with it), but my motivation for touching any part of our Loki environment is very low. It works, it hasn't eaten the server it's on and shows no sign of doing that any time soon, and I'm disinclined to do any more work with smart log collection until a clear need shows up. Our canonical source of history for logs continues to be our central syslog server.)


Comments on this page:

Freezing at an old version is definitely something I wouldn't be able to get away with in a corporate environments with its compliance requirements for patching timelines.

We're mid-trial for Loki and VictoriaLogs here to see which will make a better replacement for an old Kibana/ElasticSearch stack.

By antiphase at 2025-05-29 19:50:44:

As someone who has recently had the joy of setting up the Grafana|Mimir|Loki coven on-premises in Kubernetes (which I also had to set up beforehand because it's the preferred/supported platform for Grafana), I can solidly recommend that no-one touches any of it with a bargepole if they can avoid it. It's weeks of my life which I won't get back.

Both K8s and the Grafana Labs suite are deceptively simple on the surface but horrifyingly complex within, and I wonder how much longer it will be possible for the majority of end users to avoid simply buying a managed cloud instance because it won't be feasible to run these platforms without the equivalent of a FTE to stay on top of management and updates of the sprawl, or if the thing that worked fine one day after a one-click install now stops doing what it once did and needs debugging.

The documentation leaves a lot to be desired too, which the cynical might take to mean they want you to pay for the Enterprise version to get support. Sadly it seems like much of the "open source" freemium stuff is going the same way.

fwiw, Alloy now integrates the Loki client into its increasingly huge monolithic executable and can read systemd journals (amongst other log things) natively, although switching the configuration from Grafana Agent to Alloy in general is an exercise in major frustration as well because the config is sort of similar but formatted so differently.

Written on 28 May 2025.
« Intel versus AMD is currently an emotional decision for me
My blocking of some crawlers is an editorial decision unrelated to crawl volume »

Page tools: View Source, View Normal.
Search:
Login: Password:

Last modified: Wed May 28 23:00:54 2025
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.