In yesterday's entry I mentioned that I keep track of the official Prometheus releases. In a comment, Sean Conner asked how this goes along with my views on the problems inherent in building your own copies of software packages, most of which are about using your own versions, not just the specifics of compiling them. There is a story there, but the short version is that sometimes it's better to use the official upstream versions of software packages (and often to keep up to date on them). Our experience with Prometheus in our setup is a good example of this.

I described the overall situation in detail in an entry on my views on regularly upgrading Prometheus and Grafana. The short version is that we initially switched from the Ubuntu packaged versions of Prometheus components to the official project builds because the Ubuntu versions were outdated (even six months or so after 18.04's release), then we kept updating because it's basically what both projects require you to do. An especially good example of this comes from Pushgateway, where the format for its optional storage changed in a very narrow transition window; v0.10.0 was the only release that read the old format and wrote the new format, and the immediately following v1.0.0 removed support for the old format. Failing to keep up with Pushgateway releases could have given us an unpleasant surprise.

(Essentially you were intended to start v0.10.0 more or less once, to migrate the storage format, then switch to v1.0.0 or later.)

There are a number of things that make this less alarming than it looks, somewhat mitigating the problems I pointed out. First, both Prometheus and Grafana actually do provide official binary builds, and in fact it's the default way to use each project. That it's the default way means that each project has a relatively strong motivation to make good releases (and fix problems promptly), especially when combined with the development pace. When a new Grafana or Prometheus release comes out, each project knows that a lot of people will be updating to their provided binaries and they cannot sort of wash their hands of any problems that show up.

One of the reasons that Prometheus and Grafana provide their binaries and the binaries work well is that both are Go based projects and have essentially no build options. This makes it easy to produce universal binaries that will run on basically any Linux distribution (for example). More traditional projects have build options and use distribution dependent shared libraries and so on, so they couldn't produce such universal binaries without a lot more work. So they don't, and then if you're using the official releases you have to at least navigate through the build options to the set you want (and deal with the project's choice of build system).

This still leaves you to pick good releases as opposed to ones with problems, but this is mitigated by the fast pace of releases and made somewhat moot by the need to keep up with releases. With frequent new releases (and very fast bug fix ones), any serious new issues are likely to be fixed fast (and if they aren't, the odds are that the project considers them a feature and won't ever change them). This is unlike traditional open source programs, where a non-bugfix release every six months is often considered fast and even bugfix releases can take some time to be made.