Go modules and the problem of noticing updates to dependencies

September 6, 2019

Now that Go 1.13 has been released, we're moving that much closer to a module-based Go world. I've become cautiously but broadly positive towards Go 1.13 and this shift (somewhat in contrast to what I expected earlier), and I'm probably going to switch over to Go 1.13 everywhere and move toward modules in my own work. Thinking about working in this environment has left me with some questions.

Let's suppose that you have some programs or code that uses third party packages, and these are generally stable programs that don't really need any development or change. In the Go module world, the version of those packages that you use is locked down by your go.mod file and won't change unless you manually update, even if new versions are released. In theory you can keep on using your current versions forever, but in practice as a matter of good maintenance hygiene you probably want to update every so often to pick up bug fixes, improvements, and perhaps security updates. As always, updating regularly also makes the changes smaller and easier to deal with if there are problems.

In the pre-module world, how I found out about such updates was that I ran Go-Package-Store every so often and looked at what it reported (I could also use gostatus). I also had (and have) tools like binstale and gobin, which I could use with scripting to basically 'go get -u' everything I currently had a Go binary for (which makes some of the problems from my old entry on using Go-Package-Store not applicable any more).

I'm not sure how to do this in a world of Go modules. Go-Package-Store works by scanning your $GOPATH, but with modules the only things there are (perhaps) your actual programs, not their dependencies. You can see updates for the dependencies of any particular program or module with 'go list -u -m all' (in a cryptic format; anything with a '[...]' after it has an update to that version available), but I don't think anyone has built anything to do a large scale scan, try to find out what the changes are, and show them to you.

(The current module behavior of 'go get', 'go list', and company also seems surprising to me in some areas that complicate interpreting 'go list -u -m all' output, although perhaps it's working as intended.)

Relying on Go modules also brings up a related issue of what to do if the upstream source just goes away. In the pre-module world, you have a full VCS clone of the upstream in your $GOPATH/src, so you can turn around and re-publish it somewhere yourself (or someone else can and you know you can trust their version because it's the same as your local copy). In the module world you only have a snapshot of a specific version (or versions) in your $GOPATH/pkg/mod tree or in the Go module proxy you're using. Even if you vendor things as well, you're not going to have the transparent and full version history of the original package that you do today, and the lack of that history will make it harder to do various things to recover from a disappearing or abruptly changed package.

(I'm a cautious sysadmin who has been around for a while. I've seen all sorts of repositories just disappear one day for all sorts of different reasons.)

(Perhaps someday there will be a Go module proxy that deliberately makes a full VCS clone when you request a module.)

Comments on this page:

By Simon Pasquier at 2019-09-06 03:11:20:

For things that are hosted on GitHub, I've found that Renovate (more precisely the fork variant [1]) is doing a good job. There's also Dependabot [2] which has been acquired recently by GitHub but it requires more permissions to run and has less knobs (eg it doesn't deal with the vendor/ directory).

[1] https://github.com/apps/forking-renovate
[2] https://dependabot.com/

By Bert at 2019-09-06 12:45:18:

This sure sounds like a problem that every other package product has solved, like Microsoft NuGet, npm's registry, pypi, etc.

Is this a problem only because there is no single central "authority" hosting the go module repository?

By Greg A. Woods at 2019-09-08 16:24:59:

Yeah, go list -m -u is definitely more complicated than it should be, and it doesn't have the obvious "do it" option to turn the results into a set of go get commands and run them.

I would have thought go mod vendor should have cloned each dependency from its original repository and done the appropriate checkout, but it only seems to make a copy from the cache.

Also missing is some way to find out what you're getting when you upgrade a dependency before you get it -- especially important if you don't have a local clone of the module's source repository. There needs to be some way of finding out what sub-dependencies are updated/downgraded when some module is updated/downgraded, and then some way to get a summary of changes for each module too, e.g. a diff of the NEWS file if it exists, or a whole diff, etc.

By cks at 2019-09-09 11:06:04:

Greg: I think that vendoring has always only copied the current source into your tree, not cloned the whole repo as a sub-repo. I agree about lacking a good way to do a mass update and to see changes; for the latter, I suspect that the Go people would say 'go read the VCS changelog on the web or make a full clone'. 'go list -u -m all' sort of gives you the URL for it, and in general Go now seems surprisingly against doing VCS operations such as clones by default.

(I cannot entirely blame them; I'm pretty sure there are security implications of routinely cloning potentially untrusted repos in a general environment.)

Bert: I don't think this is an issue of central authorities; it's more an issue of having a central list of everything you're using across all of your programs. Once you have that list, you can query either a central or a distributed source of information, but without that list you've got to iterate through all of your programs and systems and run a 'list dependencies with updates' operation for each of them.

Written on 06 September 2019.
« If you use the rarfile module, make sure you're using version 3.0 (or later)
Programs that let you jump around should copy web browser navigation »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Sep 6 00:20:35 2019
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.