Wandering Thoughts archives

Detecting missing or bad Go modules and module versions

I recently wrote about the case of a half-missing Go import, where a particular version of a module had been removed upstream but was still available through the (default) Go module proxy, so things didn't really notice. This raises the interesting question of how you find this sort of thing, in all of at least three variations. Unfortunately, right now there are no great answers, but here's what I can see.

The most straightforward case is if a module has been marked as entirely deprecated or a module version has been marked as retracted in the module's go.mod. Marking the latest version of your module as retracted is actually somewhat complex; see the example in retract directive. In both cases, you can find out about this with 'go list -m -u all', although it's mixed in with all of the other modules. I think that today, you can narrow the output down to only retracted and deprecated modules by looking for '(' in it:

go list -m -u all | fgrep '('

However, this will not help you if the upstream has removed the module entirely, or removed a version without doing the right retract directive magic in go.mod. And using fgrep here counts on the Go authors not changing the format of the output for retracted or deprecated modules; as we've all found out in the module era, the Go tools are not a stable interface.

(People who are very clever with Go templates may be able to craft a 'go list -m -f' format that will only list deprecated modules and retracted module versions.)

To detect a missing version or module, you need to defeat two opponents at once; the Go module proxy and your local module cache. Defeating the Go module proxy simple and is done by setting the GOPROXY environment variable to 'direct'. There is no direct way to disable the local module cache; instead, you have to set the GOMODCACHE environment variable to a new, empty scratch directory before you run 'go mod list -m -u all'. This gives you:

export GOPROXY=direct
export GOMODCACHE=/tmp/t-$$
go list -m -u all

This will be slow. If a module version has been removed, you'll get a relatively normal error message of 'invalid version: unknown revision v<...>', which is printed to standard error. If a module has been removed entirely, you will get a much more cryptic error that will probably complain something like 'invalid version: git ls-remote -q origin in <GOMODCACHE area>: <odd stuff>'. I'm not going to put the literal message I get here because I'm not sure anyone else would get the same one.

Afterward, you'll want to remove your temporary GOMODCACHE (which has fortunately not been created with any non-writeable directories). If you intend to check a bunch of modules or programs at once, you don't have to clean and remake your scratch GOMODCACHE between each one; it's good enough that it has relatively recent contents.

The other way to detect missing modules and module versions is to try to build all of your programs with these GOPROXY and GOMODCACHE environment variable settings (and afterward you'll need to do 'go clean -modcache' to clean out this scratch module cache, which also removes your GOMODCACHE top level directory). If all programs build, all of their dependencies are still there (although some may be retracted or deprecated; 'go build' or 'go install' won't warn you about that). In some CI environments you'll start with a completely clean environment every time, with no local module cache, so all you need to do is force 'GOPROXY=direct' (of course, this will slow down builds; the Go module proxy is faster than direct fetching).

It's possible that people have already written some third party programs to do all of this for you, in a more convenient form and with nicer output than you get from the normal Go tools (and to be fair, this isn't really the job of those tools).

PS: The reason 'go build' and 'go install' generally won't warn you that you're using retracted versions or entirely deprecated modules is partly that by and large the retraction or deprecation won't be in the go.mod of the version of the module you're currently using. Instead these will usually be in a later version, which Go will only look at if you ask it to look for module updates.