2022-12-20
Detecting missing or bad Go modules and module versions
I recently wrote about the case of a half-missing Go import, where a particular version of a module had been removed upstream but was still available through the (default) Go module proxy, so things didn't really notice. This raises the interesting question of how you find this sort of thing, in all of at least three variations. Unfortunately, right now there are no great answers, but here's what I can see.
The most straightforward case is if a module has been marked as
entirely deprecated or
a module version has been marked as retracted in the module's go.mod.
Marking the latest version of your module as retracted is actually
somewhat complex; see the example in retract
directive. In both cases, you
can find out about this with 'go list -m -u all
', although it's mixed in with all
of the other modules. I think that today, you can narrow the output
down to only retracted and deprecated modules by looking for '('
in it:
go list -m -u all | fgrep '('
However, this will not help you if the upstream has removed the
module entirely, or removed a version without doing the right
retract
directive magic in go.mod. And using fgrep here counts
on the Go authors not changing the format of the output for retracted
or deprecated modules; as we've all found out in the module era,
the Go tools are not a stable interface.
(People who are very clever with Go templates may be able to craft
a 'go list -m -f
' format that will only list deprecated modules
and retracted module versions.)
To detect a missing version or module, you need to defeat two
opponents at once; the Go module proxy
and your local module cache.
Defeating the Go module proxy simple and is done by setting the
GOPROXY
environment variable to 'direct
'. There
is no direct way to disable the local module cache; instead, you
have to set the GOMODCACHE
environment variable to a new, empty
scratch directory before you run 'go mod list -m -u all
'.
This gives you:
export GOPROXY=direct export GOMODCACHE=/tmp/t-$$ go list -m -u all
This will be slow. If a module version has been removed, you'll get a relatively normal error message of 'invalid version: unknown revision v<...>', which is printed to standard error. If a module has been removed entirely, you will get a much more cryptic error that will probably complain something like 'invalid version: git ls-remote -q origin in <GOMODCACHE area>: <odd stuff>'. I'm not going to put the literal message I get here because I'm not sure anyone else would get the same one.
Afterward, you'll want to remove your temporary GOMODCACHE (which has fortunately not been created with any non-writeable directories). If you intend to check a bunch of modules or programs at once, you don't have to clean and remake your scratch GOMODCACHE between each one; it's good enough that it has relatively recent contents.
The other way to detect missing modules and module versions is to
try to build all of your programs with these GOPROXY and GOMODCACHE
environment variable settings (and afterward you'll need to do 'go
clean -modcache
' to clean out this scratch module cache, which
also removes your GOMODCACHE top level directory). If all programs
build, all of their dependencies are still there (although some may
be retracted or deprecated; 'go build
' or 'go install
' won't
warn you about that). In some CI environments
you'll start with a completely clean environment every time, with
no local module cache, so all you need to do is force 'GOPROXY=direct
'
(of course, this will slow down builds; the Go module proxy is
faster than direct fetching).
It's possible that people have already written some third party programs to do all of this for you, in a more convenient form and with nicer output than you get from the normal Go tools (and to be fair, this isn't really the job of those tools).
PS: The reason 'go build
' and 'go install
' generally won't warn
you that you're using retracted versions or entirely deprecated
modules is partly that by and large the retraction or deprecation
won't be in the go.mod
of the version of the module you're currently
using. Instead these will usually be in a later version, which Go
will only look at if you ask it to look for module updates.