How I recently used vendoring in Go

August 29, 2018

Go 1.11 comes with experimental support for modules, which are more or less Russ Cox's 'vgo' proposal. Initial versions of this proposal were strongly against Go's current feature of vendor directories and wanted to completely replace them. Later versions seem to have toned that down, but my impression is that the Go people still don't like vendoring very much. I've written before about my sysadmin's perspective on vendoring and vgo, where I wanted something that encapsulated all of the build dependencies in a single directory tree that I could treat as a self-contained artifact and that didn't require additional daemons or complicated configuration to use. However, I recently used vendoring for another case, one where I don't think Go's current module support would have worked as well.

For reasons beyond the scope of this entry, I wanted a program that counted up how many files (well, inodes) were used in a directory hierarchy, broken down by the sub-hierarchy; this is essentially the file count version of my standard du based space usage breakdown (where I now use this more convenient version). Since this basically has to manipulate strings in a big map, writing it in Go was a natural decision. Go has filepath.Walk in the standard library, but for extra speed and efficiency I turned to godirwalk. Everything worked great right up until I tried to cross-compile my new program for Solaris (okay, Illumos, but it's the same thing as far as Go is concerned) so I could run it on our fileservers. That's when I found out that godirwalk doesn't support Solaris, ultimately for the reason that Solaris doesn't support file type information in directory entries.

I was able to hack around this with some effort, but the result is a private, modified version of godirwalk that's only ever going to be used by my dircount program, and then only for as long as I care about running dircount on OmniOS (when I stop caring about that, dircount can use the official version). I definitely don't want this to be the apparent official version of godirwalk in my $GOPATH/src hierarchy, and this is not really something that Go modules can solve easily. Traditional Go vendoring solves it neatly and directly; I just put my hacked up version of godirwalk in vendor/, where it will be automatically used by dircount and not touched by anything else (well, provided that I build dircount in my $GOPATH). When or if I don't want to build with my hacked godirwalk, I can rename the vendor directory temporarily and run 'go build'.

(According to the current documentation, the closest I could come with modules is to replace the official godirwalk with my own version that I would have to set up in some tree somewhere. This replacement would be permanent until I edited go.mod; I couldn't switch back and forth easily.)

This isn't a use I'd initially thought of for vendoring, but in retrospect it's an obvious one. Vendoring makes convenient private copies of packages; normally you use this to freeze package versions, but you can just as well use this to apply and freeze your own modifications. Probably I'll run into other cases of this in the future.

(I will elide a discussion of whether this sort of local change to upstream packages is a good idea or whether you should really rename them into a new package name (and thus Go modules are forcing you to do the right thing).)

Written on 29 August 2018.
« An illustration of why it's hard to port outside code into the Linux kernel
Our problem with (Amanda) backups of many files, especially incrementals »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Aug 29 00:16:31 2018
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.