Wandering Thoughts archives

Installing Pip in Python 2 environments that don't provide it already

In theory any modern version of Python 2 (or Python 3) is bundled with pip, although it may be an out of date version that you could update (with something like 'python2 -m pip install --user --upgrade pip'). In practice, some Linux distributions split pip off into its own package and have stopped providing this separate package for their version of Python 2. This is definitely the case for Fedora 32, and may soon be the case for other distributions. If you still want a Python 2 version of Pip (for example so that you can keep updating the Python 2 version of the Python language server), you need to install one by hand, somehow.

When I had to do this on my Fedora 32 machine I was lucky enough that I had already done an update of the Python 2 pip on one machine where I used '--user' to install the new version in my $HOME, so I had all of the Pip code in .local/lib/python2.7/site-packages and could just copy it over, along with .local/bin/pip2. It turns out that this simple brute force approach is probably not necessary and there is a completely convenient alternative, which is different than the situation I expected before I started writing this entry.

(Since pip is normally installed with your Python, I expected that bootstrapping pip outside of that was not very well supported because it was infrequently used. For whatever reason, this is not at all the case currently.)

The pip people have an entire document on installing pip that walks you through a number of options. The important one for my case is Installing with get-pip.py, where you download a get-pip.py Python program to bootstrap pip. One of the options it supports is installing pip as a user package, resulting in a .local/bin/pip2 for you to use. The simple command line required is:

python2 get-pip.py --user

One of the reasons this works so well is that, well, get-pip is actually pip itself (the full version, as far as I know). The comment at the start of get-pip.py explains what is going on so well that I am just going to quote it wholesale:

Hi There!
You may be wondering what this giant blob of binary data here is, you might even be worried that we're up to something nefarious (good for you for being paranoid!). This is a base85 encoding of a zip file, this zip file contains an entire copy of pip (version 20.2.4).

Pip is a thing that installs packages, pip itself is a package that someone might want to install, especially if they're looking to run this get-pip.py script. Pip has a lot of code to deal with the security of installing packages, various edge cases on various platforms, and other such sort of "tribal knowledge" that has been encoded in its code base. Because of this we basically include an entire copy of pip inside this blob. We do this because the alternatives are attempt to implement a "minipip" that probably doesn't do things correctly and has weird edge cases, or compress pip itself down into a single file.

As a sysadmin, I fully support this very straightforward and functional approach to bootstrapping pip. The get-pip.py file that results is large for a Python program, but as installers (and executables) go, 1.9 Mbytes is not all that much.

However, there is a wrinkle probably coming up in the near future. Very soon, versions of pip itself will stop supporting Python 2; the official statement (currently here) is:

pip 20.3 was the last version of pip that supported Python 2. [...]

(The current version of pip is 20.3.3.)

The expected release date of pip 21.0 is some time this month. At some time after that point, get-pip.py may stop supporting Python 2 and you (I) will have a more difficult time bootstrapping the Python 2 version of pip on any machine I still need to add it on. Of course, at some point I will also stop having any use for a Python 2 pip, because the Python language server itself will drop support for Python 2 and I won't have any reason to upgrade my Python 2 version of it.

(Pip version 21.0 should fix, or at least work around, a long stall on startup that's experienced in some Linux configurations.)

PS: What PyPy will do about this is a good question, since they are so far planning to support Python 2 for a very long time. Perhaps they will freeze and ship pip 20.3.3 basically forever.

What you can and can't build in Go's module mode

Go modules are soon going to be our only option for building Go programs, which means that it's useful to understand what we can and can't build with Go in 'module mode', and how to do certain customary things as part of this. There are a lot of articles on using Go modules as a developer, but my major use of 'go get' and friends is to build and consume other people's programs, so that's what I'm focusing on here.

(Today in Go 1.15, you're in module mode if you use 'GO111MODULE=on' or are inside a directory tree with a go.mod. When Go 1.16 is released soon, you will be in module mode all of the time by default, and in Go 1.17 you won't even have the option to be in GOPATH mode, as covered before.)

To just install a binary for a Go program in $HOME/go/bin directly from the upstream source, you do 'go get <thing>' (provided that you're not in any source repository for a Go module). This works both for programs that are Go modules (or part of ones) and non-modular programs, even if the non-modular Go programs use third party dependencies. If there are any version tags in the source repository, you get what Go considers to be the most recent version (I believe including v2.x and so on versions). Otherwise, you get the most recent commit on the main branch. I don't believe there's any way to use this form of 'go get' to deliberately build an in-progress development version instead of a tagged release once the latter exists. If a new version is released, I believe that re-running the 'go get <thing>' will update you to the new latest version.

(There is a 'go get <thing>@latest' syntax, but it doesn't appear to do anything different than plain 'go get <thing>' in this case.)

You can also use 'go install <thing>@latest' to do this in current (and future) Go versions, which has the advantage today that it always works in module mode (or fails outright). In the future, the Go developers plan to remove 'go get's support for actually getting and building Go programs, leaving 'go install <thing>@latest' as the only option. This means people can look forward to years of annoyance from trying to follow the READMEs on old and perfectly useful Go programs (including some modularized ones).

If you have a local source repository of a Go program that's part of a Go module, you can build the current version in the repo by 'cd /where/ever; go install -mod readonly'. It's important to use '-mod readonly' unless you're actually developing the package, because otherwise Go's tooling can make changes that will cause conflicts in future VCS updates. The local source repository doesn't have to be under $HOME/go/src.

If you have a local source repository of a Go program that hasn't been modularized, it's likely that you can't build from this source repository in Go 1.16 without special steps and you won't be able to build from it in Go 1.17 at all (if the Go developers stick to their plan). In Go 1.16, because GOPATH mode remains supported, you can do non-modular builds in $HOME/go/src with 'GO111MODULE=off go get <thing>' (or 'GO111MODULE=off go install' in the right directory). If you think that the upstream will not update the program to make it a Go module, you can do 'go mod init <thing>' to create your own local go.mod and modularize the program. Modularizing the program yourself will be the only option once GOPATH mode is no longer supported at all.

(This means that it will be easier to build old un-modularized Go programs directly from Github et al than from a local copy, since 'go install ...@latest' will still work in the former case. I sure hope those upstream repositories never get removed.)

In module mode, there's no way to use 'go get' to clone the source repository of a program, whether or not it's modular. While Go still supports non-modular mode, you can force Go to clone repos into $HOME/go/src with 'GO111MODULE=off go get -d <thing>'. As far as I know there's no standard Go tool that will tell you the actual source repository and VCS used for a given Go package path, so that you can readily deal with custom import paths (a 'vanity import path, also). Perhaps there will be someday. Similarly, if you want to fetch the latest updates you must directly use an appropriate VCS command from within the source repository. This is usually 'git pull --ff-only', but there are still some people using Mercurial and other alternatives so it's up to you to keep track of it.

(If you have the source repo in the right spot under $HOME/go/src, in Go 1.16 you can force an update with 'GO111MODULE=off go get -u -d <thing>'. This will also update your local copy of any dependencies, but you probably don't care about that.)

If you were previously tracking the development of an upstream program by doing 'go get -u <thing>' periodically, you now need to do a multi step process to update and build the latest development state:

cd /where/ever || exit 0
git pull --ff-only # (probably)
go install -mod readonly

You'll also need to manually clone the repository the first time around. Although Go downloads the source code for modules, it's just the source code for whatever version you're using, not the full source repository (and Go doesn't normally expose it to you anyway).

(If you put your cloned source repositories in the right place under $HOME/go/src, you can use gostatus to check if they're out of date. Otherwise, there's 'git fetch --dry-run', although that's pretty verbose if there are updates. Perhaps someone will write or has already written a Git remote status checking program like gostatus that works on arbitrary directories.)

If you just want to periodically update to the latest released version of a program, if any, and perhaps rebuild the version with your current version of Go, I believe that 'go install <thing>@latest' will always do what you want. Further, given the module information that's embedded in binaries compiled in module mode, you can recover the necessary '<thing>' from the binary itself.

A Go binary that was built in module mode carries module information in it that can be reported by 'go version -m', which will give you the module and package of the main program. This includes non-modularized programs fetched and built directly in module mode with 'go install <thing>@latest' (or 'go get <thing>' while that still works). However, the reported information does not include the local path of the source code. If you need to get such paths once in a while, probably the simplest way today is to use Delve:

$ dlv exec someprogram
Type 'help' for list of commands.
(dlv) ls main.main
Showing <path>/<something>.go:
[....]

(As I found out when I looked into it, there's a lot of complexity in determining this information from a Go binary.)