2021-01-13
Installing Pip in Python 2 environments that don't provide it already
In theory any modern version of Python 2 (or Python 3) is bundled
with pip
, although it may be an out of date version that you could
update (with something like 'python2 -m pip install --user --upgrade
pip
'). In practice, some Linux distributions split pip
off into
its own package and have stopped providing this separate package
for their version of Python 2. This is definitely the case for
Fedora 32, and may soon be the case for other distributions. If you
still want a Python 2 version of Pip (for example so that you can
keep updating the Python 2 version of the Python language server), you need to install one by hand, somehow.
When I had to do this on my Fedora 32 machine I was lucky enough
that I had already done an update of the Python 2 pip on one machine
where I used '--user
' to install the new version in my $HOME, so
I had all of the Pip code in .local/lib/python2.7/site-packages and
could just copy it over, along with .local/bin/pip2. It turns out
that this simple brute force approach is probably not necessary
and there is a completely convenient alternative, which is different
than the situation I expected before I started writing this entry.
(Since pip is normally installed with your Python, I expected that bootstrapping pip outside of that was not very well supported because it was infrequently used. For whatever reason, this is not at all the case currently.)
The pip people have an entire document on installing pip that walks you through a number of options. The important one for my case is Installing with get-pip.py, where you download a get-pip.py Python program to bootstrap pip. One of the options it supports is installing pip as a user package, resulting in a .local/bin/pip2 for you to use. The simple command line required is:
python2 get-pip.py --user
One of the reasons this works so well is that, well, get-pip is
actually pip itself (the full version, as far as I know). The comment
at the start of get-pip.py
explains what is going on so well that
I am just going to quote it wholesale:
Hi There!
You may be wondering what this giant blob of binary data here is, you might even be worried that we're up to something nefarious (good for you for being paranoid!). This is a base85 encoding of a zip file, this zip file contains an entire copy of pip (version 20.2.4).Pip is a thing that installs packages, pip itself is a package that someone might want to install, especially if they're looking to run this get-pip.py script. Pip has a lot of code to deal with the security of installing packages, various edge cases on various platforms, and other such sort of "tribal knowledge" that has been encoded in its code base. Because of this we basically include an entire copy of pip inside this blob. We do this because the alternatives are attempt to implement a "minipip" that probably doesn't do things correctly and has weird edge cases, or compress pip itself down into a single file.
As a sysadmin, I fully support this very straightforward and functional approach to bootstrapping pip. The get-pip.py file that results is large for a Python program, but as installers (and executables) go, 1.9 Mbytes is not all that much.
However, there is a wrinkle probably coming up in the near future. Very soon, versions of pip itself will stop supporting Python 2; the official statement (currently here) is:
pip 20.3 was the last version of pip that supported Python 2. [...]
(The current version of pip is 20.3.3.)
The expected release date of pip 21.0 is some time this month. At some time after that point, get-pip.py may stop supporting Python 2 and you (I) will have a more difficult time bootstrapping the Python 2 version of pip on any machine I still need to add it on. Of course, at some point I will also stop having any use for a Python 2 pip, because the Python language server itself will drop support for Python 2 and I won't have any reason to upgrade my Python 2 version of it.
(Pip version 21.0 should fix, or at least work around, a long stall on startup that's experienced in some Linux configurations.)
PS: What PyPy will do about this is a good question, since they are so far planning to support Python 2 for a very long time. Perhaps they will freeze and ship pip 20.3.3 basically forever.
What you can and can't build in Go's module mode
Go modules are soon going to be our only option for building Go
programs, which means that it's useful to
understand what we can and can't build with Go in 'module mode',
and how to do certain customary things as part of this. There are
a lot of articles on using Go modules as a developer, but my major
use of 'go get
' and friends is to build and consume other people's
programs, so that's what I'm focusing on here.
(Today in Go 1.15, you're in module mode if you use 'GO111MODULE=on
'
or are inside a directory tree with a go.mod
. When Go 1.16 is
released soon, you will be in module mode all of the time by default,
and in Go 1.17 you won't even have the option to be in GOPATH mode,
as covered before.)
To just install a binary for a Go program in $HOME/go/bin directly
from the upstream source, you do 'go get <thing>
' (provided that
you're not in any source repository for a Go module). This works
both for programs that are Go modules (or part of ones) and non-modular
programs, even if the non-modular Go programs use third party
dependencies. If there are any version tags in the source repository,
you get what Go considers to be the most recent version (I believe
including v2.x and so on versions). Otherwise, you get the most
recent commit on the main branch. I don't believe there's any way
to use this form of 'go get
' to deliberately build an in-progress
development version instead of a tagged release once the latter
exists. If a new version is released, I believe that re-running
the 'go get <thing>
' will update you to the new latest version.
(There is a 'go get <thing>@latest
' syntax, but it doesn't appear to
do anything different than plain 'go get <thing>
' in this case.)
You can also use 'go install <thing>@latest
' to do this in current
(and future) Go versions, which has the advantage today that it
always works in module mode (or fails outright). In the future, the
Go developers plan to remove 'go get
's support for actually getting
and building Go programs, leaving 'go install <thing>@latest
' as
the only option. This means people can look forward to years of
annoyance from trying to follow the README
s on old and perfectly
useful Go programs (including some modularized ones).
If you have a local source repository of a Go program that's part
of a Go module, you can build the current version in the repo by
'cd /where/ever; go install -mod readonly
'. It's important to
use '-mod readonly
' unless you're actually developing the package,
because otherwise Go's tooling can make changes that will cause
conflicts in future VCS updates. The local source repository doesn't
have to be under $HOME/go/src.
If you have a local source repository of a Go program that hasn't
been modularized, it's likely that you can't build from this source
repository in Go 1.16 without special steps and you won't be able
to build from it in Go 1.17 at all (if the Go developers stick to
their plan). In Go 1.16, because GOPATH mode remains supported, you
can do non-modular builds in $HOME/go/src with 'GO111MODULE=off
go get <thing>
' (or 'GO111MODULE=off go install
' in the right
directory). If you think that the upstream will not update the
program to make it a Go module, you can do 'go mod init <thing>
'
to create your own local go.mod
and modularize the program.
Modularizing the program yourself will be the only option once
GOPATH mode is no longer supported at all.
(This means that it will be easier to build old un-modularized
Go programs directly from Github et al than from a local copy, since
'go install ...@latest
' will still work in the former case. I
sure hope those upstream repositories never get removed.)
In module mode, there's no way to use 'go get
' to clone the source
repository of a program, whether or not it's modular. While Go still
supports non-modular mode, you can force Go to clone repos into
$HOME/go/src with 'GO111MODULE=off go get -d <thing>
'. As far as
I know there's no standard Go tool that will tell you the actual
source repository and VCS used for a given Go package path, so that
you can readily deal with custom import paths (a 'vanity import
path,
also). Perhaps
there will be someday. Similarly, if you want to fetch the latest
updates you must directly use an appropriate VCS command from within
the source repository. This is usually 'git pull --ff-only
', but
there are still some people using Mercurial and other alternatives
so it's up to you to keep track of it.
(If you have the source repo in the right spot under $HOME/go/src, in
Go 1.16 you can force an update with 'GO111MODULE=off go get -u -d
<thing>
'. This will also update your local copy of any dependencies,
but you probably don't care about that.)
If you were previously tracking the development of an upstream
program by doing 'go get -u <thing>
' periodically, you now need
to do a multi step process to update and build the latest development
state:
cd /where/ever || exit 0 git pull --ff-only # (probably) go install -mod readonly
You'll also need to manually clone the repository the first time around. Although Go downloads the source code for modules, it's just the source code for whatever version you're using, not the full source repository (and Go doesn't normally expose it to you anyway).
(If you put your cloned source repositories in the right place under
$HOME/go/src, you can use gostatus
to check if they're out of
date. Otherwise, there's 'git fetch --dry-run
', although that's
pretty verbose if there are updates. Perhaps someone will write or
has already written a Git remote status checking program like
gostatus
that works on arbitrary directories.)
If you just want to periodically update to the latest released
version of a program, if any, and perhaps rebuild the version with
your current version of Go, I believe that
'go install <thing>@latest
' will always do what you want. Further,
given the module information that's embedded in binaries compiled
in module mode, you can recover the necessary '<thing>' from the
binary itself.
A Go binary that was built in module mode carries module information
in it that can be reported by 'go version -m
', which will give
you the module and package of the main program. This includes
non-modularized programs fetched and built directly in module mode
with 'go install <thing>@latest
' (or 'go get <thing>
' while
that still works). However, the reported information does not include
the local path of the source code. If you need to get such paths
once in a while, probably the simplest way today is to use Delve:
$ dlv exec someprogram Type 'help' for list of commands. (dlv) ls main.main Showing <path>/<something>.go: [....]
(As I found out when I looked into it, there's a lot of complexity in determining this information from a Go binary.)