Wandering Thoughts archives

2021-04-23

How I want to use pip with PyPy to install third party programs

After seeing PyPy run a moderate program faster than CPython, I wondered how easy it would be to use PyPy to run pyls (which I use for LSP-based editing for Python in GNU Emacs), since pyls is reasonably big and probably reasonably CPU intensive. Pyls is a third party program that's not packaged by Ubuntu but which is available through PyPI, so normally I install it with 'pip3 install --user ...' so that it goes in $HOME/.local/bin.

I'll start with my conclusion: for PyPy, I want to use a virtualenv to install and manage third party programs (at least for the version and setup of PyPy that Ubuntu packages). I'm not normally much of a fan of virtualenvs for various reasons and I'd avoid them here if I could, but using a virtualenv is less annoyance and more likely to succeed than trying to get a pip for PyPy to coexist with pip for CPython. Perhaps you can make it work with enough sweat, but it's a lot easier to shrug, make a virtualenv or two, and accept 30 MBytes of overhead per virtualenv.

You definitely want one virtualenv for PyPy 2 and a second one for PyPy 3. I think you can put all of the third party commands you want into the single virtualenv rather than having to have one virtualenv for pyls, one for YAPF, and so on. To set up my virtualenvs using the Ubuntu version of PyPy, I followed the PyPy documentation for this:

virtualenv -p /usr/bin/pypy3 $HOME/lib/pypy3-env
virtualenv -p /usr/bin/pypy $HOME/lib/pypy2-env

The Ubuntu packaged version of virtualenv is a Python 3 program, but it still works to set up a Python 2 PyPy virtualenv. This is probably routine to people who're familiar with it, but I'm not.

Once your virtualenvs are set up, you can start installing things with the virtualenv's pip as usual:

$HOME/lib/pypy3-env/bin/pip install python-language-server

(For pyls, you'll need the PyPy development packages to be installed. On Ubuntu 20.04 these are pypy-dev and pypy3-dev.)

You don't need to activate the virtualenv to run commands from it; as I found out earlier, virtual environments transparently add themselves to sys.path. I'm not sure what maintenance you'll need to do to a virtualenv when PyPy changes versions (or changes what version of Python 3 it claims to be). I'll probably get to find out someday.

Even if your system version of Python 2 doesn't package and supply pip (Fedora now doesn't ship it), your virtualenv appears to magically get it and it works. I don't quite know how this works (although I'm sure I could find out if I dug into it), but I'm happy with the result since it's quite convenient.

(Our Ubuntu 18.04 machines have no standard package for PyPy 3, but that's another issue. Perhaps we'll be able to switch our user login machines over to 20.04 this summer.)

PyPyPipForOutsidePrograms written at 00:25:53; Add Comment

2021-04-20

PyPy starts fast enough for our Python 2 commands

Some day, Linux distributions like Ubuntu are not going to package Python 2 even in a limited version, the way they're doing now. One way for us to deal with this would be to migrate all of our remaining little Python 2 programs and scripts to Python 3. Another option is to run them under PyPy, which says that it will always support Python 2.7.

One of the potential issues with PyPy is that its JIT has a high warm-up cost, which means that small, short-running programs are going to be slower, perhaps significantly slower. Most of the Python 2 that we have left is in small administrative commands that are mostly run automatically, where on the one hand I would expect PyPy's overhead to be at its largest and on the other hand we probably don't really care about the overhead if it's not too big. So I decided to do some quick tests.

(I've been hit by the startup overhead of small programs in Python even without PyPy, but it was in an interactive situation.)

I did my tests on one of our Ubuntu 20.04 servers, which has PyPy version 7.3.1, and the results turned out to be more interesting than I expected. The artificial and irrelevant worst case was a Python 3 program that went from about 0.05 second to about 0.17 second (under pypy3) to actually do its work. Our typical small Python 2 commands seem to go from 0.01 or 0.02 second to about 0.07 second or so. The surprising best case was a central program used for managing our password file, where the runtime under PyPy actually dropped from around 0.40 second to 0.33 second. And a heavily multithreaded program that runs a lot of concurrent ssh commands had essentially the same runtime on a different 20.04 machine.

(In retrospect, the password file processing program does have to process several thousand lines of text, so perhaps I should not have been surprised that it's CPU-intensive enough for PyPy to speed it up. Somehow it's in my mind as a small, lightweight thing.)

All of this says that PyPy starts (and runs) our Python programs more than fast enough to serve us as an alternate implementation of Python 2 if we need to turn to it.

PyPyCommandStartupTime written at 23:13:43; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.