Wandering Thoughts archives

Python 2, Apache's mod_wsgi, and its future in Linux distributions

Sometimes I have small questions about our future with Python 2, instead of big ones. Our Django web application currently runs in Apache using mod_wsgi, and the last time we attempted a Django upgrade (which is a necessary step in an upgrade to Python 3), it didn't go well. This means that we may wind up caring quite a bit about how long Ubuntu and other Linux distributions will package a version of mod_wsgi that still supports Python 2, instead of just Python 3 (assuming that the Linux distribution even provides Python 2 at all).

Fedora 31 currently still provides a Python 2 sub-package of mod_wsgi, but this should be gone in Fedora 32 since it naturally depends on Python 2 and all such (sub-)packages are supposed to be purged. Debian's 'unstable' also currently seems to have the Python 2 version of mod_wsgi, but it's included in Debian's list of Python 2 related packages to be removed (via), so I suspect it will be gone from the next stable Debian release.

(Debian is also getting rid of Python 2 support for uwsgi, which could be another way of running our WSGI application under Apache.)

What Ubuntu 20.04 will look like is an interesting question. Right now, the in-progress state of Ubuntu 'focal' (what will be 20.04) includes a libapache2-mod-wsgi package using Python 2. However, this package is listed in Ubuntu's list of Python 2 related packages to remove (via). Ubuntu could still remove the package (along with others), or it could now be too close to the release of 20.04 for the removal to be carried through by then.

(I believe that Ubuntu usually freezes their package set a decent amount of time before the actual release in order to allow for testing, and perhaps especially for LTS releases. I may be wrong about this, because the Ubuntu Focal Fossa release schedule lists the Debian import freeze as quite late, at the end of February.)

Even if the Python 2 version of mod_wsgi manages to stay in Ubuntu 20.04 LTS (perhaps along with other Python 2 WSGI gateways), it will definitely be gone by the time of Ubuntu 22.04, which is when we'd normally upgrade the server that currently hosts our Django web app. So by 2022, we need to have some solution for our Python 2 problem with the app, whatever it is.

Why a network connection becoming writable when it succeeds makes sense

When I talked about how Go deals with canceling network connection attempts, I mentioned that it's common for the underlying operating system to signal you that a TCP connection (or more generally a network connection) has been successfully made by letting it become writable. On the surface this sounds odd, and to some degree it is, but it also falls out of what the operating system knows about a network connection before and after it's made. Also, in practice there is a certain amount of history tied up in this particular interface.

If we start out thinking about being told about events, we can ask what events you would see when a TCP connection finishes the three way handshake and becomes established. The connection is now established (one event), and you can generally now send data to the remote end, but usually there's no data from the remote end to receive so you would not get an event for that. So we would expect a 'connection is established' event and a 'you can send data' event. If we want a more compact encoding of events, it's quite tempting to merge these two together into one event and say that a new TCP connection becoming writable is a sign that its three way handshake has now completed.

(And you certainly wouldn't expect to see a 'you can send data' event before the three way handshake finishes.)

The history is that a lot of the fundamental API of asynchronous network IO comes from BSD Unix and spread from there (even to non-Unix systems, for various reasons). BSD Unix did not use a more complex 'stream of events' API to communicate information from the kernel to your program; instead it used simple and easy to implement kernel APIs (because this was the early 1980s). The BSD Unix API was select(), which passes information back and forth using bitmaps; one bitmap for sending data, one bitmap for receiving data, and one bitmap for 'exceptions' (whatever they are). In this API, the simplest way for the kernel to tell programs that the three way handshake has finished is to set the relevant bit in the 'you can send data' bitmap. The kernel's got to set that bit anyway, and if it sets that bit and also sets a bit in the 'exceptions' bitmap it needs to do more work (and so will programs; in fact some of them will just rely on the writability signal, because it's simpler for them).

Once you're doing this for TCP connections, it generally makes sense for all connections regardless of type. There are likely to be very few stream connection types where it makes sense to signal that you can now send (more) data partway through the connection being established, and that's the only case where this use of signaling writability gets in the way.