Wandering Thoughts

The question of how long Python 2 will be available in Linux distributions

In theory Python 2 is now dead (sort of). In practice we have a significant number of Python 2 scripts and programs (including a Django web app, probably like many other places. Converting these to Python 3 is a make-work project we want to avoid, especially since it risks breaking important things that are working fine. One big and obvious issue for keeping on using Python 2 is that we use Linux (primarily Ubuntu) and normally use the system version of Python 2 in /usr/bin (although we're starting to shift how we invoke it). This obviously only works as long as there is a packaged version installed in /usr/bin, which raises the question of how long this will be available in Linux distributions.

Linux distributions like Debian, Ubuntu, and Fedora want to move away from officially supporting Python 2 because there is no upstream support (RHEL 8 will be supporting their version through 2024 or so). Of these, Debian and by extension Ubuntu have an idea of less-core packages that are built and maintained by interested parties, and I suspect that there will be people interested in packaging Python 2 even past when it stops being a fully supported package. Failing that being accepted, Ubuntu has the idea of PPAs, which people can use to distribute easily usable packages for Ubuntu (we get Certbot through a PPA, for example). Fedora doesn't quite have the packaging split that Debian does, but it has a PPA-like thing in COPR (also). I suspect that there is sufficient interest in Python 2 that people will provide PPAs and COPR repos for it.

(At the extreme end of things, we can and will build our own package of Python 2 if necessary by just re-building the last available version from a previous distribution release. We wouldn't get the ecology of additional Python 2 .debs or RPMs, but we don't need those.)

As far as I can tell, the current state of Python 2 in Fedora 32 is that Python 2.7 has become a legacy package as part of Fedora's plans to retire Python 2. The current Fedora plans have no mention of removing the 2.7 legacy Python package, but given how Fedora moves I wouldn't be surprised to see calls for that to happen in a few years (which would be inconvenient for me; a few years is quite fast here). Alternately, it might linger quietly for half a decade or more if it turns out to require no real work on anyone's part.

I expect Debian and Ubuntu to move more slowly than this but in the same direction. Ubuntu 20.04 may not be able to drop all packages that depend on Python 2.7, but by Ubuntu 22.04 I expect that work to be done and so Python 2 itself could be and probably will be demoted to a similar legacy status. Since 2022 is only two years away and Debian is not the fastest moving organization when it comes to controversial things like removing Python 2 entirely, I suspect that discussion of removing Python 2 itself will start no earlier than for Ubuntu 24.04. However I can't find a Debian or Ubuntu web page that talks about their future plans for Python 2 itself in any detail, so we may get surprised.

PS: In our environment, the issue with Python 2 going away (or /usr/bin/python changing which Python version it points to) isn't just our own system maintenance programs, but whatever Python programs our users may have written and be running. We likely have no real way to chase those down and notify the affected users, so any such shift would be very disruptive, especially because we run multiple versions of Ubuntu at once. With different Ubuntu versions on different machines, what /usr/bin/python gets you could vary from one machine to another. At that point we might be better off removing the name entirely; at least things with '#!/usr/bin/python' would fail immediately and clearly.

Sorting out the dates of Python 2's 'end of life'

Probably like many people, I've been hearing for years now that January of 2020 was the end of life for Python 2, specifically January 1st. As a result, I was rather surprised to hear that there will be another release of Python 2 in April, although I could have read the actual details in PEP 373 and avoided this.

The official dates, from PEP 373, are:

Planned future release dates:

• 2.7.18 code freeze January, 2020
• 2.7.18 release candidate early April, 2020
• 2.7.18 mid-April, 2020

What this actually means is not clear to me, given the four month delay between the code freeze (now) and the planned release of even a 2.7.18 release candidate (April). At a minimum, I assume that the code freeze blocks new features, should anyone want to submit any. I suspect that the Python people would not accept fixes for new bugs or for existing bugs that did not have some version of a fix accepted before the code freeze. I assume that Python developers will still accept fixes for accepted bugfixes, if testing shows that any have problems.

(If Python isn't going to accept changes into what will be released as 2.7.18 for any reason at all, they might as well release tomorrow instead of in four months.)

Although the details are set out in PEP 373, this way of describing Python 2's end of life is a little bit unusual and different from what I (and likely other people) expected from an 'End of Life' date. The usual practice with EOL dates is that absolutely nothing will be released beyond that point, not that main development stops and then a final release will be made some time later.

(This is what Linux distributions do, for example; the EOL date for a distribution release is when the last package updates will come out. I believe it's similar for the BSD Unixes.)

It's very unclear to me how Linux distributions (and the BSDs) are likely to handle Python 2 versions in light of this. At least some of them will still be packaging Python 2 in versions released beyond April of 2020. They might freeze their Python 2 version on the current 2.7.17 (or whatever they already have), or upgrade to 2.7.18 as one last Python 2 (re-)packaging.

Filenames and paths should be a unique type and not a form of strings

I recently read John Goerzen's The Fundamental Problem in Python 3, which talks about Python 3's issues in environments where filenames (and other things) are not in a uniform and predictable encoding. As part of this, he says:

[...]. Critically, most of the Python standard library treats a filename as a String – that is, a sequence of valid Unicode code points, which is a subset of the valid POSIX filenames.

[...]

From a POSIX standpoint, the correct action would have been to use the bytes type for filenames; this would mandate proper encode/decode calls by the user, but it would have been quite clear. [...]

This is correct only from a POSIX standpoint, and then only sort of (it's correct in traditional Unix filesystems but not necessarily all current ones; some current Unix filesystems can restrict filenames to properly encoded UTF-8). The reality of modern life for a language that wants to work on Windows as well as Unix is that filenames must be presented as a unique type, not any form of strings or bytes.

How filenames and paths are represented depends on the operating system, which means that for portability filenames and paths need to be an opaque type that you have to explicitly insert string-like information into and extract string-like information out of, specifying the encoding if you don't want an opaque byte sequence of unpredictable contents. As with all encoding related operations, this can fail in both directions under some circumstances.

Of course this is not the Python 3 way. The Python 3 way is to pretend that everything is fine and that the world is all UTF-8 and Unicode. This is pretty much the pragmatically correct choice, at least if you want to have Windows as a first class citizen of your world, but it is not really the correct way. As with all aspects of its handling of strings and Unicode, Python 3 chose convenience over reality and correctness, and has been patching up the resulting mess on Unix since its initial release.

If Python was going to do this correctly, Python 3 would have been the time to do it; since it was breaking things in general, it could have introduced a distinct type and required that everything involving file names change to taking and returning that type. But that would have made porting Python 2 code harder and would have made it less likely that Python 3 was accepted by Python programmers, which is probably one reason it wasn't done.

(I don't think it was the only one; early Python 3 shows distinct signs that the Python developers had more or less decided to only support Unix systems where everything was proper UTF-8. This turned out to not be a viable position for them to maintain, so modern Python 3 is somewhat more accommodating of messy reality.)

It's unfortunately time to move away from using '/usr/bin/python'

For a long time, the way to make Python programs runnable on Unix has been to start them with '#!/usr/bin/python' or sometimes '#!/usr/bin/env python' (and then chmod them executable, of course; this makes them scripts). Unfortunately this is no longer a good idea for general Python programs, for the simple reason that current Unixes now disagree on what version of Python is '/usr/bin/python'. Instead, we all need to start explicitly specifying what version of Python we want by using '/usr/bin/python3' or '/usr/bin/python2' (or by having env explicitly run python3 or python2).

For a long time, even after Python 3 came out, it seemed like /usr/bin/python would stay being Python 2 in many environments (ones where you had Python 2 and Python 3 installed side by side). I expected a deprecation of /usr/bin/python as Python 2 to take years after Python 2 itself was no longer supported, for the simple reason that there are a lot of programs and instructions out there that expect their '#!/usr/bin/python' or 'python' to run Python 2. Changing what that meant seemed reasonably disruptive, even if it was the theoretically correct and pure way.

In reality, as I recently found out, Fedora 31 switched what /usr/bin/python means, and apparently Arch Linux did it several years ago. In theory PEP 394 describes the behavior here and this behavior is PEP-acceptable. In practice, before early July of 2019, PEP 394 said that 'python' should be Python 2 unless the user had explicitly changed it or a virtual environment was active. Then, well, there was a revision that basically threw up its hands and said that people could do whatever they wanted to with /usr/bin/python (via).

(This makes PEP 394 a documentation standard. As with all documentation standards, it needs to describe reality to be useful, and the reality is that /usr/bin/python is now completely unpredictable.)

Since Fedora and Arch Linux have led the way here, other Linux distributions will probably follow. In particular, since Red Hat Enterprise is more or less based on Fedora, I wouldn't be surprised to see RHEL 9 have /usr/bin/python be Python 3. I don't think Debian and thus Ubuntu will be quite this aggressive just yet, but I wouldn't be surprised if in a couple of years /usr/bin/python at least defaults to Python 3 on Ubuntu. (Hopefully Python 2 will still be available as a package.)

Timing durations better in Python (most of the time)

As I mentioned in yesterday's entry on timeouts and exceptions, we have some Python programs that check to make sure various things are working, such as that we can log in to our IMAP servers. Since they're generating Prometheus metrics as part of this, one of the metrics they generate is how long it took.

(My feeling is that if you're going to be generating metrics about something, you should always include timing information just on general principles even if you don't have an immediate use for it.)

Until recently, my code generated this duration information in the obvious way:

stime = time.time()
do_some_check(....)
dur = time.time() - stime

There is nothing significant wrong with this and most of the time it generates completely accurate timing information. However, it has a little drawback, which is that it's using the current time of day (in general, the clock time). If the system's time gets adjusted during the check, the difference between the starting time of day and the ending time of day doesn't accurately reflect the true duration of the check.

A generally better way to measure durations is to use monotonic time, obtained through time.monotonic(), which the operating system promises can never go backwards and isn't affected by changes in the system's clock. Most of the time the two are going to be the same or very close (sufficiently close that if you're using Python, you probably don't care about the timing difference). But sometimes monotonic time will give you a true answer when clock time will be noticeably off.

The one limitation of monotonic time is that in some environments, monotonic time doesn't advance when the system is suspended, as might happen with a laptop or a virtual machine. Monotonic time is the time as the operating system experiences it, but not necessarily absolute time passing. If you need absolute time passing and you care about the system being suspended, you may have to use clock time and hope for the best.

(In my environment this code is running on a server that should never suspend or pause, so monotonic time is perfect.)

PS: time.monotonic dates back to Python 3.5 or before, so it should be pretty safe to use everywhere (fortunately for us, Ubuntu 16.04 is just recent enough to have Python 3.5). People still using Python 2 are out of luck; consider it another reason to write new code in Python 3 instead.

Thinking about timeouts and exceptions in Python

As part of our overall monitoring and metrics system, I have a Python program that logs in to our IMAP server to make sure that we can at least get that far (because we have broken that sort of thing in the past). The program emits various Prometheus metrics, including how long this took. For reasons beyond the scope of this entry, I would also like to have some very basic information on how the IMAP server is performing, such as how long it takes to do an IMAP SELECT operation on the test account's IMAP inbox. Trying to do a clean implementation has run me into issues surrounding handling timeouts.

Like any sensible program that checks a system that may have problems, my program has an overall timeout on how long it will talk to the IMAP server. In Python, the straightforward way to implement an overall timeout is with signal.setitimer and a SIGALRM signal handler that raises an exception. When you're only doing one (conceptual) thing, this is straightforward to implement by wrapping your operation in a try/except block:

try:
   signal.setitimer(signal.ITIMER_REAL, timeout)
   metrics = login_check(host, user, pw)
   signal.setitimer(signal.ITIMER_REAL, 0)
   report_success(host, metrics)
except Timeout:
   report_failure(host)

Either we finish within the timeout interval, in which case we report the timing and other metrics we generated, or we fail and we report timeout failure metrics.

This simple approach breaks down once I want to report separate metrics and success statuses for two different operations. Timing out while trying to log in to the IMAP server is quite different (and more severe) than successfully logging in to the IMAP server but timing out during the IMAP SELECT operation. Since I realized this (while I was writing the new code), I've been trying to work out the right structure to make the code natural and clean.

The theoretical clean abstraction that I think I want is that once the timeout is hit, this is recorded and all further network IO (or more generally, IMAP protocol operations) fail immediately. If this was how it worked, both the IMAP login attempt and the IMAP SELECT would report success or failure depending on where things were when the timeout happened, and I could report a 'there was a timeout' metric at the end. This would also extend very naturally to doing a series of IMAP operations (for example, SELECT'ing several different mailboxes and collecting timings on each). The code could just generate metrics in a straight line fashion and everything would work out. Unfortunately Python's network code and imaplib don't provide a straightforward way to do this, so I would have to build a layer on top of imaplib to do it for me.

(This approach is inspired by Go's network package, which supports something like this. But even then it's not quite as clean as it looks, because ideally you want every check to be aware of the possibility of timeouts, so that it distinguishes a real network error from 'we hit the time limit and my network operations started failing'.)

My current approach is to keep more or less explicit track of how far I got by what metrics have been generated, and then fill in any missing metrics with failure markers:

login_metrics = None
select_metrics = None
try:
  signal.setitimer(signal.ITIMER_REAL, timeout)
  login_metrics, conn = login_check(host, user, pw)
  # logging in may have failed
  if conn:
    select_metrics = select_check(conn, host, "INBOX")
    # Let's ignore logging out for now
  signal.setitimer(signal.ITIMER_REAL, 0)
  did_timeout = False
except Timeout:
  did_timeout = True

if not login_metrics:
  login_metrics = failed_login(host, user)
if not select_metrics:
  select_metrics = failed_select(host, "INBOX")

report_metrics(login_metrics, select_metrics)
report_maybe_timeout(host, did_timeout)

This approach works, but it has a scaling problem; if I add more IMAP operations, I have to add code in several places and I'd better not miss one. This isn't very generic and it feels like there should be a better way. On the other hand, this code is at least explicit and free of magic; it may be brute force, but it's straightforward to follow. A high-magic approach is probably not the right one for a program that I touch at most once every six months.

(Some of my problems may be because of how I generate almost all login metrics in one function, which was a clever idea in the original code but perhaps isn't any more. I'm not sure what would be a better approach, though.)

This approach also puts all timeout handling in one place, at the top level, instead of forcing all of the individual operations to be aware of the possibility that they will hit a timeout (or be invoked after the timeout has triggered, and so that's why all of their IMAP operations are failing). It may be that this is the best option for code structure, especially in Python where exceptions are how we deal with many global concerns.

(This is related to phase tracking for better error reporting. In a sense, what my current code is doing is tracking 'phase' in a collection of variables.)

How we structure our Django web application's configuration settings

We have a relatively long-standing Django web application, first written for Django 1.2. That was a long time ago and I didn't really know what I was doing, so when I began the web application I did what Django more or less suggests. I ran 'django-admin startproject' to set up the initial directory structure and files, including the settings.py that contains various configuration settings for the application, and then started editing that generated settings.py to add all of the customizations we needed. This turned out to be a little bit of a mistake.

The problem is that what Django wants you to have in settings.py and how it's been structured has varied over Django versions; the settings.py from Django 1.2 is not what you'd get (and not really what you want) if you re-ran 'django-admin startproject' today. Because I had mingled our local customizations into settings.py, I couldn't just have Django re-generate it and replace our old Django 1.2 version with the new, up to date one that would be set up like Django 1.9 wanted. So when I redid our settings.py for Django 1.9, I restructured how it was set up to put many of our settings into a separate file (or actually a cascade of them).

(I also made a little mistake by accidentally dropping a setting we needed because I hadn't commented it.)

My current approach is that the only things that go in settings.py itself are minimal modifications to Django settings that are already specified there, especially ones that are structured and ordered data, like INSTALLED_APPS, MIDDLEWARE, and TEMPLATES. Our modifications to all of these are commented with explicit markers, so I can find them when I need to propagate them into a new version of settings.py. All of our other settings go into a separate file, ((cslabsettings.py), which is pulled into settings.py in an addition at the end:

# CSLAB:
# This is where our actual settings live.
from app.cslabsettings import *

This has all of our settings for our application, and also some Django settings that aren't set in settings.py and so don't have to be modified there, such as ADMINS (cf). Because we pull this into settings.py, all of the usual Django things just work without special configuration.

We then have a layer or two of additional settings files on top of this. For development, I have a settings_dev.py that imports (settings.py)) and then overrides bits and pieces (for example, to set 'DEBUG=true'). To use it, I have a manage_dev.py which is just the standard manage.py modified to use settings_dev. I also have a settings_wsgi.py, which contains settings that are specific to running under WSGI instead of in management commands, and a settings_wsgi_dev.py that is for the obvious thing.

(Now that I look at settings_wsgi.py, it's possible that it has too many settings and should be trimmed back. The problem with having a whole cascade of settings files is that it's easy to lose track of what is set in what and what still needs to be overridden in the latest one.)

This is much the same idea as my ancient modular approach to Apache configuration, and for the same reason. The Django generated settings.py is pretty much a system supplied file, which means that periodically you want to replace it with a new system supplied version. The less you change in it, the less work you have to do when this happens.

(This entry is the long delayed explanation for the little bit at the end of this entry.)

Remembering that Django template code is not quite your Django Python code

Recently I was doing some work with our Django web application where I wanted to do something special for one particular field of a form. As is usual in Django forms, we iterate through the form fields with the '{% for %}' template tag. Stripped down, this is something like:

{% for field in form %}
  {{ field.label_tag }} <br>
  {{ field }}
{% endfor %}

I had to make one field generate extra HTML, which meant that I had to recognize it somehow. At first I used a '{% if %}' on the label, but that felt wrong; the label is text that's shown to the user and we might want to change it. So I did some Internet searches and found 'field.name' (perhaps from here), which is the actual name of each field. I put it in, everything worked, I went on, and then a few days later I was struck by a Python question, namely how did the form field know its name?

Form fields are defined as what looks like class attributes:

class RequestForm(forms.Form):
  login = forms.CharField([...])
  [...]

However, Python objects don't know their names (for good reasons), including attributes on classes or class instances. At first I thought that Django was using metaclass magic so that RequestForm.login.name was set up when the RequestForm class was defined, but form classes turn out to be more magic than that and once you extract the relevant objects, they don't have .name attributes.

At this point I realized (and remembered) that the pseudo-Python code you write in Django templates is not the same as the Python code you write in your app. You can't assume that what works for one of them (here, referring to 'field.name') will work for the other, and it goes both ways. It's possible to rip the covers off the magic and find an explanation for what Django is doing, but it will lead you several levels deep and your Django template code is still not the same as what you'll write in your Python view code.

PS: One bit of magic that is showing already in my example is that field.label_tag is a method, not an attribute. The Django template language automatically calls it and uses the result.

Sidebar: What is going on here

The 'field' template variable being used here winds up holding a real Python object, but it is not from the class you think it is. As covered in Looping over the form's fields, the 'field' object is a BoundField instance. The actual Field instance, what your view code deals with, is hiding in field.field. BoundField does have a .name attribute, which is set up when it's initialized by the overall form code. The overall form code does know the name of each field (and has to).

(Somewhat to my surprise, .field is a plain ordinary instance attribute, not any sort of property or fancy thing. You can see it being set up in boundfield.py.)

Some magical weirdness in Django's HTML form classes

In Django, HTML forms are defined in what seems to be the conventional approach; each form is a Python class and fields are variables defined in the class definition. This looks like this, to crib a skeleton from our Django web application :

class RequestForm(forms.Form):
  login = forms.CharField([...])
  name  = forms.CharField([...])
  email = forms.EmailField([...])
  aup_agree = forms.BooleanField([...])

You instantiate an instance of RequestForm either with initial data (when you're displaying the form for the first time) or with data from the POST or GET form, and then call various standard Django API functions on it and so on. It's all covered in the Django documentation.

What I didn't remember until I started looking just now is that this Python class definition we wrote out is kind of a lie. Django model classes mostly work look like they look, so for example the model version of a Request has a Request.login attribute just as its class definition does. Forms are significantly different. Although we set up what looks like class attributes here, our actual RequestForm class and class instances do not have, say, a RequestForm.login attribute. All of the form fields we seemed to define here get swept up and put in Form.fields.

At one level this is documented and probably the safest option, given that the data ultimately comes from an untrusted source (ie, a HTTP request). It also means that you mostly can't accidentally use a form instance as a model instance (for example, by passing the wrong thing to some function); if you try, it will blow up with attribute errors.

(The 'mostly' is because you can create a login attribute on a RequestForm instance if you write to it, so if a function that writes to fields of a model instance is handed a form instance by accident, it may at least half work.)

At another level, this is another way that Django's classes are non-Python magic. What looks like class attributes aren't even properties; they've just vanished. Conventional Python knowledge is not as much use for dealing with Django as it looks, and you have to know the API (or look it up) even for things that look like they should be basic and straightforward.

(I don't have an opinion any more about whether the tradeoffs here are worth it. Our Django app just works, which is what we really care about.)

CentOS 7 and Python 3

Over on Twitter, I said:

Today I was unpleasantly reminded that CentOS 7 (still) doesn't ship with any version of Python 3 available. You have to add the EPEL repositories to get Python 3.6.

This came up because of a combination of two things. The first is that we need to set up CentOS 7 to host a piece of commercial software, because CentOS 7 is the most recent Linux release it supports. The second is that an increasing number of our local management tools are now in Python 3 and for various reasons, this particular CentOS 7 machine needs to run them (or at least wants to ) when our existing CentOS 7 machines haven't. The result was that when I set up various pieces of our standard environment on a newly installed CentOS 7 virtual machine, they failed to run because there was no /usr/bin/python3.

At one level this is easily fixed. Adding the EPEL repositories is a straightforward 'yum install epel-release', and after that installing Python 3.6 is 'yum install python36'. You don't get a pip3 with this and I'm not sure how to change that, but for our purposes pip3 isn't necessary; we don't install packages system-wide through PIP under anything except exceptional circumstances.

(The current exceptional circumstances is for Tensorflow on our GPU compute servers. These run Ubuntu 18.04, where pip3 is available more or less standard. If we had general-use CentOS 7 machines it would be an issue, because pip3 is necessary for personal installs of things like the Python LSP server.)

Even having Python 3.6 instead of 3.7 isn't particularly bad right now; our Ubuntu 16.04 machines have Python 3.5.2 and even our 18.04 ones only have 3.6.8. Even not considering CentOS 7, it will be years before we can safely move any of our code past 3.6.8, since some of our 18.04 machines will not be upgraded to 20.04 next year and will probably stay on 18.04 until early 2023 when support starts to run out. This is surprisingly close to the CentOS 7 likely end of life in mid 2024 (which is much closer than I thought before I started writing this entry), so it seems like CentOS 7 only having Python 3.6 is not going to hold our code back very much, if at all.

(Hopefully by 2023 either EPEL will have a more recent version of Python 3 available on CentOS 7 or this commercial software will finally support CentOS 8. I can't blame them for not supporting RHEL 8 just yet, since it's only been out for a relatively short length of time.)

PS: I don't know what the difference is between the epel-release repositories you get by doing it this way and the epel-release-latest repositories you get from following the instructions in the EPEL wiki. The latter repos still don't seem to have Python 3.7, so I'm not worrying about it; I'm not very picky about the specific version of Python 3.6 I get, especially since our code has to run on 3.5 anyway.