Wandering Thoughts archives

I switched to explicit imports of things in our Django application

When I wrote our Django application it was a long time ago, I didn't know Django, and I was sort of in a hurry, so I used what I believe was the style at the time for Django of often doing broad imports of things from both Django modules and especially the application's other modules:

from django.conf.urls import *

from accounts.models import *

This wasn't universal; even at the time it was apparently partly the style to import only specific things from Django modules, and I followed that style in our code.

However, when I moved the application to Python 3 I also switched all of these over to specific imports. This wasn't required by Django (or by Python 3); instead, I did it because it made my editor complain less. Specifically it made Flycheck in GNU Emacs complain less (in my setup). I decided to do this change because I wanted to use Flycheck's list of issues to check for other, more serious issues, and because Flycheck specifically listed all of the missing or unknown imports. Because Flycheck listed them for me, I could readily write down everything it was reporting and see the errors vanish. When I had everything necessary imported, Flycheck was nicely quiet (about that).

Some of the import lines wound up being rather long (as you can imagine, the application's views.py uses a lot of things from our models.py). Even still, this is probably better for a future version of me who has to look at this code later. Some of what comes from the application models is obvious (like core object types), but not all of it; I was using some imported functions as well, and now the imports explicitly lists where they come from. And for Django modules, now I have a list of what I'm using from them (often not much), so if things change in a future Django version (such as the move from django.conf.urls to django.urls), I'll be better placed to track down the new locations and names.

In theory I could have made this change at any time. In practice, I only made it once I'd configured GNU Emacs for good Python editing and learned about Flycheck's ability to show me the full error list. Before then all of the pieces were two spread apart and too awkward for me to reach for.

(Of course, this isn't the first time that my available tools have influenced how I programmed in a way that I noticed.)

Solving one of our Django problems in a sideways, brute force way

A few years ago I wrote about an issue with propagating some errors in our Django application. We have two sources of truth for user authorization, one outside of Django (in Unix group membership that was used by Apache HTTP Basic Authentication), and one inside Django in a 'users' table; these two can become desynchronized, with someone in the Unix group but not in the application's users table. The application's 'retrieve a user record' function either returns the user record or raises an Http404 exception that Django automatically handles, which means that someone who hasn't been added to the user table will get 404 results for every URL, which isn't very friendly. I wanted to handle this by finding a good way to render a different error page in this case, either by customizing what the 'Http404' error page contained or by raising a different error.

All of this is solving the problem in the obvious way and also a cool thing to (try to) do in Django. Who doesn't want to write Python code that handles exceptional cases by, well, raising exceptions and then having them magically caught and turn into different rendered pages? But Django doesn't particularly support this, although I might have been able to add something by writing an application specific piece of Django middleware that worked by catching our custom 'no such user' exception and rendering an appropriate template as the response. However, this would have been my first piece of middleware, so I held off trying anything here until we updated to a modern version of Django (partly in the hopes it might have a solution).

Then, recently a simpler but rather less cool option to deal with this whole issue occurred to me. We have a Django management command that checks our database for consistency in various ways (for example, unused records of certain types, or people in the application's users table who no longer exist), which we run every night (from cron). Although it was a bit of a violation of 'separation of concerns', I could have that command know about the Unix group(s) that let people through Apache, and then have it check that all of the group members were in the Django user table. If people were omitted, we'd get a report. This is pretty brute force and there's nothing that guarantees that the command's list of groups stays in synchronization with our Apache configuration, but it works.

It's also a better experience for people than the cool way I was previously considering, because it lets us proactively fix the problem before people encounter it, instead of only reactively fixing it after someone runs into this and reports the issue to us. Generally, we'll add someone to the Unix group, forget to add them to Django, and then get email about it the next day before they'll ever try to use the application, letting us transparently fix our own mistake.

(This feels related to something I realized very early about not trying to do everything through Django's admin interface.)

Our Django application is now using Python 3 and a modern Django

We have a long standing Django web application to handle the process of people requesting Unix accounts here and having the official sponsor of their account approve it. For a long time, this web app was stuck on Python 2 and Django 1.10 after a failed attempt to upgrade to Django 1.11 in 2019. Our reliance on Python 2 was obviously a problem, and with the not so far off end of life of Ubuntu 20.04 it was getting more acute (we use Apache's mod_wsgi, and Ubuntu 22.04 and later don't have a Python 2 version of that for obvious reasons). Recently I decided I had to slog through the process of moving to Python 3 and a modern Django (one that is actually supported) and it was better to start early. To my pleasant surprise the process of bringing it up under Python 3 and Django 4.2 was much less work than I expected, and recently we migrated the production version. At this point it's been running long enough (and has done enough) that I'm calling this upgrade a success.

There are a number of reasons for this smooth and rapid sailing. For a start, it turns out that my 2019 work to bring the app up under Python 3 covered most of the work necessary, although not all of it. Our previous problems with CSRF and Apache HTTP Basic Authentication have either been sidestepped by Django changes since 1.11 or perhaps mitigated by Django configuration changes based on a greater understanding of this area that I worked out two years ago. And despite some grumpy things I've said about Django in the past, our application needed very few changes to go from Django 1.10 to Django 4.2.

(Most of the Django changes seem to have been moving from 'load staticfiles' to 'load static' in templates, and replacing use of django.conf.urls.url() with django.urls.re_path(), although we could probably do our URL mapping better if we wanted to. There are other minor changes, like importing functions from different places, changing request.POST.has_key(X) to X in request.POST, and defining DEFAULT_AUTO_FIELD in our settings.)

Having this migration done and working takes a real load off of my mind for the obvious reasons; neither Python 2 nor Django 1.10 are what we should really be using today, even if they work, and now we're free to upgrade the server hosting this web application beyond Ubuntu 20.04. I'm also glad that it took relatively little work now.

(Probably this will make me more willing to keep up to date with Django versions in the future. We're not on Django 5.0 because it requires a more recent version of Python 3 than Ubuntu 20.04 has, but that will probably change this summer or fall as we start upgrades to Ubuntu 24.04.)