Wandering Thoughts


Please don't try to hot-reload changed Python files too often

There is a person running a Python program on one of our servers, which is something that people do regularly. As far as I can tell, this person's Python program is using some Python framework that supports on the fly reloading (often called hot-reloading) of changed Python code for at least some of the loaded code, and perhaps much or all of it. Naturally, in order to see if you need to hot-reload any code, you need to check whether a bunch of files have changed (at least in our environment, some environments may be able to do this slightly better). This person's Python code is otherwise almost always idle.

The particular Python code involved has decided to check for a need to hot-reload code once every second. In our NFS fileserver environment, this has caused one particular fileserver to see a constant load of about 1100 NFS RPC operations a second, purely from the Python hot-reload code rechecking what appears to be a pile of things every second. These checks are also not cheap on the machine where the code is running; this particular process routinely uses about 7% to 8% of one CPU as it's sitting there otherwise idle.

(There was a time when you didn't necessarily care about CPU usage on otherwise idle machines. In these days of containerization and packing multiple services on one machine and renting the smallest and thus cheapest VPS you can get away with, there may be no such thing as a genuinely idle machine, and all CPU usage is coming from somewhere.)

To be fair, it's possible that the program is being run in some sort of development mode, where fast hot-reload can be potentially important. But people do run 'development mode' in more or less production, and it's possible to detect that. It would be nice if hot-reload code made some efforts to detect that, and perhaps also some efforts to detect when things were completely idle and there had been no detected changes for a long time and it should dial back the frequency of hot-reload checks. But I'm probably tilting at windmills.

(I also think that you should provide some sort of option to set the hot-reload frequency, because people are going to want to do this sooner or later. You should do this even if you only try to do hot reloading in development mode, because sooner or later people are going to run your development mode in pseudo-production because that's the easiest way for them.)

PS: These days this also applies to true development mode usage of things. People can easily step away from their development environment for meetings or whatever, and they may well be running it on their laptop, where they would like you to not burn up their battery constantly. Just because someone has a development mode environment running doesn't mean they're actively using it right now.

HotReloadingNotTooOften written at 22:08:52; Add Comment


Platform peculiarities and Python (with an example)

I have a long standing little Python tool to turn IP addresses into verified hostnames and report what's wrong if it can't do this (doing verified reverse DNS lookups is somewhat complicated). Recently I discovered that socket.gethostbyaddr() on my Linux machines was only returning a single name for an IP address that was associated with more than one. A Fediverse thread revealed that this reproduced for some people, but not for everyone, and that it also happened in other programs.

The Python socket.gethostbyaddr() documentation doesn't discuss specific limitations like this, but the overall socket documentation does say that the module is basically a layer over the platform's C library APIs. However, it doesn't document exactly what APIs are used, and in this case it matters. Glibc on Linux says that gethostbyaddr() is deprecated in favour of getnameinfo(), so a C program like CPython might reasonably use either to implement its gethostbyaddr(). The C gethostbyaddr() supports returning multiple names (at least in theory), but getnameinfo() specifically does not; it only ever returns a single name.

In practice, the current CPython on Linux will normally use gethostbyaddr_r() (see Modules/socketmodule.c's socket_gethostbyaddr()). This means that CPython isn't restricted to returning a single name and is instead inheriting whatever peculiarities of glibc (or another libc, for people on Linux distributions that use an alternative libc). On glibc, it appears that this behavior depends on what NSS modules you're using, with the default glibc 'dns' NSS module not seeming to normally return multiple names this way, even for glibc APIs where this is possible.

Given all of this, it's not surprising that the CPython documentation doesn't say anything specific. There's not very much specific it can say, since the behavior varies in so many peculiar ways (and has probably changed over time). However, this does illustrate that platform peculiarities are visible through CPython APIs, for better or worse (and, like me, you may not even be aware of those peculiarities until you encounter them). If you want something that is certain to bypass platform peculiarities, you probably need to do it yourself (in this case, probably with dnspython).

(The Go documentation for a similar function does specifically say that if it uses the C library it returns at most one result, but that's because the Go authors know their function calls getnameinfo() and as mentioned, that can only return one name (at most).)

PythonAndPlatformPeculiarities written at 22:53:03; Add Comment


I switched to explicit imports of things in our Django application

When I wrote our Django application it was a long time ago, I didn't know Django, and I was sort of in a hurry, so I used what I believe was the style at the time for Django of often doing broad imports of things from both Django modules and especially the application's other modules:

from django.conf.urls import *
from accounts.models import *

This wasn't universal; even at the time it was apparently partly the style to import only specific things from Django modules, and I followed that style in our code.

However, when I moved the application to Python 3 I also switched all of these over to specific imports. This wasn't required by Django (or by Python 3); instead, I did it because it made my editor complain less. Specifically it made Flycheck in GNU Emacs complain less (in my setup). I decided to do this change because I wanted to use Flycheck's list of issues to check for other, more serious issues, and because Flycheck specifically listed all of the missing or unknown imports. Because Flycheck listed them for me, I could readily write down everything it was reporting and see the errors vanish. When I had everything necessary imported, Flycheck was nicely quiet (about that).

Some of the import lines wound up being rather long (as you can imagine, the application's views.py uses a lot of things from our models.py). Even still, this is probably better for a future version of me who has to look at this code later. Some of what comes from the application models is obvious (like core object types), but not all of it; I was using some imported functions as well, and now the imports explicitly lists where they come from. And for Django modules, now I have a list of what I'm using from them (often not much), so if things change in a future Django version (such as the move from django.conf.urls to django.urls), I'll be better placed to track down the new locations and names.

In theory I could have made this change at any time. In practice, I only made it once I'd configured GNU Emacs for good Python editing and learned about Flycheck's ability to show me the full error list. Before then all of the pieces were two spread apart and too awkward for me to reach for.

(Of course, this isn't the first time that my available tools have influenced how I programmed in a way that I noticed.)

DjangoExplicitImportsSwitch written at 21:50:01; Add Comment


Solving one of our Django problems in a sideways, brute force way

A few years ago I wrote about an issue with propagating some errors in our Django application. We have two sources of truth for user authorization, one outside of Django (in Unix group membership that was used by Apache HTTP Basic Authentication), and one inside Django in a 'users' table; these two can become desynchronized, with someone in the Unix group but not in the application's users table. The application's 'retrieve a user record' function either returns the user record or raises an Http404 exception that Django automatically handles, which means that someone who hasn't been added to the user table will get 404 results for every URL, which isn't very friendly. I wanted to handle this by finding a good way to render a different error page in this case, either by customizing what the 'Http404' error page contained or by raising a different error.

All of this is solving the problem in the obvious way and also a cool thing to (try to) do in Django. Who doesn't want to write Python code that handles exceptional cases by, well, raising exceptions and then having them magically caught and turn into different rendered pages? But Django doesn't particularly support this, although I might have been able to add something by writing an application specific piece of Django middleware that worked by catching our custom 'no such user' exception and rendering an appropriate template as the response. However, this would have been my first piece of middleware, so I held off trying anything here until we updated to a modern version of Django (partly in the hopes it might have a solution).

Then, recently a simpler but rather less cool option to deal with this whole issue occurred to me. We have a Django management command that checks our database for consistency in various ways (for example, unused records of certain types, or people in the application's users table who no longer exist), which we run every night (from cron). Although it was a bit of a violation of 'separation of concerns', I could have that command know about the Unix group(s) that let people through Apache, and then have it check that all of the group members were in the Django user table. If people were omitted, we'd get a report. This is pretty brute force and there's nothing that guarantees that the command's list of groups stays in synchronization with our Apache configuration, but it works.

It's also a better experience for people than the cool way I was previously considering, because it lets us proactively fix the problem before people encounter it, instead of only reactively fixing it after someone runs into this and reports the issue to us. Generally, we'll add someone to the Unix group, forget to add them to Django, and then get email about it the next day before they'll ever try to use the application, letting us transparently fix our own mistake.

(This feels related to something I realized very early about not trying to do everything through Django's admin interface.)

DjangoSolvingProblemSideways written at 21:44:04; Add Comment


Our Django application is now using Python 3 and a modern Django

We have a long standing Django web application to handle the process of people requesting Unix accounts here and having the official sponsor of their account approve it. For a long time, this web app was stuck on Python 2 and Django 1.10 after a failed attempt to upgrade to Django 1.11 in 2019. Our reliance on Python 2 was obviously a problem, and with the not so far off end of life of Ubuntu 20.04 it was getting more acute (we use Apache's mod_wsgi, and Ubuntu 22.04 and later don't have a Python 2 version of that for obvious reasons). Recently I decided I had to slog through the process of moving to Python 3 and a modern Django (one that is actually supported) and it was better to start early. To my pleasant surprise the process of bringing it up under Python 3 and Django 4.2 was much less work than I expected, and recently we migrated the production version. At this point it's been running long enough (and has done enough) that I'm calling this upgrade a success.

There are a number of reasons for this smooth and rapid sailing. For a start, it turns out that my 2019 work to bring the app up under Python 3 covered most of the work necessary, although not all of it. Our previous problems with CSRF and Apache HTTP Basic Authentication have either been sidestepped by Django changes since 1.11 or perhaps mitigated by Django configuration changes based on a greater understanding of this area that I worked out two years ago. And despite some grumpy things I've said about Django in the past, our application needed very few changes to go from Django 1.10 to Django 4.2.

(Most of the Django changes seem to have been moving from 'load staticfiles' to 'load static' in templates, and replacing use of django.conf.urls.url() with django.urls.re_path(), although we could probably do our URL mapping better if we wanted to. There are other minor changes, like importing functions from different places, changing request.POST.has_key(X) to X in request.POST, and defining DEFAULT_AUTO_FIELD in our settings.)

Having this migration done and working takes a real load off of my mind for the obvious reasons; neither Python 2 nor Django 1.10 are what we should really be using today, even if they work, and now we're free to upgrade the server hosting this web application beyond Ubuntu 20.04. I'm also glad that it took relatively little work now.

(Probably this will make me more willing to keep up to date with Django versions in the future. We're not on Django 5.0 because it requires a more recent version of Python 3 than Ubuntu 20.04 has, but that will probably change this summer or fall as we start upgrades to Ubuntu 24.04.)

DjangoAppNowPython3 written at 23:06:25; Add Comment


Putting a Python executable in venvs is probably a necessary thing

When I wrote about getting the Python LSP server working with venvs in a brute force way, Ian Z aka nobrowser commented (and I'm going to quote rather than paraphrase):

I'd say that venvs themselves are "aesthetically displeasing". After all, having a separate Python executable for every project differs from having a separate LSP in degree only.

On Unix, this separate executable is normally only a symbolic link, although other platforms may differ and the venv normally will have its own copy of pip, setuptools, and some other things, which can amount to 20+ Mbytes even on Linux. However, when I thought about it, I don't think there's any good option other than for the venv to have its own (nominal) copy of Python. The core problem is that venvs are very convenient when they're more or less transparently activated.

A Python venv is marked by a special file in the root of the venv, pyvenv.cfg. There are two ways that Python could plausibly decide when to automatically activate a venv without you having to set any environment variables; it can look around the environment of the Python executable you ran for this marker (which is what it does today), or it could look around the environment of your current directory, traversing up the filesystem to see if it could find a pyvenv.cfg (in much the same way that version control systems look for their special .git or .hg directory to mark the repository root).

The problem with automatically activating a venv based on what you find in the current directory and its parents is that it makes Python programs (and the Python interpreter) behave differently depending on where you are when you run them, including random system utilities that just happen to be written in Python. If the program requires any packages beyond the standard library, it may well fail outright because those packages aren't installed in the venv, and if they are installed in the venv they may not be the version the program needs or expects. This isn't a particularly good experience and I'm pretty confident that people would be very unhappy if this was what Python did with venvs.

The other option is to not automatically activate venvs at all and always require you to set environment variables (or the local equivalent). The problem for this is that it's a terrible experience for actually using venvs to, for example, deploy programs as encapsulated entities. You can't just ship the venv and have people run programs that have been installed into its bin/ subdirectory; now they need cover scripts to set the venv environment variables (which might be automatically generated by pip or whatever, but still).

So on the whole embedding the Python interpreter seems the best choice to me. That creates a clear logic to which venv is automatically activated, if any, that can be predicted by people; it's the venv whose Python you're running. Of course I wish it didn't take all of that disk space for extra copies of pip and setuptools, but you can't have everything.

VenvsAndEmbeddedPython written at 21:28:13; Add Comment


Getting the Python LSP server working with venvs the brute force way

Recently I wound up doing some Django work using a Python venv, since this is the easiest way to get a self-contained Python environment that has some version of Django (or other applications) installed. However, one part of the experience was a little bit less than ideal. I normally write Python using GNU Emacs and the Python LSP server, and this environment was complaining about being unable to find Django modules to do code intelligence things with them. A little thought told me why; GNU Emacs was running my regular LSP server, which is installed through pipx into its own venv, and that venv didn't have Django installed (of course).

As far as I know, the Python LSP server doesn't have any specific support for recognizing and dealing with venvs, and in general this is a bit difficult (the version of Python being used by a venv may not even match the version that pylsp is running with; in fact this was my case, since my installed pylsp was using pypy but the Django venv was using the system Python 3). Rather than try to investigate deeply into this, I decided to solve the problem with brute force, which is to say that I installed the Python LSP server (with the right set of plugins) into the venv, along with all of the rest of things, and then ran that instance of GNU Emacs with its $PATH set to use the venv's bin/ directory and pick up everything there, including its Python 3 and python-lsp-server.

This is a little bit aesthetically displeasing for at least two reasons. First, the Python LSP server and its plugins and their dependencies aren't a small thing and anyway they're not a runtime package dependency, it's purely for development convenience. Second, the usual style of using GNU Emacs is to start it once and then reuse that single Emacs instance for everything, which naturally gives that Emacs instance a single $PATH and make it want to use a single version of python-lsp-server. I'm okay with deviating from this bit of Emacs practice, but other people may be less happy.

(A hack that deals with the second issue would be a 'pylsp' cover script that hunts through the directory tree to see if you're running it from inside a venv and if that venv has its own 'pylsp' binary; if both are true, you run that pylsp instead of your regular system-wide one. I may write this hack someday, partly so that I can stop having to remember to add the venv to my $PATH any time I want to fire up Emacs on the code I'm working on in the venv.)

PythonVenvAndLSPServer written at 21:50:05; Add Comment


A Django gotcha with Python 3 and the encoding of CharFields

We have a long standing Django web application, which has been static for a long time in Python 2. I've recently re-started working on moving it to Python 3 (an initial round of work was done some time ago), and in the process of this I ran into a surprising issue involving text encoding and database text fields (a 'CharField' in Django terminology).

As part of one database model, we have a random key, which for various reasons is represented in the database model as a modest size string:

class Request(models.Model):
  access_hash = models.CharField(..., default=gen_random_hash)

(We use SQLite as our database, which may be relevant here.)

The actual access hash is a 64-bit random value read from /dev/urandom. We could represent this in a variety of ways; for instance, I could have just treated it as a 64-bit unsigned decimal number in string form, or a 64-bit (unsigned) hex number. But for no particularly strong reason, long ago I decided to base64 encode the raw random value. Omitting error checking, the existing version of this is:

def gen_random_hash():
  fp = open("/dev/urandom", "rb")
  c = fp.read(8)
  # Trim off trailing awkward '=' character
  return base64.urlsafe_b64encode(c)[:-1]

(The use of "rb" as the open mode stems from my first round of updates for Python 3.)

When I ran our web application under Python 3 in testing mode and looked at the uses of this access hash, I discovered that the URLs we were generating for it in email included a couple of ''' in them. Inspection of the database table entry itself in the Django admin interface showed that the actual value for access hash was, for example (and this is literal):


(0x27 is ', so I was getting "b'm5AWGlUSR1c&#x27" in the URL when written out for email in HTML. Once I looked I could see that giveaway leading 'b' too.)

There are two things going on here. The first is that in Python 3, base64.urlsafe_b64encode operates on bytes (which we're giving it since we read /dev/urandom in binary mode, making c a bytes object) and returns a bytes object, not a string. The second thing is that when we ask Django to store a bytes object in a CharField (possibly only as the result of a callable default value), Django string-ized it, yielding the b'...' form as the stored value.

At one level this is reasonable; Django is doing its best to store some random Python object into a string field, probably by just doing a str() on it. At another level, I wish Django would specifically refuse to do this conversion for bytes objects, because one Python 3 issue is definitely bytes/str confusions and this specific representation conversion is almost certainly a bad idea, unlike str()'ing things in general. Raising an exception by default would be much more useful.

The solution is to explicitly convert to a Unicode str and specify a suitable character set encoding, which for base64 can be 'ascii':

return str(base64.urlsafe_b64encode(c)[:-1], "ascii")

This causes the CharField values to look like they should, which means that URLs using the access hash no longer have ''' in them.

Hopefully there aren't any other cases of this lurking in our Django web application, but I suppose I should do some more testing and examine the database for alarming characters (which is relatively readily done with the management dumpdata command).

DjangoPython3FieldEncodingGotcha written at 22:36:51; Add Comment


Using argparse in my Python programs encourages me to add options to them

Today, for reasons outside the scope of this entry, I updated one of my very old little Python utilities to add some features and move it to Python 3 in the process. This program was so old it was using getopt, so as part of updating it I switched it over to argparse, which is what I use in all of my modern programs. The change was positive in a variety of ways, but one of the things it did was it immediately caused me to add some more command line options. This isn't due to anything specific to this program, because over and over again I've had much the same experience; my argparse based programs have more options (and often better structured ones).

In thinking about it I believe there's a couple of reasons for this. First, argparse makes it easy to add the basic support for an option in a single place, in a ArgumentParser.add_argument() call. Unlike with getopt's much more primitive argument handling, I don't have to modify code in several places just to have my program accept the option, handle it to set or record some value, and include it in usage and help. Even if the option does nothing this is an easy first step.

Second, that argparse generates an 'args' (or 'options') object with all of the parsed information on it often makes it easy to expose new options to the code that needs to look at them. My usual pattern with argparse is to pass the 'opts' object you get from ArgumentParser.parse_args() down to functions that get called to do the work, rather than break out specific flags, options, and so on separately. This means that a new option is usually pervasively available through the code and doesn't have to be passed in specifically; I can just check 'opts.myNewOption' wherever I want to use it. By contrast, getopt didn't create a natural options object so I tended to pass around options separately (or worse, set global variables because it was the easy lazy way).

This doesn't always work; sometimes I need the new option in a place where I haven't already passed in the opts object. But it works a lot of the time, and when it works it makes it that much easier to code up the use of the new option.

(This also means it's easy to reverse out of an option I decide that I don't want after all. I can delete the creation of it and the use of it without having to go through changing function arguments around. Similarly it's easy to change how an option works because that too doesn't touch many places in the code.)

ArgparseEncouragesOptions written at 22:33:51; Add Comment


My first Django database migration worked fine

We have a long standing Django application, using SQLite as the database layer because that's easy and all we need at our tiny scale. For as long as we've had the application I've not so much as breathed on its database schema (which is to say its model), because the thought of trying to do any sort of database migration was a bit scary. For reasons outside the scope of this entry, we recently decided that it was time to add some fields to our application model, so I got to (or had to) try out Django's support for more or less automatic database migrations. The whole experience was pretty painless in my simple case, although I had some learning experiences.

The basics of Django migrations are well covered by Django's documentation. For straightforward changes, you change your model(s) and then run the 'makemigrations' Django subcommand, supplying the name you want for your migration; Django will write a new Python file to do the work. Once you've made the migration you can then apply it with the 'migrate' subcommand, and in at least some cases un-apply it again. Our changes added simple fields that could be empty, which is about the simplest case you can get; I don't know how Django handles more complicated cases, for example introducing a mandatory non-null field.

One learning experience is that Django will want to create a additional new migrations if you tinker with the properties of your (new) model things after you've created (and probably applied) the initial migration in your development environment. For me, this happened relatively naturally as I was writing Django code to use these new fields, now that we had them. You probably don't want to wind up with a whole series of iterating migrations, so you're going to want to somehow squash these down to a single final migration for the production change. Since we use SQLite, I wound up just repeatedly removing the migration file and reverting my development database to start again, rather than tinker around with un-applying the migration and trying to get Django to rewrite it.

(Now that I'm reading the documentation all the way through there's a section on Squashing migrations, but it seems complex and not necessarily quite right for this case.)

While it's not strictly a migration issue as such, one thing that I initially forgot to do when I added new model fields was to also update the settings for our Django admin interface to display them. Partly this happened because it's been so long since I touched this code that I'd somewhat forgotten how it all worked until I looked at the admin interface, didn't see these fields in the particular model object, and started digging.

Although the Django migration documentation contains scary warnings about doing migrations with a SQLite database, I had no problem doing it in production with our generally barely used web application (we normally measure activity in 'hits per month'). Given how Django does database migrations on SQLite, a sufficiently active web application might have to shut down during the migration so that the database could be fiddled with without problems. In general, shutting down will avoid a situation where your running code is incompatible with the state of your database, which can definitely happen even in simple changes.

(Some people probably deploy such a database migration in multiple steps, but with a tiny application we did it the single step way, deploying a new version of the code that used the new fields at the same time as we performed the database migration on the production database.)

My overall experience is quite positive; changing our model was nowhere near as scary or involved as I expected it to be. I suspect I'm going to be much more willing to make model changes in the future, although I still don't think of it as a casual thing.

DjangoDBMigrationWentOkay written at 23:12:51; Add Comment

(Previous 10 or go back to October 2023 at 2023/10/11)

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.