I brought our Django app up using Python 3 and it mostly just worked

July 10, 2019

I have been worrying for some time about the need to eventually get our Django web application running under Python 3; most recently I wrote about being realistic about our future plans, which mostly amounted to not doing anything until we had to. Well, guess what happened since then.

For reasons beyond the scope of this entry, last Friday I ended up working on moving our app from Django 1.10.7 to 1.11.x, which was enlivened by the usual problem. After I had it working under 1.11.22, I decided to try running it (in development mode, not in production) using Python 3 instead of Python 2, since Django 1.11.22 is itself fully compatible with Python 3. To my surprise, it took only a little bit of cleanup and additional changes beyond basic modernization to get it running, and the result is so far fully compatible with Python 2 as well (I committed the changes as part of the 1.11 move, and since Monday they're running in production).

I don't think this is particularly due to anything I've done in our app's code; instead, I think it's mostly due to the work that Django has done to make everything work more or less transparently. As the intermediate layer between your app and the web (and the database), Django is already the place that has to worry about character set conversion issues, so it can spare you from most of those. And generally that's the big difference between Python 2 and Python 3.

(The other difference is the print statement versus 'print()', but you can make Python 2.7 work in the same way as Python 3 with 'from __future__ import print_function', which is what I did.)

I haven't thoroughly tested our web app under Python 3, of course, but I did test a number of the basics and everything looks good. I'm fairly confident that there are no major issues left, only relatively small corner cases (and then the lurking issue of how well the Python 3 version of mod_wsgi works and if there are any traps there). I'm still planning to keep us on Python 2 and Django 1.11 through at least the end of this year, but if we needed to I could probably switch over to a current Django and Python 3 with not very much additional work (and most of the work would be updating to a new version of Django).

There was one interesting and amusing change I had to make, which is that I had to add a bunch of __str__ methods to various Django models that previously only had __unicode__ methods. When building HTML for things like form <select> fields, Django string-izes the names of model instances to determine what to put in here, but in Python 2 it actually generates the Unicode version and so ends up invoking __unicode__, while in Python 3 str is Unicode already and so Django was using __str__, which didn't exist. This is an interesting little incompatibility.

Sidebar: The specific changes I needed to make

I'm going to write these down partly because I want a coherent record, and partly because some of them are interesting.

  • When generating a random key to embed in a URL, read from /dev/urandom using binary mode instead of text mode and switch from an ad-hoc implementation of base64.urlsafe_b64encode to using the real thing. I don't know why I didn't use the base64 module in the first place; perhaps I just didn't look for it, since I already knew about Python 2's special purpose encodings.

  • Add __str__ methods to various Django model classes that previously only had __unicode__ ones.

  • Switch from print statements to print() as a function in some administrative tools the app has. The main app code doesn't use print, but some of the administrative commands report diagnostics and so on.

  • Fix mismatched tabs versus spaces indentation, which snuck in because my usual editor for Python used to use all-tabs and now uses all-spaces. At some point I should mass-convert all of the existing code files to use all-spaces, perhaps with four-space indentation.

  • Change a bunch of old style exception syntax, 'except Thing, e:', to 'except Thing as e:'. I wound up finding all of these with grep.

  • Fix one instance of sorting a dictionary's .keys(), since Python 3 now returns an iterator here instead of a sortable object.

Many of these changes were good ideas in general, and none of them are ones that I find objectionable. Certainly switching to just using base64.urlsafe_b64encode makes the code better (and it makes me feel silly for not using it to start with).

Comments on this page:

Hey there Chris:

Congrat's on taking the leap and migrating successfully (it would seem?!) to Python 3!

I only just now began following your saga in that regard, and just discovered your blog, after someone posted a link to it today on Reddit.

I have to say your blog is amazingly clear and concisely written article, loaded with tons of useful info, especially for newbies like me to gain insights into C programming, Python, and maintaining a Unix environment.

So you've certainly gained at least 1 new fan of your blog today (although I'm guessing you probably have several fans who follow these great posts of yours).

Anyways, looking forward to reading more of your blog entries, and also see how this migration to Python 3 holds up! (Crossing my fingers for you!)

- Jack AKA: user/Orbital_Dynamics

By Daniel Harding at 2019-07-14 21:10:20:

It sounds like you've done the work already, but for anyone who comes across this page, Django provides a decorator, @python_2_unicode_compatible that given a model class with a __str__()​ method that returns a text value (​str under Python 3, unicode under Python 2) will leave the class unchanged under Python 3, but under Python 2 will rename the __str__()​ method to __unicode__()​ and generate a __str__()​ method which returns an encoded version of the value returned by __unicode__()​. This avoids needing to have both __str__()​ and __unicode__()​ methods for your Django models.

By Ben Hutchings at 2019-07-26 22:06:20:

You could now use os.urandom() instead of opening /dev/urandom yourself. It will use the getrandom() system call if available, which may be good or bad depending on whether this might run before the Linux RNG decides it's collected enough entropy.

Written on 10 July 2019.
« Systemd services that always restart should probably set a restart delay too
Reflections on almost entirely stopping using my (work) Yubikey »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Jul 10 21:46:22 2019
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.