Wandering Thoughts archives

I've decided I'll port DWiki to Python 3, but not any time soon

At this point I have only one significant Python 2 program that I care deeply about and that is DWiki, the pile of code that underlies Wandering Thoughts. What to do about DWiki in light of Python 3 has been something that has been worrying and annoying me for some time, because doing a real port (as opposed to a quick bring-up experiment) is going to involve a lot of wrestling with old code and Unicode conversion issues. Recently I've come around to a conclusion about what I plan to do about the whole issue (perhaps an obvious one).

In the end neither grimly staying on Python 2 forever or rewriting DWiki in something else (eg in Go) are viable plans, which leaves me with the default: sooner or later I'll port DWiki to Python 3. However I don't expect to do this any time soon, for two reasons. The first is that Python 3 itself is still being developed and in fact the Python landscape as a whole is actively evolving. As a result I'd rather defer a port until things have quieted down and gotten clearer in a few years (who knows, perhaps I'll want to explicitly revise DWiki to be PyPy-friendly by then). As far as I'm concerned the time to port to Python 3 is when it's gotten boring, because then I can port once and not worry about spending the next few years keeping up with exciting improvements that I'd like to revise my code to take advantage of.

The second reason is more pragmatic but is related to the rapid rate of change in Python 3, and it is that the systems I want to run DWiki on are inevitably going to be behind the times on Python 3 versions. Right now, the rapid rate of improvements in Python 3 means that being behind the times leaves you actively missing out on desirable things. In a few years hopefully that will be less so and a Python 3 version that was frozen a year or three ago will not be so much less attractive than a current version. This too is part of Python 3 slowing down and becoming boring.

(If you are saying 'who freezes Python 3 versions at something a few years old?', you haven't looked at long term support Linux distributions or considered how long people will run eg older FreeBSD versions. There is a long and slow pipeline from the latest Python 3 release to when it appears in OS distributions that many people are using, as I've covered before.)

I don't have any particular timeline on DWiki's Python 3 port except that I don't intend or expect to do this within, oh, the next three years. Probably I'll start looking at this seriously about the time the Python developers start clearing their throats and trying to once again persuade everyone that 2.7 support will be dropped soon, this time for sure. A clear slowdown in Python 3 development plus OS distros catching up to current versions might push that to sooner, but probably not much sooner.

Hopefully thinking through all of this and writing it down means that I can stop worrying about DWiki's future every so often. I may not be doing anything about it, but at least I now have a reasonable plan (and I've kind of made my peace with the idea of going through all the effort to get a production quality version of DWiki running under Python 3 (and yes, the amount of effort it's going to take still irritates me and probably always will)).

(Although every so often I toy with the idea of a from-scratch rewrite of DWiki in Go that addresses various things I'd do differently this time around, the reality is that DWiki's creation took place in unusual circumstances that I'm unlikely to repeat any time soon.)

Tweaking code when I'm faced with the urge to replace it entirely

The one of the core parts of DWiki (the program behind all of what you're reading) is the code that turns DWikiText into HTML (and in many ways it is the most important component, since it's what ultimately renders all my content and all the comments). I spent a significant chunk of today tweaking a test version in an attempt to improve the conversion process for a number of corner cases that I care about and would like to make better.

There are two problems with this. The first is that one of the consequences of having what is now a long-running block is that I have a lot of content written in my wikitext dialect and pretty much all of it had better keep coming out just the same. I have no interest in trying to go through all of my entries to revise them for some bright new wikitext idea; backwards compatibility is quite important, warts and all. This is unfortunate in practice because I made some mistakes in my wikitext dialect way back when. It's sometimes possible to do little things around the corner of these mistakes, but that creates hacks and special rules and special magic code.

The other problem is that DWiki the program is also old and by now rather tangled, especially in the DWikiText renderer. Part of this tangle is just history, part of it is that it has been heavily optimized for speed, and part of it is that I made some fundamental structural mistakes in the beginning that have been carried through ever since then (one of them is not parsing to an AST, but it's not the only one). Faced with a complex set of code that I don't work with regularly, I descend to tweaking as carefully as I can rather doing anything deeper, which of course builds up the accumulated layers of hacks in the code and makes it harder to do anything except more tweaks and hacks around the edges.

(Python makes it surprisingly easy to do this sort of tweaking for various reasons, including its support for optional function arguments.)

At the same time, the more I worked on this code today the more clearly I saw how I wanted a modern version of the code to work. The more I have to stick in hacks and make tweaks, the more I also want to raze the whole complicated mess to the ground and redo it from scratch with a much better restructured version (or at least what I think would be such a thing, in my current state of not having written it and thus not having been faced by any messy gaps in my current grand vision).

(Any grand rewrite immediately starts to run into Python 3 thoughts, which lead to other thoughts I'm not going to try to cover here.)

I don't have any answers and I'm not even sure I'm going to deploy the tweaked version (I have a history of this sort of indecision), although I probably will now that I've written this up. But at least I had a reasonably enjoyable time fiddling around in the depths of DWiki once again and perhaps a bit more impetus towards doing some significant cleanups someday.

(While in the past I've lamented that I don't have a test suite for DWiki, I do actually have one for this sort of change; I can render all of this thing in both the old and new versions of the code and see what's different. This can be an interesting (re)learning experience, but that's another entry.)