Fifteen years of DWiki, the Python engine of Wandering Thoughts

October 26, 2020

DWiki, the wiki engine that underlies Wandering Thoughts (this blog), is fifteen years old. That makes it my oldest Python program that's in active, regular, and even somewhat demanding use (we serve up a bunch of requests a day, although mostly from syndication feed fetchers and bots on a typical day). As is usual for my long-lived Python programs, DWiki's not in any sort of active development, as you can see in its github repo, although I did add a an important feature just last year (that's another story, though).

DWiki has undergone a long process of sporadic development, where I've added important features slowly over time (including performance improvements). This sporadic development generally means that I come back to DWiki's code each time having forgotten much of the details and have to recover them. Unfortunately this isn't as easy as I'd like and is definitely complicated by historical decisions that seemed right at the time but which have wound up creating some very tangled and unclear objects that sit at the core of various important processes.

(I try to add comments for what I've worked out when I revisit code. It's probably not always successful at helping future me on the next time through.)

DWiki itself has been extremely stable in operation and has essentially never blown up or hit an unhandled exception that wasn't caused by a very recent code change of mine. This stability is part of why I can ignore DWiki's code for long lengths of time. However, DWiki operates in an environment where DWiki processes are either transient or restarted on a regular basis; if it was a persistent daemon, more problems might have come up (or I might have been forced to pay more attention to reference leaks and similar issues).

Given that it's a Unix based project started in 2005, Python has been an excellent choice out of the options available at the time. Using Python has given me long life, great stability in the language (since I started as Python 2 was reaching stability and slowing down), good enough performance, and a degree of freedom and flexibility in coding that was probably invaluable as I was ignorantly fumbling my way through the problem space. Even today I'm not convinced that another language would make DWiki better or easier to write, and most of the other options might make it harder to operate in practice.

(To put it one way, the messy state of DWiki's code is not really because of the language it's written in.)

Several parts of Python's standard library have been very useful in making DWiki perform better without too much work, especially pickle. The various pickle modules make it essentially trivial to serialize an object to disk and then reload it later, in another process, which is at the core of DWiki's caching strategies. That you can pickle arbitrary objects inside your program without having to make many changes to them has let me easily add pickle based disk caches to various things without too much effort.

At the same time, the very strong performance split in CPython between things implemented in C and things implemented in Python has definitely affected how DWiki is coded, not necessarily for the better. This is particularly obvious in the parsing of DWikiText, which is almost entirely done with complex regular expressions (some of them generated by code) because that's by far the fastest way to do it in CPython. The result is somewhat fragile in the face of potential changes to DWikiText and definitely hard for me to follow when I come back to it.

(With that said, I feel that parsing all wikitext dialects is a hard problem and a high performance parser is probably going to be tricky to write and follow regardless of the implementation language.)

DWiki is currently written in Python 2, but will probably eventually be ported to Python 3. I have no particular plans for when I'll try to do that for various reasons, although one of the places where I run a DWiki instance will probably drop Python 2 sooner or later and force my hand. Right now I would be happy to leave DWiki as a Python 2 program forever; Python 3 is nicer, but since I'm not changing DWiki much anyway I'll probably never use many of those nicer things in it.

Written on 26 October 2020.
« Remotely upgrading my office workstation to Fedora 32 worked fine
Link: [Firefox] Navigational Instruments »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Oct 26 00:14:28 2020
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.