Wandering Thoughts archives

Taking advantage of polymorphic WSGI

I call WSGI polymorphic because with some work, you can run the same WSGI application in more than one way; all you need is a WSGI adaptor for the particular environment, whether that is a straight CGI or a SCGI server.

(Running a WSGI app as a straight CGI might seem strange, but it may be easier to debug it as a CGI even if you intend to run your application in a persistent daemon for deployment. If nothing else, you can be sure that you're not debugging your SCGI or FastCGI daemon bit as well.)

For DWiki, I've exploited this to contrive a relatively Rube Goldberg environment where it runs either as a CGI or as a SCGI server. This gives me the simplicity and ease of management of a CGI along with the performance advantages under load of an SCGI server, depending on what the system needs at the time. How it works is that Apache runs a CGI to handle every DWiki requests, which is a small frontend written in C. The frontend either exec()s the main Python program as a CGI if the load is low or forwards the request using SCGI to the SCGI server, starting the SCGI daemon process as necessary.

I wrote a C frontend, instead of using a small Python program, because I wanted it to start fast and run light. This is especially important under load with an SCGI server, because while all the heavyweight stuff is happening in the daemon, Apache still has to start all the CGI processes and each sticks around until their request is finished. While C is more annoying to write than Python, SCGI is simple enough to implement that it wasn't too much of a pain, and it uses much less resources than a Python version would.

There are a number of advantages of doing things this way:

• I don't need any special web server modules or configuration; all I need is the ability to run a compiled CGI-BIN.
• I don't have to worry about starting my SCGI daemon when the system boots, or restarting it when it dies; the CGI frontend handles that.
• I have a nice lightweight place to generate a polite 'the system is overloaded, try again later' message if and when necessary.

For me, the biggest win is that I don't have to do anything to manage the whole SCGI daemon, since that's the part I like least about persistent daemons.

(I've also used polymorphic WSGI for performance tuning.)

An object oriented design mistake illustrated

Being written in Python, DWiki uses a certain amount of object oriented design (not huge amounts; it has basically no inheritance hierarchy among its classes, although they have various interrelations). In particular, it has a Page class that represents a particular real page at the model level (not the presentation level, which is where template processing and so on happens).

Here is a trick question: is a page's URL a property of the page? (That is, should the Page class have a url method or property.)

It certainly seems intuitive that a page's URL is a property of the page, and when I started writing DWiki I started out trying to code things up this way. It turned into a disaster, because the intuitive answer is wrong. In a web server environment, it turns out that a page's full absolute URL must be built from a combination of data from three objects:

• the page itself is what knows its path relative to the root of the model (technically the model's data store).
• the configuration is what knows the URL of the model's root, to which the page's path is added to get the absolute URL.
• only the request knows what the web server's hostname and port is, because that comes from the Host: header in the request itself.

(A DWiki instance doesn't require you to talk to it with any particular hostname and port, because I dislike burning that sort of system-level information into what is effectively application configuration.)

One of the reasons I didn't realize this up front before I started coding DWiki was that I hadn't done enough work with HTTP and the web to really be aware of all of the issues; in fact I was so ignorant that I didn't realize I was ignorant, so I forged on in blithe confidence and ignored various warning signs.

(And I have to admit that I made some terrible howlers in this area even after I had DWiki and WanderingThoughts going. Writing DWiki has been not merely educational, but repeatedly educational.)