Taking advantage of polymorphic WSGI

I call WSGI polymorphic because with some work, you can run the same WSGI application in more than one way; all you need is a WSGI adaptor for the particular environment, whether that is a straight CGI or a SCGI server.

(Running a WSGI app as a straight CGI might seem strange, but it may be easier to debug it as a CGI even if you intend to run your application in a persistent daemon for deployment. If nothing else, you can be sure that you're not debugging your SCGI or FastCGI daemon bit as well.)

For DWiki, I've exploited this to contrive a relatively Rube Goldberg environment where it runs either as a CGI or as a SCGI server. This gives me the simplicity and ease of management of a CGI along with the performance advantages under load of an SCGI server, depending on what the system needs at the time. How it works is that Apache runs a CGI to handle every DWiki requests, which is a small frontend written in C. The frontend either exec()s the main Python program as a CGI if the load is low or forwards the request using SCGI to the SCGI server, starting the SCGI daemon process as necessary.

I wrote a C frontend, instead of using a small Python program, because I wanted it to start fast and run light. This is especially important under load with an SCGI server, because while all the heavyweight stuff is happening in the daemon, Apache still has to start all the CGI processes and each sticks around until their request is finished. While C is more annoying to write than Python, SCGI is simple enough to implement that it wasn't too much of a pain, and it uses much less resources than a Python version would.

There are a number of advantages of doing things this way:

• I don't need any special web server modules or configuration; all I need is the ability to run a compiled CGI-BIN.
• I don't have to worry about starting my SCGI daemon when the system boots, or restarting it when it dies; the CGI frontend handles that.
• I have a nice lightweight place to generate a polite 'the system is overloaded, try again later' message if and when necessary.

For me, the biggest win is that I don't have to do anything to manage the whole SCGI daemon, since that's the part I like least about persistent daemons.

(I've also used polymorphic WSGI for performance tuning.)