Wandering Thoughts archives

A decorator for decorators that accept additional arguments

Once you're happy with functions returning other functions (technically closures), basic ordinary decorators in Python are easy enough to use, write, and understand. You get things like:

from functools import wraps

def trace(func):
    @wraps(func)
    def _d(*args, **kwargs):
        print "start", func.__name__
        try:
            return func(*args, **kwargs)
        finally:
            print "end", func.__name__
    return _d

@trace
def jim(a, b):
    print "in jim"
    return a + b

(If you're decorating functions that all have the same argument signature, you can make the _d() closure take the specific arguments and pass them on instead of using the *args and **kwargs approach.)

But sometimes you have decorators that want to take additional arguments (over and above the function that they get handed to decorate). The syntax for doing this when you're declaring functions that will get decorated is easy and obvious:

@tracer("barney")
def fred(a, b, c):
    print "in fred:", a
    return b - c

Unfortunately the niceness stops there; an implementation of tracer() is much more complicated than trace(). Because of how Python has defined things, tracer() is no longer a decorator but a decorator factory, something that when called creates and returns the decorator that will actually be applied to fred(). You wind up with functions returning functions that return functions, or with tracer() actually being a class (so that calling it creates an instance that when called will actually do the decorating).

What we would like is for tracer() to be a regular decorator that just has extra arguments. Well, we can do that; all we need is another decorator that we will use on tracer() itself. Like so:

from functools import partial

def decor(decorator):
    @wraps(decorator)
    def _dd(*args, **kwargs):
        return partial(decorator, *args, **kwargs)
    return _dd

@decor
def tracer(note, func):
    fname = func.__name__
    @wraps(func)
    def _d(*args, **kwargs):
        print "-> %s: start %s" % (note, fname)
        try:
            return func(*args, **kwargs)
        finally:
            print "-> %s: end %s" % (note, fname)
    return _d

(You can make tracer()'s arguments have the function to be decorated first, but then you have to do more work because you can't use functools.partial(). While I think that func belongs as the first argument, I don't quite feel strongly enough to give up partial().)

The one nit with this is that positional arguments really are positional and keyword arguments really are keywords. You can't, for example, write:

@tracer(note="greenlet")
def fred(....):
    ....

(The only way around this is changing the order of arguments to tracer() so that func is first, which means giving up the convenience of just using partial().)

I've been thinking about decorators lately (they're probably the right solution for a code structure problem) and had this issue come up in my tentative design, so I felt like writing down my solution for later use. I'm sure that regular users of decorators already know all of these tricks.

(Decorators are one of the Python things I should use more. I don't for complex reasons that involve my history with significant Python coding.)

You should avoid using socket.SOMAXCONN

There you are, writing a socket-based server in Python, and you've decided that you should heed my advice and make its listen() backlog as large as possible. So you write the following code:

s = socket.socket(....)
s.bind(....)
s.list(socket.SOMAXCONN)

Given that SOMAXCONN is the traditional BSD sockets API name for 'the maximum value you can give listen()', this looks good and should do what you want. Sadly it does not.

The problem is that socket.SOMAXCONN is a constant. On many modern systems, the system-wide maximum listen() value is a kernel tuneable; it can be adjusted up (or down). If it's adjusted down, there is no problem with this code since the kernel will reduce the listen() backlog value down to the system setting. But if the system limit has been raised, the kernel does not adjust your listen() backlog up for you; it quietly stays at what you told the kernel. Which means that if someone has adjusted the limit up on your system, your code is not actually specifying a maximum backlog.

I can't blame Python for this one, because Unix doesn't actually expose an API for finding out the maximum socket listen() backlog that's allowed. The best that the socket module can do is set SOMAXCONN to some compile-time value, and in this case it's using the C-level API SOMAXCONN constant definition (which is usually 128).

So what should your code do? Since the kernel will quietly limit the listen() backlog to the system maximum for you, the obvious answer is to specify a large number, or a very large number, or even an absurdly large number. My tastes run to:

s.listen(max(1024, socket.SOMAXCONN))

This is a hack, of course, and you can argue that 1024 is not large enough and so on (and also that being paranoid about SOMAXCONN being larger than 1024 is pointless, so I should take the max() out). This is just what I do.

You may even feel that a large listen() backlog is pointless because in reality clients will start timing out long before you can service their requests. This is also a valid view. The whole area is complicated. But I still think you should avoid using socket.SOMAXCONN because it simply doesn't mean what people might think it means, not in real life on real systems.

Some brief opinions and notes on uWSGI

From my perspective, uWSGI is a WSGI server with a boatload of additional features and options that I'm not interested in. It will serve your WSGI applications via either FastCGI or its own 'uwsgi' protocol; the former is supported by most everything and the latter is supported by at least nginx. The problems with uWSGI are twofold; it is fearsomely complex and its documentation is mostly written as reference for people who already understand it.

(I started out planning to write some sort of opinionated quickstart guide to WSGI on uWSGI, but it turns out I haven't used it enough to feel confident and the Django people sort of already beat me to it.)

uWSGI can be configured through command line arguments or by creating a configuration file. I feel that the Django example adequately shows why you want to use a configuration file.

I feel nervous about uWSGI's manageability as far as things like loading new versions of a WSGI application's code and so on. There are probably ways to do all of this if you find them in the uWSGI documentation, but at the least uWSGI does not make it anywhere near as obvious as I'd like.

For all its fearsome complexity uWSGI does work and work well, and it was not too hard to configure for my testing once I sorted everything out. I have not engaged in an extensive search for WSGI servers or testing thereof, but uWSGI handily beats flup (although now that I look, Django no longer points you to flup as a deployment choice). I expect that it's the highest-performing WSGI server available today and probably also one of the most memory-efficient ones.

(WSGI server performance may well not particularly matter to you. To put it bluntly, a relatively slow application is unlikely to care about a little extra overhead added by a slower server. I was torture-testing things with a relatively fast WSGI application in a situation with significant resource constraints.)

However, after mulling it over I've decided that uWSGI is overkill (and overly complex) for my own use. While it's better than my own hand-rolled SCGI-based WSGI server it's not hugely so, and I am much happier with my own server; I understand it better, I can control it better, and I'm not impressed by uWSGI's mostly lack of options for adaptive control of how many worker processes you're running.

(The other WSGI server I've heard of is Gunicorn, which has the advantage of being one of Django's current recommended choices for deployment but the drawback of being HTTP-based. Also, I'm sorry to say that I have a very strong bias against people who require that I turn JavaScript on in order to navigate their main site (although apparently not their documentation once you find it).)

Comparing uWSGI to Apache's mod_wsgi is really an apples to oranges kind of thing. If you already have Apache and have relatively modest needs, using mod_wsgi is easier by far than running a separate WSGI server (regardless of the specific WSGI server). The drawback is that it requires you to run Apache and you pay a relatively significant memory penalty. I'm pretty sure that I couldn't really run mod_wsgi on a a 512 MB virtual server and stand up to any significant load (at least not without very advanced Apache tuning), whereas both uWSGI and my SCGI WSGI server (both running behind nginx) did quite well with minimal tuning.

Sidebar: some notes on uWSGI

The Django example shows an unrealistically low number of worker processes in my opinion; you probably want more, and if you want more you probably want to turn on uWSGI's adaptive process handling.

Unless you turn it off, uWSGI logs one (verbose) line per request it handles. This is turned off with the misleadingly named 'disable-logging' directive, which should really be called 'disable-request-logging'. If not changed, uWSGI logs to standard output (or maybe standard error).

The cost of an API mistake in the socket module's fromfd()

Suppose that you get handed a file descriptor that is a socket and you want to turn it into a Python socket object (clearly you are on Unix). The socket module has a Unix-only fromfd() function with the argument signature:

socket.fromfd(fd, family, type[, proto])

So how do you determine the family and type of the socket file descriptor you have, since you have to supply them?

Ha ha, silly you. The helpful socket module answer is 'we're not going to help you with that'. In fact the socket module provides no direct, official way of doing this; in order to do so, you need to sneak in through two increasingly baroque back doors in just the right way.

(And at least some things may go wrong if you get it wrong.)

The official Unix way of finding out the type of a socket is to issue a getsockopt(fd, SOL_SOCKET, SO_TYPE) call. Unfortunately the socket module does not allow you to do getsockopt() on file descriptors, only on actual socket objects. Fortunately the socket module does not actually care if you get the family and type right, at least as far as getsockopt goes, so:

s = socket.socket(fd, socket.AF_UNIX, socket.SOCK_STREAM)
styp = s.getsockopt(socket.SOL_SOCKET, socket.SO_TYPE)

Inconveniently there is no portable getsockopt() query that will give you the family. The official Unix way of doing this is more or less to make a getsockname() call with a plain struct sockaddr and then examine the sockaddr.sa_family field afterwards. The socket module doesn't provide a direct way to make raw getsockname() calls or see sa_family, but it does have a .getsockname() method on socket objects that gives you decoded, friendly results.

When I started this exercise, I expected that calling s.getsockname() on a socket created via fromfd() with the wrong family would raise a socket.error exception. I was far, far too innocent. Depending on exactly what you do, you get either the correct getsockname() results for the actual type of socket you are dealing with or, sometimes, interestingly mangled results. On Python 3 you can also get UnicodeDecodeErrors in the right circumstances. The safest thing to do turns out to be to make your dummied-up socket be an AF_UNIX socket; you can then call s.getsockname() with reasonable safety and examine the resulting name to reverse engineer the socket family.

(It's the safest because AF_UNIX sockets have the biggest version of struct sockaddr; you've got the greatest chance that a full copy of any other socket family's sockaddr structure will fit into it. Python is presumably blindly making the getsockname() call with the sockaddr appropriate for the apparent family, then interpreting it based on the actual returned socket family. If the sockaddr structure is truncated, odd things happen.)

What this really illustrates is that the socket module completely dropped the ball on fromfd()'s API. You should not be able to give it a family and type at all; since the rest of the socket code clearly counts on those being correct, the socket module code should determine them itself. This would be easier to use and render .getsockname() non-crazy.

(getsockname()'s implementation is completely sensible if a socket's family is always correct.)

The sinking feeling of discovering a design mistake in your code

It is not a pleasant feeling to slowly realize that you've made a design mistake down in the depths of your program's core data structures and code flow. I am having that experience now, so this is a war story about how what looks like the obvious right choice can be anything but.

One of the major things that DWiki does is convert DWikiText into HTML. Back when I started coding DWiki, I decided that the fundamental result of this rendering process would be a blob of HTML. After all, what could be more logical? The job of the renderer was to turn DWikiText into HTML, so clearly its output was a blob of HTML.

In retrospect the wheels started coming off this particular wagon almost immediately; I just didn't pay attention to the warning signs. You see, the output of the DWikiText rendering process is not just a blob of HTML. It's also things like the title of the page (in several different formats), whether or not the page is cacheable, what directory modification times are important for caching the page, and whether the page specifies permissions settings. And the giant blob of HTML has important structure itself; for a start, part of it is the title and part of it is the rest of the text, and it would be nice to be able to get one without the other.

(Simplifying, if a page starts with any level of header the header is taken to be the title. But the header is also rendered as part of the HTML and can't be separated from it later. This is why the Atom feed here has always repeated the entry titles in the actual text of the entry; in retrospect that should have been a warning sign.)

All of these extra things that DWikiText rendering produced are sort of glued on the side of things in the current code in various somewhat hacky ways (which should have been a warning sign to me). What the core rendering process should produce is an actual data structure that represents all of these bits explicitly. Code that just wants the HTML blob would then generate it from appropriate bits of the data structure.

Why didn't I see this when I wrote the code? Because I started from the basic operation of 'turn DWikiText into HTML' and never lifted my head up to see the growing big picture. Every time I needed something else from the rendering process I basically got out a hammer and took another shot at the code, because that was the easy way (and towards the end, the thought of making major changes to the rendering process scared me).

Now I've reached the point where that doesn't work any more. To make the long delayed changes to DWiki that I want I'm going to have to totally rip apart and redo core rendering (complete with the instability that that implies, plus I'm going to have to understand the code again; it's been years (and yes, that's a bad sign too)).

(One benefit of the change will be better Atom feeds. Another will be better caching. Right now I'm sort of caching the generated HTML when what I should be caching is the underlying data structure that results from rendering.)

Sidebar: the insoluble problem that pushed me over the edge

When people look at a single entry I want there to be a little discreet entry date below the entry title (I've tried a version with the date above the title and I don't like it). However the entry title and the entry text are all part of the same HTML blob and are currently inseparable; I can't crack them apart to insert template logic for this the way I want to.

There are a pile of ugly hack workarounds, none of which I like. For example I could make something that chopped the title out of the HTML blob with a regular expression, but ew. I could also hack up the rendering process to directly insert HTML for the date (in various ways), but that's equally unclean and also has unpleasant interactions with the disk caches.

All of this would be simple if CSS allowed you to relocate <div>s in the page layout, but as far as I know this is impossible (short of manipulating the DOM with JavaScript). You can fix the position of a <div> in various ways but not say 'slice it out of here and put it right after that thing, then lay everything out normally'. All things considered I can't really blame CSS for that omission.