Wandering Thoughts archives

How DWiki uses partial function evaluation

As part of a longer entry, Muharem Hrnjadovic asks:

I would love to see examples or code snippets that are made possible and/or improved greatly by leveraging functools.partial().

I'm not sure that my example qualifies, but I'll take a shot at it.

DWiki has a WSGI-like processing pipeline to handle requests; they get passed from function to function, possibly getting mutated on the way down and possibly having the results mutated on the way back up.

The pipeline functions look like this:

def TimeLayer(next, request):
  t0 = time.time()
  resp = next(request)
  td = time.time() - t0
  request.log("Time: %.3g sec" % td)
  return resp

Notice how next is called: there is no hypothetical next_next parameter. This means that there's a disconnect between how these functions want to be called and how they want to call the next step in the pipeline (and how the outside world wants to start the pipeline); they want to know the next function to call, but they don't want to have to know the next function's next function (and so on down the line).

The answer is partial function evaluation. All of the next functions, and the function called at the top of the entire pipeline, are partially evaluated functions with the next parameter filled in in the process of building the pipeline.

DWiki predates Python 2.5 and the functools module, so it currently uses a closure and lambda-based approach. However, functools.partial is clearly the better way to write the function that assembles the pipeline; it would come out roughly like this:

def genDwikiStack(stklst):
  # The bottom function takes no
  # next argument.
  cur = stklst.pop()
  while stklst:
    nxt = stklst.pop()
    cur = functools.partial(nxt, cur)
  return cur

(This can also be written as a reduce one-liner, assuming you have more than one function in stklst and you change it so that stklst is bottom to top instead of top to bottom.)

One of the small issues with DWiki's current lambda-based approach is that stack backtraces are somewhat confusing and overly verbose, since they alternate real functions with the lambdas that wrap them. One potential advantage of a real implementation of partial function evaluation is avoiding this, although I suspect that the current implementation doesn't.

More fun with Python's indentation rules

Python's indentation rules apply only to logical lines (as the documentation is careful to spell out), although the indentation level only comes from the first physical line. This opens up various sorts of peculiarity and evil, some of which I mentioned last time.

There are two ways for multiple physical lines to be joined into one logical lines: being explicitly continued with a \ at the end of the line (explicit line joining), or being inside a (, {, or [ (implicit line joining).

Of the two, implicit line joining is more wacky. Explicit line joining cannot have comments (except on the last physical line; this is sort of spelled out in the comments rules) or completely blank lines (although it can have lines that are blank except for the \).

(Note that blank physical lines are not necessarily the same thing as blank logical lines.)

Implicit line continuation takes priority over explicit line continuation, so you can write:

def foo(a, \
        # testing
        b):
  pass

All of these rules are implemented in the tok_get routine in Parser/tokenizer.c in the CPython source. (As usual, because not all of them are in the language spec, counting on them all is somewhat hazardous.)

It's somewhat unfortunate that Python doesn't have any rules at all for the indentation levels of physical lines. I think it would not be difficult to have a basic requirement that (for example) continued lines had to be indented at least as much as the logical line that they're on.

(Triple-quoted strings are handled by a different mechanism that wouldn't be affected by this. Although it wouldn't necessarily be a bad thing if they were; the need to de-indent triple quoted strings is arguably one of those small Python warts. (Unfortunately it is a problem with no good answer, since it is a conflict between literal representations and program layout aesthetics.))