try:/finally: and generators

June 15, 2009

Suppose that you have code that generates an abstract 'list' of some sort and returns it inside a try:/finally: block. There are two common ways to code this; you can return a real list, or you can use yield to be a generator. You might even code it one way and change it to the other later, which is generally a transparent change.

Not this time, though. If you are using finally:, the two options can have quite different behavior. Constructing an example is tedious, but explaining the issue is simple:

Using yield postpones the execution of your finally: blocks from when your results are generated and returned to when your results are used, which may be some time later.

In simple situations you won't notice this, because the results are used immediately after they're returned. In more complex situations you'll probably get mysterious ordering issues about when your finally: statements run, as they'll appear to run well after they 'should' (and when they did when your function wasn't using yield).

(For example, consider a finally: that closes down a database connection. If the result of your database lookup function is just put in a data structure and only looked at later, you could build up a lot more database connections than you expect.)

In fact this is one part of a general issue: when you use yield, all of the objects and resources held alive by your function are only released when your results are used. Effectively yield turns your ordinary function into a closure, with the resulting consequences for potential resource leaks.

(Credit where credit is due department: I was exposed to this issue by Using yield Statements in WSGI Middleware can be Very Harmful.)

Written on 15 June 2009.
« How to set up your vacation messages to get thrown off mailing lists
Maybe understanding blogrolls »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Jun 15 01:30:29 2009
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.