Wandering Thoughts archives

Created functions and exception stack backtraces

There's a problem with functions that make functions: they obscure information in exception stack backtraces, because all of the created functions have the same name. If you have a lot of auto-created functions, just which one is blowing up and how do you tell them apart?

(I noted a similar effect with lambdas back here.)

A function's name is part of its func_code attribute, specifically func_code.co_name. Unfortunately this attribute is read-only, so you can't have your function creating function rewrite the name of a created function to something more useful for stack backtraces. (While you can change the function's __name__ attribute, it is not used for stack backtraces.)

A similar effect happens with functions that create classes (don't look at me like that; sometimes it can look like the right thing to do). Fortunately, you can rename classes by writing to their __name__ attribute and thus make it so that objects of each different created class have a a distinct repr() and so on, which is quite useful for debugging the resulting potential mess.

(I think changing __name__ is somewhat less work than writing a custom __repr__ function in the generic class framework, and besides you'd have to save the naming information somewhere anyways.)

I consider __dict__ an implementation detail

Almost all Python objects implement their namespace as a Python dictionary, which they expose as their __dict__ member. You can do interesting things by playing around with this dictionary directly, for example adding bulk name/value mappings to an object by just doing something like obj.__dict__.update(pairs).

I don't like doing this, because I consider __dict__ to be an implementation detail, or at least fairly magical, and thus I feel that I shouldn't be playing around with it; I prefer to get the same results with more general higher-level things like setattr. Seeing code that uses __dict__ makes me vaguely twitchy.

In some sense __dict__ isn't merely an implementation detail, because it is sufficiently well documented (and necessary for things like __setattr__) that it will be sticking around. In another sense it is, because not all objects have a __dict__ (and even when they have one, not all of their attributes necessarily show up there), so you can't assume that you can use it to fish around in an arbitrary object. If you do anyways, you're making assumptions about how other code implements its objects that may come to bite you someday.

(The most obvious case of objects without a __dict__ is instances of classes that use __slots__.)

I'm aware that this is just a personal twitch, and possibly a silly one at that. A lot of Python programmers are perfectly happy to use __dict__, and I've committed worse hacks in my own code without thinking too hard.

Handling lines with something-separated fields for Python

As a system administrator, I spend a bunch of my time dealing with files made up of lines that are composed of fields separated by some character. A classical example is /etc/passwd, with colon-separated fields. These file formats are ordered lists with named fields, which should sound familiar, but they don't show up as Python lists, they show up as lines of text and they want to be output as text; that we use lists to represent them is just an implementation detail.

This only takes a little bit of extra work to implement on top of our previous SetMixin class:

class FieldLine(SetMixin, list):
    separator = ":"
    def __init__(self, line):
        n = line.split(self.separator)
        super(FieldLine, self).__init__(n)
    def __str__(self):
        return self.separator.join(self)

class PasswdLine(FieldLine):
    fields = gen_fields('name', 'passwd',
                        'uid', 'gid', 'gecos',
                        'dir', 'shell')

(Where gen_fields is basically the dict()-ized version of enum_args from here.)

Now that I've written these entries, I have a confession: this is actually what I started out doing. I didn't first build a general ordered list with named fields class and then realized it could be used to deal with /etc/passwd lines; I started out needing to deal with /etc/passwd lines, decided that I wanted read/write access to named fields, and then built downwards. I just wrote it up backwards because it looks neater that way.

(In fact this is the cleaned up and idealized version of this class. The real one in my program does not subclass list; instead it is a normal class with a private 'field_store' list and everything just directly manipulates that. It also doesn't handle the slicing cases, because I didn't need to. I did the new version for here for various reasons, including that it was a good excuse to play around with subclassing built in types.)