Wandering Thoughts archives

2018-05-10

Python modules use operator overloading in two different ways

In Python (as in elsewhere), there are at least two different things that people use operator overloading for. That there's more than one thing makes a difference because some patterns of designing how operator overload work aren't sufficiently general to handle both things; if you want to serve both groups, you need to design a more general mechanism than you might expect, one that delegates more power to objects.

The first use of operator overloading is to extend operators so that they work (in the traditional ways) on objects that they wouldn't normally work on. The classical examples of this is complex numbers and rational numbers (both of which Python has in the standard library), and in general various sorts of things built with numbers and numeric representations. However you can go beyond this, to objects that aren't strictly numeric but which can use at least some of the the traditional numeric operators in ways that still obey the usual rules of arithmetic and make sense. Python sets implement some numeric operations in ways that continue to make sense and are unsurprising.

The second use is to simply hijack the operations in order to do something convenient for your objects with a handy symbol for it. Sometimes these operations are vaguely related to their numeric equivalents (such as string multiplication, where "a" * 4 gets you "aaaa"), but sometimes they have nothing to do with it. The classic example of the latter is the string % operator, which has nothing at all to do with arithmetic but instead formats a string using % formatting codes. Using the % operator for this is certainly convenient and it has a certain mnemonic value and neatness factor, but it definitely has nothing to do with %'s normal use in arithmetic.

Now, let us consider the case of Python not allowing you to overload boolean AND and OR. In a comment on that entry, Aneurin Price said:

I'm not at all convinced by this argument. My expectation for this hypothetical __band__ is that it would be called after evaluating a and finding it truthy, at which point b is evaluated either way. [...]

This is definitely true if you think of operator overloading as only for the first case. But, unfortunately for the design of overloading AND and OR, this is not all that people would like to use it for. My understanding is that ORMs such as Django's and SQLAlchemy would like to intercept AND and OR in order to build up complicated conditional SQL queries with, essentially, a DSL based on Python expressions. In this DSL, they would like to be able to write something like:

Q.descfield.startswith("Who") or Q.descfield.startswith("What")

This wouldn't evaluate or produce any sort of truth value; instead it would produce an object representing a pending SQL query with a WHERE clause that encoded this OR condition. Later you'd execute the SQL query to produce the actual results.

If operator overloading for AND and OR paid any attention to the nominal truth value of the left expression, there is no way to make this work. Instead, allowing general overloading of AND and OR requires allowing the left side expression to hijack the process before then. In general, operator overloading that allows for this sort of usage needs to allow for this sort of early hijacking; fortunately this is generally easy for arithmetic operators.

(I'm not sure Python has truly general support for mixing unusual numerical types together, but then such general support is probably very hard to implement. I think you want to be able to express a compatibility table, where each type can say that its overloads handle certain other types or types that have certain properties or something. Otherwise getting your rational number type to interact well with my Point type gets really complicated really fast, if not impossible.)

TwoSortsOfOverloading written at 00:28:02; Add Comment

By day for May 2018: 7 10; before May; after May.

Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.