Wandering Thoughts archives

Part of why Python 3.5's await and async have some odd usage restrictions

Python 3.5 added a new system for coroutines and asynchronous programming, based around new async and await keywords (which have the technical details written up at length in PEP 492). Roughly speaking, in terms of coroutines implemented with yield from, await replaces 'yield from' (and is more powerful). So what's async for? Well, it marks a function that can use await. If you use await outside an async function, you'll get a syntax error. Functions marked async have some odd restrictions, too, such as that you can't use yield or yield from in them.

When I described doing coroutines with yield from here, I noted that it was potentially error prone because in order to make everything work you had to have an unbroken chain of yield from from top to bottom. Break the chain or use yield instead of yield from, and things wouldn't work. And because both yield from and yield are used for regular generators as well as coroutines, it's possible to slip up in various ways. Well, when you introduce new syntax you can fix issues like that, and that's part of why async and await have their odd rules.

A function marked async is a (native) coroutine. await can only be applied to coroutines, which means that you can't accidentally treat a generator like a coroutine the way you can with yield from. Simplifying slightly, coroutines can only be invoked through await; you can't call one or use them as a generator, for example as 'for something in coroutine(...):'. As part of not being generators, coroutines can't use 'yield' or 'yield from'.

(And there's only await, so you avoid the whole 'yield' verus 'yield from' confusion.)

In other words, coroutines can only be invoked from coroutines and they must be invoked using the exact mechanism that makes coroutines work (and that mechanism isn't and can't be used for or by anything else). The entire system is designed so that you're more or less forced to create that unbroken chain of awaits that makes it all go. Although Python itself won't error out on import time if you try to call a async function without await (it just won't work at runtime), there's probably Python static checkers that look for this. And in general it's an easy rule to keep track of; if it's async, you have to await it, and this status is marked right there in the function definition.

(Unfortunately it's not in the type of the function, which means that you can't tell by just importing the module interactively and then doing 'type(mod.func)'.)

Sidebar: The other reason you can only use await in async functions

Before Python 3.5, the following was completely valid code:

def somefunc(a1, b2):
   ...
   await = interval(a1, 10)
   otherfunc(b2, await)
   ...

In other words, await was not a reserved keyword and so could be legally used as the name of a local variable, or for that matter a function argument or a global.

Had Python 3.5 made await a keyword in all contexts, all such code would immediately have broken. That's not acceptable for a minor release, so Python needed some sort of workaround. So it's not that you can't use await outside of functions marked async; it's that it's not a keyword outside of async functions. Since it's not a keyword, writing something like 'await func(arg)' is a syntax error, just as 'abcdef func(arg)' would be.

The same is true of async, by the way:

def somefunc(a, b, async = False):
  if b == 10:
     async = True
  ....

Thus why it's a syntax error to use 'async for' or 'async with' outside of an async function; outside of such functions async isn't even a keyword so 'async for' is treated the same as 'abcdef for'.

(I'm sure this makes Python's parser that much more fun.)