Wandering Thoughts archives

How we can use yield from to implement coroutines

Give my new understanding of generator functions and yield from, we can now see how to use yield from to implement coroutines and an event loop. Consider a three level stack of functions, where on the top layer you have an event loop, in the middle you have the processing code you write, and on the bottom are event functions like wait_read() or sleep().

Let's start with an example processing function or two:

def countdown(n):
   while n:
      print("T-minus", n)
      n -= 1
      yield from sleep(1)

def launch(what, nsecs):
   print("counting down for", what)
   yield from countdown(nsecs)
   print("launching", what)

To start a launch, we call something like 'coro.start(launch("fred", 10))', which looks a bit peculiar since it sort of seems like coro.start() should get control only after the launch. However, we already know that calling a generator function doesn't do exactly what it looks like. What coro.start() gets when we do this is an unstarted generator object (which handily encapsulates those arguments to launch(), so we don't have to do it by hand).

When the coroutine scheduler starts the launch() generator object, we wind up with a chain of yield froms that bottoms out at sleep(). What sleep() yields is passed back up to the coroutine scheduler and the entire call chain is suspended; this is no different that what I did by calling .send() by hand yesterday. What sleep() returns to the scheduler is an object (call it an event object) that tells the coroutine scheduler under what conditions this coroutine should be resumed. When the scheduler reaches the point that the coroutine should be run again, the scheduler will once again call .send(), which will resume execution in sleep(), which will then return back to countdown(), and so on. The scheduler may use this .send() to pass information back to sleep(), such as how long it took before the coroutine was restarted.

Here yield and yield from are being used for two things. First, they create a communication channel between the coroutine scheduler and the low-level event functions like sleep(). Our launch() and countdown() functions are oblivious to this since they don't touch either the value sleep() yields up to the scheduler or the value that the scheduler injects to sleep() with .send(). Second, the chain of yield from and the final yield neatly suspend the entire call stack.

In order for this to work reliably, there are two rules that our user-written processing functions have to follow. First, they must never accidentally attempt to do anything with the sleep() generator function. It is okay but unclear for a non-generator function to call sleep() and return the result:

def sleep_minutes(n):
   return sleep(n * 60)

def long_countdown(n):
   while n:
      print("T-minus", n, "minutes")
      yield from sleep_minutes(1)
      n -= 1

This is ultimately because 'yield from func()' is equivalent to 't = func(); yield from t'. We don't care just how the generator object got to us so we can yield from it, we just care that it did.

However, at no stage in our processing functions can we attempt to look at the results of iterating sleep()'s generator object, either directly or indirectly by writing, say, 'for i in countdown(10):'. This rules out certain patterns for writing processing functions, for instance this one:

def label_each_sec(label, n):
   for _ in tick_once_per_sec(n):
      print(label)

This leads to the second rule, which is that we must have an unbroken chain of yield froms from the top to the bottom of our processing functions, right down to where you use an event function such as sleep(). Each function must 'call' the next using the 'yield from func()' idiom. In effect we don't have calls from one processing function to another; instead we're passing control from one function to the next. In my example, launch() passes control to countdown() until the countdown expires (and countdown() passes control to sleep()). If we actually call a processing function normally or accidentally use 'yield' instead of 'yield from', the entire collection explodes into various sorts of errors without getting off the launch pad and you will not go to space today.

As you might imagine, this is a little bit open to errors. Under normal circumstances you'll catch the errors fairly fast (when your main code doesn't work). However, since errors can only be caught at runtime when a non-yield from code path is reached, you may have mistakes that lurk in rarely executed code paths. Perhaps you have a rarely invoked last moment launch abort:

def launch(what, nsecs):
   print("counting down for", what)
   yield from countdown(nsecs)
   if launch_abort:
      print("Aborting launch! Clear the launch pad for", what)
      yield sleep(1)
      print("Flooding fire suppression ...")
   else:
      print("launching", what)

It might be a while before you discovered that mistake (I'm doing a certain amount of hand-waving about early aborts in countdown()).

(See also my somewhat related attempt at understanding this sort of thing in a Javascript context in Understanding how generators help asynchronous programming. Note that you can't use my particular approach from that entry in Python with 'yield from' for reasons beyond the scope of this entry.)