Chris's Wiki :: blog/python/GeneratorListModification

Another way that generators are not lists: modifying them

June 8, 2009

A long time ago, I wrote some stuff on how generators are not lists (okay, technically it was about iterators), and one of the things that I mentioned is that generators do not have list methods. Well, there's a consequence of that that only struck me recently: you need completely different code to modify a returned generator than to modify a returned list.

Suppose you have a function that returns something that is conceptually a list of items. Further suppose that you have another function that modifies what the first function returns; perhaps you want to add something on the end. If you know you're dealing with a list, you write:

def append(func, extra):
    r = func()
    r.extend(extra)
    return r

If func() is a generator, this code blows up. You have two choices; first, you can forcefully turn the result of func() into a list, and second, you can rewrite append() as a generator (which will work regardless of what func() returns, but may have consequences that make it undesirable):

def append(func, extra):
   for it in (func(), extra):
       for e in it:
           yield e

(Yes, yes, one can write this using itertools.chain(). Then people would have to look it up.)

In either case, you have to actively make a decision about what your function will do. You cannot passively modify whatever you get handed and pass it up to your caller without changing its nature; you must decide that no matter what func() returns, you're either returning a list or an iterator.

(Technically you can, since you can see if you got handed something that follows the iterator protocol or whether it looks sequence-like. But that way lies madness.)

Comments on this page:

From 128.33.22.52 at 2009-06-26 15:24:19:

What if you make the generator support append just like a list?

def appendable(gen):
   """ given a generator, return a pseudogenerator with an append function """

   class appender(object):
       def __init__(self, *args, **kwargs):
           self.g = gen(*args, **kwargs)
           self.appends = []

       def __iter__(self):
           for x in self.g:
               yield x
           while self.appends:
               yield self.appends.pop(0)

       def append(self, val):
           self.appends.append(val)

   return appender

if __name__ == "__main__":
   @appendable
   def foo(max):
       for x in range(max):
           yield x

   f = foo(3)
   f.append(8)
   f.append(9)
   f.append(10)
   print "gen:   got", list(f), "; expecting [1, 2, 3, 8, 9, 10]"
   f.append(11)
   f.append(12)
   print "gen:   got", list(f), "; expecting [11, 12]"

   l = [1,2,3]
   l.append(8)
   l.append(9)
   l.append(10)
   print "list:  got", l, "; expecting [1, 2, 3, 8, 9, 10]"

Which gives:

  gen:   got [0, 1, 2, 8, 9, 10] ; expecting [1, 2, 3, 8, 9, 10]
  gen:   got [11, 12] ; expecting [11, 12]
  list:  got [1, 2, 3, 8, 9, 10] ; expecting [1, 2, 3, 8, 9, 10]

Jeff

By cks at 2009-07-07 00:17:07:

Department of belated replies: .append() is only one of the modifications that you can wind up wanting to make to a conceptual list; another is, for example, prepending things. The other issue is that the things that you're wrapping probably aren't going to be using your nice whatever-able generator wrapper. You can fix this but then you have to know that they return a generator, which gets us back to this very issue.

Written on 08 June 2009.

«	Monitoring systems should be always be usefully informative

Users are lazy