Another way that generators are not lists: modifying them

June 8, 2009

A long time ago, I wrote some stuff on how generators are not lists (okay, technically it was about iterators), and one of the things that I mentioned is that generators do not have list methods. Well, there's a consequence of that that only struck me recently: you need completely different code to modify a returned generator than to modify a returned list.

Suppose you have a function that returns something that is conceptually a list of items. Further suppose that you have another function that modifies what the first function returns; perhaps you want to add something on the end. If you know you're dealing with a list, you write:

def append(func, extra):
    r = func()
    return r

If func() is a generator, this code blows up. You have two choices; first, you can forcefully turn the result of func() into a list, and second, you can rewrite append() as a generator (which will work regardless of what func() returns, but may have consequences that make it undesirable):

def append(func, extra):
   for it in (func(), extra):
       for e in it:
           yield e

(Yes, yes, one can write this using itertools.chain(). Then people would have to look it up.)

In either case, you have to actively make a decision about what your function will do. You cannot passively modify whatever you get handed and pass it up to your caller without changing its nature; you must decide that no matter what func() returns, you're either returning a list or an iterator.

(Technically you can, since you can see if you got handed something that follows the iterator protocol or whether it looks sequence-like. But that way lies madness.)

Comments on this page:

From at 2009-06-26 15:24:19:

What if you make the generator support append just like a list?

def appendable(gen):
   """ given a generator, return a pseudogenerator with an append function """

   class appender(object):
       def __init__(self, *args, **kwargs):
           self.g = gen(*args, **kwargs)
           self.appends = []

       def __iter__(self):
           for x in self.g:
               yield x
           while self.appends:
               yield self.appends.pop(0)

       def append(self, val):

   return appender

if __name__ == "__main__":
   def foo(max):
       for x in range(max):
           yield x

   f = foo(3)
   print "gen:   got", list(f), "; expecting [1, 2, 3, 8, 9, 10]"
   print "gen:   got", list(f), "; expecting [11, 12]"

   l = [1,2,3]
   print "list:  got", l, "; expecting [1, 2, 3, 8, 9, 10]"

Which gives:

  gen:   got [0, 1, 2, 8, 9, 10] ; expecting [1, 2, 3, 8, 9, 10]
  gen:   got [11, 12] ; expecting [11, 12]
  list:  got [1, 2, 3, 8, 9, 10] ; expecting [1, 2, 3, 8, 9, 10]


By cks at 2009-07-07 00:17:07:

Department of belated replies: .append() is only one of the modifications that you can wind up wanting to make to a conceptual list; another is, for example, prepending things. The other issue is that the things that you're wrapping probably aren't going to be using your nice whatever-able generator wrapper. You can fix this but then you have to know that they return a generator, which gets us back to this very issue.

Written on 08 June 2009.
« Monitoring systems should be always be usefully informative
Users are lazy »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Jun 8 23:15:13 2009
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.