Two problems with Python's file iterators

December 26, 2007

Modern versions of Python let you process each line in a file in a simple way, with just 'for line in fp: ..', replacing either manual while loops with .readline() or the memory inefficiency of letting .readlines() pull the entire thing into memory. But there's two bugs, both of which can be illustrated by running a 'pycat' program:

import sys
for line in sys.stdin:
    sys.stdout.write(line)

If you run this without standard input redirected, you will immediately notice the problems:

  • the program only gets lines from standard input in big blocks, instead of one line at a time.
  • you (almost always) have to give two ^D's to the program before it sees end of file and exits.

Both problems are caused by the same underlying decision: despite using Unix's traditional stdio functions, which do their own buffering, Python adds its own layer of forced buffering for file iteration. This forced buffering even has the perverse effect that you can't mix file iteration and explicit .readline() et al, even if you break out of the iteration loop.

(Since this is a deliberate and long standing design decision, I suspect that the Python people are not interested in bug reports.)

These bugs might seem relatively minor, except that reading from terminals isn't the only case where you really need to handle input a line at a time, without insisting on buffering up a bunch of it; another is dealing with line oriented network protocols.

As a result of running into these issues I reflexively avoid file iteration in my own code, which makes me grumpy when I write yet another 'read the lines' loop. (By now, I have the necessary while pattern memorized.)

Sidebar: coding around the problem

The necessary while pattern for reading from files is:

while 1:
    line = fp.readline()
    if not line:
        break
    ... process line ...
# have reached EOF

Note that even if you want the newline stripped off the end of the line, you do not want to strip it before you do the 'if not line' check; otherwise you will think that blank lines are the end of the file.

(Speaking from personal experience, this is an embarrassing mistake to make, although you usually catch it fast.)

It's also possible to fix things up with an 'iterfile' routine, like this:

def iterfile(fp):
    while 1:
        line = fp.readline()
        if not line:
            return
        yield line

Then instead of 'for line in fp:', just use 'for line in iterfile(fp):'. And of course you can mix this with regular reads from the file without anything getting too confused. You may still have the double EOF problem, depending on how you structure your program; unfortunately, file objects don't remember if they've seen an EOF, so iterfile() itself can't avoid the problem.


Comments on this page:

From 82.32.138.13 at 2007-12-29 00:56:45:

If you do python -h you will see documentation on this option:

-u     : unbuffered binary stdout and stderr (also PYTHONUNBUFFERED=x)
         see man page for details on internal buffering relating to '-u'

It should help ...

- Paddy3118

By cks at 2007-12-29 14:05:10:

Unfortunately the -u argument doesn't help in this particular case, because the problem is that reading is being buffered, not writes to standard output and standard error.

(I assume that Python defaults to the standard IO system for output buffering, which does line buffered output when the output is to a terminal and block buffered otherwise.)

From 70.17.55.46 at 2007-12-31 08:05:07:

The double-ctrl-D thing can be avoided by changing iterfile to:

def iterfile(fp):
  while not fp.closed:
    line = fp.readline()
    if not line:
      fp.close()
    else:
      yield line

-- DanielMartin, who's too lazy to go find the piece of paper with his CSpace password written on it.

By cks at 2007-12-31 13:01:05:

Unfortunately, fp.closed isn't true after the you've hit end of file; it's only true after you've called fp.close().

Written on 26 December 2007.
« Process memory layout for 32-bit Linux programs
Why I am not entirely fond of Solaris 10 x86's boot archive »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Dec 26 23:33:12 2007
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.