Wandering Thoughts archives

A gotcha with 'bidirectional' pipes to commands

By bidirectional pipes I mean a situation where you start a subordinate program and both write to its standard input and read from its standard output. There are a fair number of programs that are programmed like the following pseudo-code:

to, from = pipecmd("subprogram", "rw")
write(to, stuff)
reply = read(from)

People write these programs, test them a bit, start using them, have them work, and then one day wake up to discover that their program has locked up; it and the subprogram are both running, doing nothing. (A system call tracer will tell them that both programs are blocked in write().)

It's not really surprising that people write pipes like this; this code is simple, obvious, and works most of the time. Specifically, it works as long as the subprogram reads almost all of its input before producing enough output to fill up its standard output pipe. After that the subprogram blocks trying to write output, because no one is reading it, and the main program blocks trying to write input to the subprogram, because the subprogram is blocked writing output.

(Often this means that the program works in testing and early in its production use, when it is only being asked to deal with relatively small amounts of data, and it is only when things grow that it blows up.)

Unfortunately there is no good simple way out of this. The best but most complicated approach is to make your program use a select() or poll() based loop to simultaneously write to the subprogram and read its output. The brute force way is to use either fork() or threads to let you do the write() and the read() at the same time.