Wandering Thoughts archives

2006-08-31

SIGCHLD versus Python: a problem of semantics

In the process of looking at my program's code again to write the last entry, I think I may have solved the mystery of how my impossible exception gets generated.

My program does a lot of forking and thus cleanups of now-dead children. The code that it generally dies on is:

def _delip(pid, ip):
  del ipmap[ip][pid]
  if len(ipmap[ip]) == 0:
    del ipmap[ip]

It takes a KeyError on the len(ipmap[ip]) operation and goes down. (Because of previous fun, the main thread forks all the children and waits for them, so this kills the entire program.)

Clearly there is some concurrency problem, but my problem with the exception was that I've never seen where it could come from. The main thread is the only thread that adds or removes things from the ipmap dictionary, and the SIGCHLD handler that reaps children is only active when the thread is idling in select() (partly to avoid just this sort of concurrency issue).

To avoid various problems and just create sanity, Unix SIGCHLD handlers are not reentrant; even if more children die, you won't receive a second SIGCHLD until you return from the signal handler. (This is an interesting source of bugs if you bail out of the signal handler without telling the kernel, and is one reason for the existence of siglongjmp().)

And in thinking about all of this I came to a horrible realization: those are Unix semantics, not Python semantics. Python does not run your Python-level SIGCHLD handler from the actual C level signal handler; it runs them from the regular bytecode interpreter. All the C level SIGCHLD handler does is set a flag telling the interpreter to run your SIGCHLD handler at the next bytecode, where it gets treated pretty much as an ordinary function call.

This would neatly explain my mysterious exceptions. When there are two connections from an IP address and both of them die in short succession, if we are extremely unlucky the SIGCHLD for the second will be processed between _delip's first and second lines and delete the ipmap[ip] dictionary entry out from underneath the first.

I personally believe that this is a bug in the CPython interpreter, but even if I can persuade the Python people of this, I still need to come up with a Python-level workaround for the mean time (ideally one that doesn't involve too much code reorganization).

python/SIGCHLDVsPython written at 23:32:37; Add Comment

How dd does blocking

For a conceptually simple program, dd has a number of dark corners. One of them (at least for me) is how it deals with input and output block sizes, and how the various blocking arguments change things around.

  • ibs= sets the input block size, the size of the read()s that dd will make. Since you can get partial reads in various situations, this is really the maximum size that dd will ever read at once.
  • obs= sets the output block size and makes dd 'reblock' output; dd will accumulate input until it can write a full sized output block (except at EOF, where it may write a final partial block).
  • bs= sets the (maximum) IO block size for both reads and writes, but it turns off reblocking; if dd gets a partial read, it will immediately write that partial block.

Because of the reblocking or lack thereof, 'ibs=N obs=N' is subtly different from 'bs=N'. The former will accumulate multiple partial reads together in order to write N bytes, while the latter won't.

(On top of this is the 'conv=sync' option, which pads partial reads.)

So if you're reading from a network or a pipe but want to write in large efficient blocks, you want to use obs, not bs (and you probably want to use ibs too, because otherwise you'll be doing a lot of 512 byte reads, which are kind of inefficient).

sysadmin/DdBlocking written at 19:14:44; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.