Wandering Thoughts archives

A Bourne shell irritation: piping just stderr

I generally like the Bourne shell, but I will easily admit that it has a number of things that are less than ideal. One of those less than ideal things is a peculiar omission: you cannot easily redirect just standard error into a pipeline.

This sounds like a peculiar thing to want, but there are situations where you do need to process stderr separately; for example, you might need to timestamp all stderr output and log it. In a hypothetical shell with this feature you could have a 'timestamper' program and just write:

process |[2] timestamper >>error-log

In the Bourne shell, not so much. You can do it, but it is what you would call intricate:

exec 3>&1
process 2>&1 >&3 3>&- | timestamper >>error-log

What's going on here is that first we save a copy of the real stdout on file descriptor 3, and then in the command itself we:

• send stdout to the timestamper process (remember, pipelines are set up before redirection)
• redirect stderr to stdout, which is the pipeline
• redirect stdout to fd 3, which is our original stdout
• close fd 3 so that process does not inherit a surprise file descriptor.

(We must save the original stdout separately via exec because of the pipeline vs redirection timing issue; if we wrote 'process 3>&1 ...' we would be capturing the pipe'd stdout, not the original stdout.)

(This incantation is not original to me; I've seen it written up elsewhere on the Internet, but I never really understood it until I wrote out everything here (sort of cf, especially the comment).)

Lighttpd, CGIs, and standard error

Here's an issue that I ran into recently: from version 1.4.20 onwards, lighttpd has stopped passing stderr to CGI processes, should one be peculiar enough to run CGIs on lighttpd (which can be done). If you need error logging from your CGIs, you will have to roll something on your own.

(Fedora 10 has lighttpd 1.4.22 and so is affected by this.)

There is a bug report for this, but I don't know if it has been accepted by the lighttpd developers as an actual bug that will get fixed. That two versions of lighttpd have been released with this issue suggests that, as a minimum, it is not being considered an important issue.

(I suspect that relatively few people use lighttpd to run plain CGIs; most people probably use FastCGI or SCGI instead if they need dynamic things.)

The simplest thing is to just write a cover script for your CGI that dumps standard error into a logfile, but this means that it won't be timestamped. If you need that, you'll have to post-process your program's stderr while leaving standard output alone, which the Bourne shell unfortunately makes rather difficult.

(My solution was to write a program that ran a subordinate command while capturing and timestamping its stderr and leaving its stdout alone. This is relatively trivial in anything that has access to Unix system calls; I wrote mine in C for obscure reasons.)