Wandering Thoughts archives

Never kill the screen locker

This is a grump.

Dear X applications (or any application on any window system): you should never, ever kill or otherwise force-terminate the screen locker. In particular, you cannot assume that just because you have been run that the user is sitting there in front of the screen and wants it unlocked. There are any number of ways that you can start up when the user is not present, and unlocking the screen in this situation can easily make things go horribly wrong.

For one example, perhaps some resource such as an NFS filesystem wasn't available and your program hung as it was starting, so the user gave up, locked their session, and walked away from the computer to do something productive. When the NFS server recovers, your NFS IO suddenly starts working, and you actually start up, the user is going to be very peeved to come back to their machine being unlocked.

(I'm pretty sure that I've seen KDE applications do this. The situation that caused it aren't clear to me, since I wasn't in front of the machine at the time, but I believe that the application was either trying to show a splash screen or trying to show a 'I have run into some problem, do you want to send a bug report' note.)

I'm sure it's tempting to decide that your program has something so important to communicate to the user that they need to see it right now and you need to make it visible. However, you're wrong; it can wait until we know that the user is present because they've just typed in their unlock password.

(The sole exception to this is if the system itself is about to crash or reboot, and thus your message literally cannot wait because it's about to disappear in a short time no matter what. But you'd better be very sure that you're not about to leave the user's session unlocked for any more than a very short time.)

Sidebar: starting an X application remotely

Suppose that you are ssh'd in to your workstation and you want to start an X application, displaying on your workstation's screen. The easy incantation to do this (in Bourne shell) is:

DISPLAY=:0 x-program &

You may need to do something to get the program to live happily in the background, and you may have to specify where to put the program's window (depending on your window manager and the state of your system at the time; many window managers now auto-place new windows without user intervention).

(In theory many X programs accept a command line argument to specify the X display to talk to, but with so many toolkits and systems these days, who knows what command line argument any particular program accepts and what restrictions it has. Setting $DISPLAY is much simpler and it works for everything.)

The advantage of garbage collection for APIs

Here's a question that interests me: why don't I have a standard C version of my Python warn() function? The Python version shows up in most programs I write, but in my C programs I generally directly fprintf() errors to stderr, instead of having a utility function for it.

Part of the reason is that a C version of warn() really calls for a different and more complicated set of arguments. Instead of a single string, the natural C-ish approach is printf()-like, creating a prototype of 'warn(const char *fmt, ...)'. In turn this makes warn() a non-trivial function, because varargs functions in C are vaguely annoying.

(Things get even more interesting when one implements die(), since you can't have it call warn(); either you duplicate warn()'s code or you wind up needing a vwarn() with the prototype 'vwarn(const char *fmt, va_list ap)' that both warn() and die() call.)

But why does the C version of warn() need a different API than the Python one? One answer is that Python has a first class operator to format printf-like strings (the string '%' operator, in a marvelous abuse of operator overloading) and C doesn't, so in Python it's idiomatic to format strings directly in your code, instead of handing the format and the arguments to another function. This isn't the whole story, though, since C has sprintf() and could do much of the same tricks if people wanted to.

The problem with sprintf() (and why in practice it is not used for this) is that you have to give it a buffer. If you use sprintf() you have to think about how big a buffer you need and what you do if it isn't big enough. If you use fprintf(), you don't have to think about all of that, so people use fprintf(); in C, people will go to a lot of effort to avoid having to think about buffer issues.

And the reason that sprintf() needs you to supply the buffer is so that no one has to worry about allocating and especially deallocating the buffer, because memory management is a pain in the rear in C. Not just because you have to do it, but also because you have to decide where it's done and which function has to look after it and this generally complicates the code. (For example, warn() can't just free the string it's passed, which means that you can't write 'warn(strformat(fmt, ...))' because this leaks the string.)

Thus I come to the conclusion:

A subtle advantage of garbage collection is that it enables APIs that would otherwise be impossible, or at least dangerous.

Garbage collection is the essential glue that enables my Python versions of warn() and die() to have their simple APIs, because it is what makes Python's '%' operator convenient (among many other Python APIs). You could implement all of the operations of the Python version in C, but you shouldn't; in a non-GC'd language, object allocation is to be avoided as much as possible, and certainly turned into someone else's problem.

(Hence sprintf() does not allocate anything; finding a buffer for it is your problem.)

How to write to stderr so people will like you

Suppose that you're writing a command like, say, make; it writes progress output to standard output and error reports to standard error, and it doesn't necessarily immediately terminate when it encounters an error (or prints a warning; the important thing is that it keeps running after it prints things to standard error).

There's a subtle catch in how you want to print things (such as standard error messages) to stderr in such a program. You don't want to just do the straightforward thing of printing to stderr and being done with it. Instead, you want to print messages to stderr with the following sequence: flush stdout, write your message to stderr, then flush stderr.

(This is obviously inapplicable if all of your IO is complete unbuffered and there is nothing to flush. But this is an uncommon case.)

To convince you of this, here's a little test program to illustrate what goes wrong if you don't:

$ cat tst.c
#include    <stdio.h>
int main(int argc, char **argv)
{
    printf("line 1\n");
    fprintf(stderr, "stderr line 1\n");
    printf("line 2\n");
    fprintf(stderr, "stderr line 2\n");
}

And now let's show what goes wrong:

$ ./tst >/tmp/logit 2>&1; cat /tmp/logit
stderr line 1
stderr line 2
line 1
line 2

(Note that these results are system and language dependent; your choices may behave differently.)

That's not exactly the output that you'd expect or that you want, seeing as the error messages have completely been relocated out of their actual context.

In most implementations of buffered IO, interactive IO is line buffered (it is written out to you after every newline) while IO to anything else is block buffered, generally in reasonable size blocks. The problem is that when you redirect the output of this program into a file, stdout is buffered separately from stderr even though they have been redirected to the same place. As a result, at the end of tst you get all of the stdout messages flushed out in one block, not interleaved with the stderr messages.

(Note that on some systems and in some languages stderr is always line buffered or unbuffered, even when it is being written to files.)

Doing the flush dance defeats this separate buffering and so makes stdout and stderr be properly interleaved even when redirected into a file. Note that it may not sufficient to just flush stdout before writing to stderr, because if stderr is block buffered there's nothing that prevents subsequent messages to stdout being flushed out before the stderr message.

(Note that programs that run other programs need to do more than just this; they need to flush stdout and perhaps stderr before they start another program.)

The processing flow of a network copying program

For my sins, I have dabbled in writing netcat-like programs for some time, things that take standard input, send it off to somewhere over the network, and write to standard out what they get back from the network. In the process I have formed very definite opinions about how these programs should behave in order to be most useful (in scripts and so on), and I feel like writing it down.

For the most part, a network copying program is straightforward; you use select() or poll(), handle buffering carefully, and bounce data back and forth. The tricky things are how you handle various ways that the conversation can end:

• when you see end of file on standard input, signal to the network that there is no more input coming but keep processing network input. On standard TCP connections, you do a shutdown(fd, SHUT_WR); I don't know what you do with SSL, but I hope there's some similar equivalent. (You're on your own with UDP.)

  You have to keep reading from the network because it may well still be sending you output, especially in a scripted situation where you are creating an immediate end of file on standard input by doing things like:

  (echo A; echo B) | tcp host port

  You have to tell the network server that you've stopped sending things, because otherwise some network servers will never close the connection; you'll just sit there forever. Also, it makes it simpler to use the program to do certain sorts of checks (eg, to see if a port is answering at all) without needing special program features.

• when you see end of file on network input, just exit. This is not symmetrical with the handling of standard input, but it turns out to be the most useful in general; there are very few situations where a network server will shut down only the output direction of a TCP conversation. (If you think you're going to run into one of those, feel free to put a switch into your program to control this.)

Doing a really correct implementation of buffering in a network copying program is a big pain in the rear end and most of the time you will never need it, so feel free to cheat here. If you really want to do it correctly, my approach was to have a buffer for each sink (network output and standard output) and to stop poll()'ing on the corresponding source (standard input and network input) when a sink's buffer filled up. I guarantee that you will have a bunch of twitch inducing logic about polling and not polling sinks and sources under various sorts of circumstances.

(I know this because I once decided to simultaneously learn the bstring library and poll() at the same time by writing a neurotically correct buffering network copying program that used bstrings as the buffering mechanism. I am fond of my btcp program (source here) in the abstract, but I never want to have to think about the logic again.)

(PS: in the credit where credit is due department, I learned the shutdown() on stdin EOF trick (a very long time ago) from Marc Moorcroft.)