Wandering Thoughts archives

screen -x

I have an confession: I don't really like screen. I never have; it's always struck me as an poor substitute for a real windowing system, something you only resort to when you can't get the real thing for one reason or another.

(What really boggles me is the people who aggressively use screen inside X, winding up with one terminal window with everything inside a screen session.)

One of the most painful limitations of screen for me has been that you can't see two screen-bits side by side; you have to keep flipping back and forth. (Naturally, side by side windows are one of the big things I use real windows for.)

Then someone I know happened to mention screen's -x option. To quote the manpage:

-x: Attach to a not detached screen session. (Multi display mode).

Translated into ordinary language, this lets you have multiple windows connected to the same screen session, where they can then view different screen-bits. Or simpler: side-by-side viewing of two different screen-bits; I can get the benefit of windows and screen at once, if I want to.

The one limitation is that every window really needs to be the same size, or rather peculiar things start happening. (I like both very large windows and standard 80x24 windows, so this actually matters to me.)

Respecting Unix signals

There is a certain class of program that decides it knows better than you about when you should be able to stop its operation. The most irritating way to do this is to catch ^C (and even ^\); such programs win a modest prize from me, usually awarded with 'kill -9'.

Yes, some operations are dangerous if interrupted half-way through and programs need to protect themselves against them. Short operations. It is very rare that a program needs to block signals for very long, and when it does it should take receiving a ^C as a sign that it needs to shut down as soon as it can.

(The Solaris patch process may block ^Cs for a long time, but that's more a sign that it's achingly slow and desperately needs to be fixed than anything else.)

The most irritating programs pay some attention to ^C, but not enough to actually exit or stop. They'll notice your desire for them to stop, but reinterpret it as, oh, 'interrupt the current network transfer' instead of actually stopping.

A tragic example of this is Red Hat's yum, which mostly ignores ^C (and entirely ignores ^\) during precisely the time it would be most useful: when doing a series of network fetches. Nothing vital is going on at this point and it can be quite time-consuming, yet rather than exiting on ^C yum takes it as a signal to abort the current transfer and start fetching the next file. Since you may be fetching 30 files, this is more than a bit annoying.

An example of handling this issue well is mpg123, the command line MP3 player. When it's playing multiple MP3 files, one ^C will stop the current file and skip to the next, but two ^C's in quick succession will cause it to just exit. (Some experimentation suggests that it does this by just exiting on ^Cs in the first few seconds of a file, which is simple enough to implement.)

An odd choice of hostname

Today's winner of the amusing hostname prize is 'GE-queer-A.gay.it' aka 'server1-gay-it.it.net', which is what the IP address 151.1.32.3 currently claims to be in SMTP (the first name) and its PTR records (the second name; there is no matching A record).

I do have to wonder what sort of naming scheme gave rise to this. (It has neighbors with different 'serverN' PTRs, so it's not just one machine.)

/etc/inittab versus /etc/rc.d

One aspect of yesterday's problem is a little ordering issue between /etc/inittab and stuff in /etc/rc.d. Namely:

Everything in /etc/rc.d runs before other things in /etc/inittab really takes effect.

(The paths here are for Red Hat; elsewhere, what Red Hat puts in /etc/rc.d is just put straight into /etc, so you have /etc/init.d and /etc/rc3.d and so on.)

Normally the only effect this has is things like being able to log in via ssh well before you can log in on the console. In the case yesterday, the machine had one QMail listener being started via rc.local as part of the regular rcN.d startup scripts, and another one being started through Dan Bernstein's svcscan system that's kicked off directly from a line in /etc/inittab; the former won, to my confusion when I was looking at the latter.

(The init and inittab documentation doesn't really say what order /etc/inittab entries are run in, but I suspect that in the order they occur in the file is not a bad assumption. The lines that run the system startup scripts are usually very early in inittab.)

Reading /etc/inittab and following everything there can be interesting in general. Almost all of the bits and pieces are shell scripts, and it really gives you an understanding of just what the system does when. (And every so often it's useful to know the commands to force a poweroff or reboot right now.)

Dispelling a nightmare (a sysadmin tale)

One quiet sysadmin nightmare is discovering that you don't know how to reproduce the running setup on a system that you've taken over. You know, things like: there's nothing that seems to start important programs; programs that the configuration seems to say should be running aren't (or aren't running successfully); programs are running as UIDs that aren't even in /etc/passwd.

I spent part of today mostly dispelling a nightmare like that, and therein lies a tale.

I discovered the problem when I was looking into adding more DNS blocklists to the system's QMail configuration. This required finding the configuration, and rapidly led to the discovery that the running SMTP listener didn't seem to be the configured SMTP listener. I did some poking at the time and wound up thinking that it had been started by hand (or by something that had been removed since then, which is almost as much fun).

The last thing you want to do with a working house of cards like this is touch anything (touching stuff is how you get to have exciting days), so at the time I just carefully saved the output of 'ps augxwww' and tiptoed away. (This is often all that you can really do: save as much about the current system state as possible, in the hopes that you'll have enough information to reconstruct and restart things if something dies.)

But this Tuesday morning we had a large scale power failure. When I looked at the mail machine to see how broken it was, to my surprise everything was actually running the way it had been before the power failure. Clearly I'd been wrong, and somewhere there was a startup script; I just had to find it.

Brute force to the rescue. The running QMail command line had a relatively distinctive string in it, so:

# find / /var -xdev -type f -print0 | xargs -0 fgrep -l <string>

This turned up the script that was starting the actual daemons, and some poking around found an invocation of the script tucked away innocently at the end of /etc/rc.d/rc.local.

Now I at least have a starting point for rationalizing things a bit, and I can see how it's working. (Well, mostly how it's working. There's still several mysterious bits.)