2007-11-18
I love Linux's serial console support
I just rebooted and recovered one of our Linux machines that had hit a
panic and then locked up in an endless cycle of 'soft lockup detected
on CPU #0' reports. I did all of this from home, through the magic of a
serial console, a console server, and Linux's magic SysRq key support.
And as far as I'm concerned, one of the best things about this is that
it took hardly any bandwidth and no Java.
(While I'm currently on DSL at home, it wasn't too long ago that I was stuck at 28.8 Kbps dialup PPP, and even now my DSL flakes out periodically. So I still value being able to do things without saturating my link or fighting through overly graphics laden and occasionally flaky web interfaces.)
Linux's serial console support does have limitations, including that there are multiple levels of console. One of the big ones is that you cannot reboot the machine unless the kernel is at least somewhat responsive, enough to handle magic SysRq keystrokes. But its advantage is that it is general; it works, and works the same, on every Linux machine with a serial port.
(If you don't have a serial port, there's the kernel's netconsole support, but it only sends kernel messages somewhere else; you don't get magic SysRq key support.)
Sidebar: serial consoles and lights out management hardware
To be fair, a serial console isn't the only way to get this sort of thing; it's just the easiest and most universal. On a machine with lights out management hardware, you can power cycle it remotely and even get console access (so-called 'KVM over IP'). But this involves working inside whatever web interface the vendor has come up with and takes quite a bit more bandwidth, especially for the console, and doesn't do anything to capture messages that scroll off the console.
(And a serial console doesn't give me the ability to remotely power cycle a machine. If the machine had been so far gone that it didn't respond to the magic SysRq keys, I would have been cursing the fact that it didn't have an ELOM and that we still haven't gotten around to setting up our smart power units.)