Wandering Thoughts archives

Why we've wound up moving away from serial consoles on our machines

Back some time ago we really liked serial consoles here; we configured all of our machines with them, whether they were Linux or OpenBSD or whatever. But lately we've been moving distinctly away from serial consoles, to the point where none of our current generation machines are set up with them any more. We're doing this because in the end serial consoles got in the way of our troubleshooting during serious issues.

What we found is that often the times we deal with machines are when they're bad enough to lock up mysteriously or otherwise need hands-on attention (for example, to swap network wires to make a spare firewall the active one). When we're already physically interacting with a machine to try to figure out the problem, what we found we wanted to do was wheel over a cart, plug in a keyboard and a monitor, and have full interaction with everything from the BIOS on up. We didn't want to have to go back and forth from the physical machine to a desktop that was connected to the console server that the serial console emerged on.

What would be ideal would be a serial console that was a real mirror of everything on the physical console; both would get all kernel messages and all boot time messages from init et al, both could be used as the console in single-user mode, and you could log in on both after boot. But nothing gives us that and if we have to chose one thing to be the real console, the physical console wins. Remote administration is nice and periodically convenient but it's not as important as easy troubleshooting when things really go bad and we're in the machine room trying to deal with it.

(We have the Linux kernel console configured to send messages to both the physical console and the serial console on our Linux machines, so we can at least capture kernel messages during a crash. Unfortunately I believe Linux is the only Unix that can do that. And we're still running gettys on the serial ports so we can log in over them if networking or the ssh daemon has problems.)

PS: IPMIs with KVM over IP are great but they're not a complete replacement for serial consoles. They give us the remote access but not the logging of all console output so that we can look back later to find messages and so on.