Chris's Wiki :: blog/sysadmin/DroppingSerialConsoles

Why we've wound up moving away from serial consoles on our machines

February 3, 2015

Back some time ago we really liked serial consoles here; we configured all of our machines with them, whether they were Linux or OpenBSD or whatever. But lately we've been moving distinctly away from serial consoles, to the point where none of our current generation machines are set up with them any more. We're doing this because in the end serial consoles got in the way of our troubleshooting during serious issues.

What we found is that often the times we deal with machines are when they're bad enough to lock up mysteriously or otherwise need hands-on attention (for example, to swap network wires to make a spare firewall the active one). When we're already physically interacting with a machine to try to figure out the problem, what we found we wanted to do was wheel over a cart, plug in a keyboard and a monitor, and have full interaction with everything from the BIOS on up. We didn't want to have to go back and forth from the physical machine to a desktop that was connected to the console server that the serial console emerged on.

What would be ideal would be a serial console that was a real mirror of everything on the physical console; both would get all kernel messages and all boot time messages from init et al, both could be used as the console in single-user mode, and you could log in on both after boot. But nothing gives us that and if we have to chose one thing to be the real console, the physical console wins. Remote administration is nice and periodically convenient but it's not as important as easy troubleshooting when things really go bad and we're in the machine room trying to deal with it.

(We have the Linux kernel console configured to send messages to both the physical console and the serial console on our Linux machines, so we can at least capture kernel messages during a crash. Unfortunately I believe Linux is the only Unix that can do that. And we're still running gettys on the serial ports so we can log in over them if networking or the ssh daemon has problems.)

PS: IPMIs with KVM over IP are great but they're not a complete replacement for serial consoles. They give us the remote access but not the logging of all console output so that we can look back later to find messages and so on.

Comments on this page:

By SR at 2015-02-03 01:37:31:

Hi,

Thanks for the interesting article!
>PS: IPMIs with KVM over IP are great but they're not a complete replacement
>for serial consoles. They give us the remote access but not the logging of  
>all console output so that we can look back later to find messages and so on.
Could you elaborate on this (maybe in another post). Thought this was exactly what IPMI SOL (Serial Over Lan) is about, together with software console servers like conman, conmux, conserver,.... or hardware ones.

By informatimago at 2015-02-03 03:52:48:

You can't say that nothing gives you that. There are console servers that forward several consoles thru the network. http://www.tripplite.com/products/series/sid/764 That said, it's rather expensive.

By cks at 2015-02-03 08:13:05:

Serial over LAN has the 'pick one' problem of serial in general; you can have the physical console or the serial console, but not both. It just gives you another method of getting the serial console to somewhere else. My personal view is that SOL is mostly interesting for places that don't already have a well-developed infrastructure for handling serial consoles. Our infrastructure can get any serial console to a console server that logs all serial output and gives us remote access to it through the console server.

(For scale, it appears that we currently have over a hundred serial consoles defined. We use Digi Etherlites to turn serial lines into network traffic and conserver on the console server host to manage things.)

By contrast, KVM over IP gives you both the physical console and a remotely accessible version of the physical console; there's no 'pick one' problem because as far as the host knows there's only the physical console. But since it's just the physical console you don't have logging of the console output; if you're not there to see a message fly by, you lose. Need to see a boot message that was scrolled off the screen by a parade of error messages? You're out of luck.

(Apparently some KVM over IP systems hack around this by taking screenshots and saving them for a while. This is only a hack workaround and is quite inferior to real serial console logs, which you can (for example) grep through and cut & paste from and scan rapidly.)

Written on 03 February 2015.

«	Why people were enthused about gcc early on in its life

How our console server setup works