How our console server setup works

February 4, 2015

I've mentioned before that we have a central console server machine where all of our serial consoles and other serial things all get centralized, automatically logged, and so on. While I don't think we're doing anything unusual in this area, I've realized that doing decent sized console servers is probably no longer common and so it might be interesting to describe how ours works.

The obvious way to do a (serial) console server is just to build a machine with a bunch of serial ports. This kind of works at small or moderate scale, but once you're talking about thirty or fifty or a hundred or more serial ports, things break down. There are two problems with this; first, you just can't fit that many serial ports into one piece of hardware for sane amounts of money, and second, you can't feasibly run serial lines to everything in remote locations (like master switches in building wiring closets and so on).

The thing that makes it possible to deal with all of this is serial port to Ethernet concentrators; we use various models of Digi's Etherlite series, generally the rack-mountable 16 and 32 port versions. These have some number of RJ-45 ports which we plug serial connections into and an Ethernet port over which the system talks to their software on our console server, where a Digi kernel module turns those networked serial ports into /dev/... serial port entries that look just like hardware serial ports.

On the console server we use conserver to manage the serial ports; it logs their traffic, handles actual interactive access to them, and so on. Conserver is probably not the only system for this (and may well not be the best); it's just what we use. It works and you can probably find it packaged for your Linux distribution of choice.

(As far as I know there's nothing that will directly talk the Digi Etherlite protocol so you can cut out the middleman of the fake kernel serial ports. I believe this is partly because the protocol is at least undocumented. It's possible that there are other serial port to Ethernet concentrators with documented protocols and thus direct support in projects like conserver.)

If we need serial ports in a remote location, for example to give access to a switch's console, we put an Etherlite in the location and connect it up. The serial connection to the Etherlite is subject to reasonable length limitations but obviously the network traffic is not. We run Etherlites and similar things over a physically separated and independent management network (described at the end of here).

Sidebar: How you connect serial ports to Etherlites

Etherlites don't have conventional serial ports; instead they use plain RJ-45 ports for higher density (which makes them look like Ethernet switches). Plain RJ-45 is also known as 'Ethernet connectors', so we wire things up using ordinary cat-5 Ethernet cables that plug into RJ45 to DB9 adaptors, which then plug into the servers.

(I don't think we have anything left with full sized serial ports; these day's it's DB9 or nothing. Fortunately servers are still coming with DB9 serial ports.)

I don't think you need to use full bore Ethernet cables for this. We just happen to already have everything we need to make cat-5 Ethernet cables already, so this way we keep everything standardized. To avoid confusion we use a special colour.


Comments on this page:

By anon at 2015-02-04 06:47:23:

Interesting... I haven't seen set ups like your in a long time. Most sites I've seen use mgmt cards like ILOMs which are connected over ethernet and provide Serial Over LAN among many other things, not to mention a build-in KVM.

By cks at 2015-02-04 10:52:37:

The big advantage of a setup like ours with servers that already have ILOMs and Serial Over Lan is that you can easily log all of their console output. This means that you can catch things like kernel messages that the machine dumped out before it crashed.

(The other advantage is that this setup works for servers that don't have ILOMs and for non-server things like network switches.)

By utor-pk at 2015-02-04 13:18:10:

Digi Etherlites sound like a nice evolution from the days when we cobbled together a console server using an idle Sun-3/280 with ALM-2 16-port modules and then hacked its tip(1) command to log the interactive serial I/O.

Written on 04 February 2015.
« Why we've wound up moving away from serial consoles on our machines
All of our important machines are pets and special snowflakes »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Feb 4 01:25:02 2015
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.