My current mixed views on the Linux kernel netconsole

July 6, 2022

The Linux kernel's netconsole allows you to "log kernel printk messages over UDP" to a remote system; this makes it another form of kernel (message) console, alongside the video console and serial console. I've looked into netconsole periodically and played with it a bit, and the whole experience has left me with decidedly mixed feelings and very little desire to actually use it in our environment.

The first problem with netconsole in practice is that the story on the receiving side is what you could politely call lacking. The usual advice is to send netconsole messages to either a listening syslogd or to some manual command. No one seems to have built a usable, ready to go 'netconsole server' that would reliably capture netconsole messages, put them in per-host files or the like, and so on (Facebook's netconsd could be extended into this but it doesn't ship with a file logger).

The second problem is that on the sending side, the usage case for netconsole is limited in our environment. If you want to capture normal kernel messages off the machine in normal circumstances, a remote syslog server generally works just as well. If you want to capture kernel messages in crazy situations, you're almost certainly better off with a serial console, including "serial over IPMI", because there are a lot fewer moving parts in the kernel for sending output over a serial link than there are for sending even relatively hard-coded UDP messages.

This reduces netconsole to two situations, where either you can't use a serial console (so netconsole is the best you can do) or you're mostly worried about intermediate kernel situations, ones where the modern and relatively complicated user level remote syslog path isn't going to work but a low level UDP direct transmission probably will. This can sometimes make sense, but then you run into the lack of a good 'netconsole server' setup, which makes the whole thing harder to deal with.

(The modern user level path for remote syslog is that kernel messages are read by systemd's journald, which then passes them to the local rsyslogd, which then writes them off to the remote syslog server. As far as I know there's no way to get journald to directly write messages to a remote server (syslog or otherwise). For a kernel message to make it to your remote syslog server, both journald and (r)syslogd have to run and be able to process the message, and then the network send has to work. In a serious kernel problem, this is in no way certain.)

If there was a good, ready to go netconsole server setup, I might set one up here and configure our machines to send to it, just as a backup (especially for lower priority machines that we don't currently have hooked up to our serial console server). Without one, it's probably not important enough to try to put something together, since our machines almost never panic in the first place and we already have a central syslog server that captures routine kernel messages (along with everything else).

(If there is such a netconsole server out there in the world, it needs better publicity and I would love to hear about it.)

Comments on this page:

By Andrew at 2022-07-07 09:27:17:

I do believe that rsyslog would work to be your "netconsole server". imudp obeys the syslog protocol recommendation that any UDP packet received on the listen port is considered a message, even if it's not "syslog formatted" with a priority, timestamp, etc. Rsyslog templates can be used to get the file-per-host behavior that you asked for, and also to format a received timestamp into the output file (since the netconsole messages don't contain one).

By Arnaud Gomes at 2022-07-07 11:35:58:

We use rsyslog as a receiver, it Just Works(tm). Not that we use this source of messages much, but it is easy enough to set up that it's not worth not doing.

On the other hand, we tend to avoid serial consoles because they are mostly slow and, at least in some old kernel versions, blocking. When we used them on heavily loaded machines, the time needed to dump the process list during an OOM-kill could block everything else long enough to make keepalived miss a couple of steps and trigger a VRRP failover (if we were lucky -- this kind of situation can easily turn into a split brain).

   -- A
By Seth at 2022-07-07 12:48:37:

Another recommendation for rsyslog. We just run a separate instance listening on an alternate port so we can do the following:

 $template netconsole,"/var/lib/console/%FROMHOST%/netconsole-%$YEAR%%$MONTH%%$DAY%"
 *.* -?netconsole
By Zouppen at 2022-09-14 05:57:55:

I've written a small tool to collect logs from multiple netconsoles to a systemd-journald of one system and tags each line with the source host name.

It lacks extended console format parsing but that should be very straightforward to add.

You're probably not the target use-case for NetConsole.

NetConsole is invaluable for embedded devices. These devices have no display output, limited "management capabilities" (like IPMI), and often, you don't want to constantly be plugging into UART lines to debug why a kernel isn't booting (e.g. because the device is remote, or inaccessible).

NetConsole gives us insight into why a device isn't booting, well before rsyslog/syslog-ng starts up and is able to forward messages (unless you bake it into the initramfs).

Written on 06 July 2022.
« A surprise: you can only have one Linux kernel serial console
DKIM signature types (algorithms) that we see (as of July 2022) »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Jul 6 22:29:49 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.