2022-07-06
My current mixed views on the Linux kernel netconsole
The Linux kernel's netconsole allows you to "log kernel printk messages over UDP" to a remote system; this makes it another form of kernel (message) console, alongside the video console and serial console. I've looked into netconsole periodically and played with it a bit, and the whole experience has left me with decidedly mixed feelings and very little desire to actually use it in our environment.
The first problem with netconsole in practice is that the story on the receiving side is what you could politely call lacking. The usual advice is to send netconsole messages to either a listening syslogd or to some manual command. No one seems to have built a usable, ready to go 'netconsole server' that would reliably capture netconsole messages, put them in per-host files or the like, and so on (Facebook's netconsd could be extended into this but it doesn't ship with a file logger).
The second problem is that on the sending side, the usage case for netconsole is limited in our environment. If you want to capture normal kernel messages off the machine in normal circumstances, a remote syslog server generally works just as well. If you want to capture kernel messages in crazy situations, you're almost certainly better off with a serial console, including "serial over IPMI", because there are a lot fewer moving parts in the kernel for sending output over a serial link than there are for sending even relatively hard-coded UDP messages.
This reduces netconsole to two situations, where either you can't use a serial console (so netconsole is the best you can do) or you're mostly worried about intermediate kernel situations, ones where the modern and relatively complicated user level remote syslog path isn't going to work but a low level UDP direct transmission probably will. This can sometimes make sense, but then you run into the lack of a good 'netconsole server' setup, which makes the whole thing harder to deal with.
(The modern user level path for remote syslog is that kernel messages are read by systemd's journald, which then passes them to the local rsyslogd, which then writes them off to the remote syslog server. As far as I know there's no way to get journald to directly write messages to a remote server (syslog or otherwise). For a kernel message to make it to your remote syslog server, both journald and (r)syslogd have to run and be able to process the message, and then the network send has to work. In a serious kernel problem, this is in no way certain.)
If there was a good, ready to go netconsole server setup, I might set one up here and configure our machines to send to it, just as a backup (especially for lower priority machines that we don't currently have hooked up to our serial console server). Without one, it's probably not important enough to try to put something together, since our machines almost never panic in the first place and we already have a central syslog server that captures routine kernel messages (along with everything else).
(If there is such a netconsole server out there in the world, it needs better publicity and I would love to hear about it.)