Wandering Thoughts archives

Ubuntu limits the console kernel log level even on servers

We have a serial console server that we have connected to all of our important servers, and we have our servers set up so that the serial console is one of the places that Linux kernel messages go when they're printed to the 'console'. Since we log all of the console output, we want all kernel messages to reliably go to the (serial) console. Recently we discovered that our Ubuntu servers were not doing this. Instead, Ubuntu limits the console to log level '4' and higher priority messages, covering what syslog(2) describes as 'warning', 'err', 'crit', 'alert', and 'emerg', and excluding what it describes as 'notice', 'info', and 'debug'. Unfortunately this is not what you want because kernel log levels are relatively meaningless.

As covered in syslog(2), the log level of the console is reported and changed through /proc/sys/kernel/printk, which is also known as the sysctl 'kernel.printk'. By default the kernel sets this to '7 4 1 7', where the first number is the console log level. Since 7 is the lowest kernel log level, the default is to print all kernel messages to the console. In Ubuntu, this is changed by /etc/sysctl.d/10-console-messages.conf to '4 4 1 7', causing the kernel to print only messages of log level 0 through 4 to the console. This sysctl configuration file comes from 'procps'.

Procps is the source of /etc/sysctl.conf and /etc/sysctl.d in both Debian and Ubuntu, but the Debian package only puts a 'README.sysctl' file in sysctl.d. Ubuntu has modified the procps package to add a number of additional non-default sysctl settings, currently (in 20.04):

10-console-messages.conf
10-ipv6-privacy.conf
10-kernel-hardening.conf
10-link-restrictions.conf
10-magic-sysrq.conf
10-network-security.conf
10-ptrace.conf
10-zeropage.conf

(The specific list can change over time.)

It would be better if Ubuntu put all of these in a separate Ubuntu settings package, especially because then they might have a different version of the package for server installs. But this is Ubuntu and my expectations are low.

There is probably a proper Debian way to remove these configuration files in a way where they won't come back no matter what Ubuntu does with package updates. However, we have chosen the simpler approach of having higher-numbered /etc/sysctl.d files that override the Ubuntu custom settings that we don't want. This now includes restoring kernel.printk to '7 4 1 7', so we get all of the console messages.