Ubuntu limits the console kernel log level even on servers

February 18, 2022

We have a serial console server that we have connected to all of our important servers, and we have our servers set up so that the serial console is one of the places that Linux kernel messages go when they're printed to the 'console'. Since we log all of the console output, we want all kernel messages to reliably go to the (serial) console. Recently we discovered that our Ubuntu servers were not doing this. Instead, Ubuntu limits the console to log level '4' and higher priority messages, covering what syslog(2) describes as 'warning', 'err', 'crit', 'alert', and 'emerg', and excluding what it describes as 'notice', 'info', and 'debug'. Unfortunately this is not what you want because kernel log levels are relatively meaningless.

As covered in syslog(2), the log level of the console is reported and changed through /proc/sys/kernel/printk, which is also known as the sysctl 'kernel.printk'. By default the kernel sets this to '7 4 1 7', where the first number is the console log level. Since 7 is the lowest kernel log level, the default is to print all kernel messages to the console. In Ubuntu, this is changed by /etc/sysctl.d/10-console-messages.conf to '4 4 1 7', causing the kernel to print only messages of log level 0 through 4 to the console. This sysctl configuration file comes from 'procps'.

Procps is the source of /etc/sysctl.conf and /etc/sysctl.d in both Debian and Ubuntu, but the Debian package only puts a 'README.sysctl' file in sysctl.d. Ubuntu has modified the procps package to add a number of additional non-default sysctl settings, currently (in 20.04):

10-console-messages.conf
10-ipv6-privacy.conf
10-kernel-hardening.conf
10-link-restrictions.conf
10-magic-sysrq.conf
10-network-security.conf
10-ptrace.conf
10-zeropage.conf

(The specific list can change over time.)

It would be better if Ubuntu put all of these in a separate Ubuntu settings package, especially because then they might have a different version of the package for server installs. But this is Ubuntu and my expectations are low.

There is probably a proper Debian way to remove these configuration files in a way where they won't come back no matter what Ubuntu does with package updates. However, we have chosen the simpler approach of having higher-numbered /etc/sysctl.d files that override the Ubuntu custom settings that we don't want. This now includes restoring kernel.printk to '7 4 1 7', so we get all of the console messages.


Comments on this page:

By moschlar at 2022-02-19 06:47:50:

For proper overriding files from packages take a look at dpkg-divert!

Written on 18 February 2022.
« The Linux kernel's message log levels are relatively meaningless
The important things about Unix init systems aren't booting the system »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Feb 18 21:28:49 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.