On sending all syslog messages to one file

July 18, 2021

Over on Twitter, I had a view on where syslog messages should go:

Tired sysadmin take: Different sorts of syslog messages going to different places are a mistake. Throw it all into /var/log/allmessages and I'll sort it out myself.

Like many Twitter takes of mine, in retrospect this one is heartfelt but a little bit too extreme as presented. Specifically, I think you should log all syslog messages to one place, but also log some sorts of messages to their own additional places so you can look through them more easily.

In the old days, I used to carefully curate my syslog.conf so that every different syslog facility had its own different file. Often, the net result of this is that I would end up using grep on every current syslog file in /var/log because I'd forgotten (or never knew) what facility a given program logged under. Trying to predict what facility a program will use is often almost as futile as predicting what priority level messages will be logged under.

(This is worse if you rely on the Unix vendor stock syslog.conf instead of customizing it. Unix vendors are inevitably different from each other, and some of them have rather strange ideas of what should go where.)

All of this leads to the tired sysadmin take of putting everything into one file (/var/log/allmessages is what I prefer) and then searching it. An allmessages file is the brute force solution to unpredictable programs and Unix vendor variability, and it also makes sure everything gets logged. But sending all syslog messages to only a single place is a little bit of overkill. Despite my tired take, there are often syslog facilities that it's sensible to also log to separate files, so you can look at just them.

The obvious case is kernel messages, and it's so obvious that systemd's journalctl has a dedicated flag to show you only kernel messages. If I was starting a syslog configuration from scratch, I would also have a log file dedicated to "auth" and "authpriv" messages, one dedicated to "mail" messages, and on my own systems, one dedicated to "daemon" messages. Everything would still go to allmessages; these files are in addition to it.

(And on some systems you might opt to have specific programs log to specific facilities, like "user" or "local0", and have specific files so you can monitor and see the activities of just those programs.)

Sending all syslog messages to an allmessage file is a blunt hammer, and like all blunt hammers it's possible to overuse it. Being able to scan through a single file that has everything has a lot of positive features, but not everything is best served by searching for it through a giant file. Sometimes you want both options.

Comments on this page:

By Ingo at 2021-07-19 04:28:27:

A long time ago, I used to have a syslog configuration that logged all of a program's messages to /var/log/programs/<programname>.log in addition to the usual files. It was pretty useful, but I stopped using it because it was yet another thing that had to be merged manually during Debian upgrades.

One issue with logging messages to separate locations is that the ordering of messages across locations is lost unless specifically recorded. Syslog preserves much of it by timestamping messages, but the order reconstructed from timestamps is limited by their resolution – typically one second in case of syslog. So the ordering of messages logged during the same second but to different files is lost.

The nice thing about an everything logfile is that it automatically preserves the exact original order of messages (or at least, to be pedantic, the order in which syslog logged them) – no additional thought or measures required.

It’s not something you’re likely to need very much, but is something you might need unexpectedly – and with an everything logfile you get it basically for free.

Written on 18 July 2021.
« The minimum for syslog configurations should be to log (nearly) everything
Making a Go program build with Go modules can be not a small change »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Jul 18 23:13:42 2021
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.