Plaintext is not a great format for (system) logs

June 29, 2024

Recently I saw some grumpiness on the Fediverse about systemd's journal not using 'plain text' for storing logs. I have various feelings here, but one of the probably controversial ones is that in general, plain text is not a great format for logs, especially system logs. This is independent of systemd's journal or of anything else, and in fact looking back I can see signs of this in my own experiences long before the systemd journal showed up (for instance, it's part of giving up on syslog priorities).

The core problem is that log messages themselves almost invariably come with additional metadata, often fairly rich metadata, but if you store things in plain text it's difficult to handle that metadata. You have more or less three things you can do with any particular piece of metadata:

  • You can augment the log message with the metadata in some (text) format. For example, the traditional syslog 'plain text' format augments the basic syslog message with the timestamp, the host name, the program, and possibly the process ID. The downside of this is that it makes log messages themselves harder to pick out and process; the more metadata you add, the more the log message itself becomes obscured.

    (One can see this in syslog messages from certain sorts of modern programs, which augment their log messages with a bunch of internal metadata that they put in the syslog log message as a series of 'key=value' text.)

  • You can store the metadata by implication, for example by writing log messages to separate files based on the metadata. For example, syslog is often configured to use metadata (such as the syslog facility and the log level) to control which files a log message is written to. One of the drawbacks of storing metadata by implication is that it separates out log messages, making it harder to get a global picture of what was going on. Another drawback is that it's hard to store very many different pieces of metadata this way.

  • You can discard the metadata. Once again, the traditional syslog log format is an example, because it normally discards the syslog facility and the syslog log level (unless they're stored by implication).

The more metadata you have, the worse this problem is. Perhaps unsurprisingly, modern systems can often attach rich metadata to log messages, and this metadata can be quite useful for searching and monitoring. But if you write your logs out in plain text, either you get clutter and complexity or you lose metadata.

Of course if you have standard formats for attaching metadata to log messages, you can write tools that strip or manipulate this metadata in order to give you (just) the log messages. But the more you do this and rely on it, the less your logs are really plain text instead of 'structured logs stored in a somewhat readable text format'.

(The ultimate version of this is giving up on readability in the raw and writing everything out as JSON. This is technically just text, but it's not usefully plain text.)


Comments on this page:

The most recent standard on syslog, RFC 5425 from 2009, has an entire section structured data that may be of interest (though it may not be structured enough for many folks):

ยง6.3.5 has examples.

There are two pet peeves I have with journald: the first being that they chose to create their own file format for whatever reason. Not sure why they did, and maybe there were good reasons for it, but I've never heard them outlined. If they had used something like SQLite it could have been tied into a lot more tooling. Howard Chu of OpenLDAP wrote about this many years ago:

The second pet peeve is that the lack of remote logging support: perhaps for a desktop focused system that's fine, but as a sysadmin I now have to run (r)syslog(-ng) regardless for log shipping.

By Anonymous at 2024-06-30 10:54:04:

Only slightly related, but :

I remember some heated discussions against the use of systemd/journald, for the reason that it doesn't let you 'just grep a textfile'. I never really understood the argument, because even though you cannot do this anymore :

grep foo logfile.txt

you can just as easily do :

journalctl | grep foo

Others never seemed to agree with me though. Oh, well.

The second pet peeve is that the lack of remote logging support: perhaps for a desktop focused system that's fine, but as a sysadmin I now have to run (r)syslog(-ng) regardless for log shipping.

That's not really true, journald itself can send using the ForwardToSocket option, receivable on the other end by systemd-journal-remote or some other service that understands the format

By Michael Kohne at 2024-07-01 19:57:00:

My only real beef with journals is that I could never find a way to get it to give me the messages in the order they hit the log. - important when you are tracking clock hardware and sync issues.

By Yump at 2024-07-27 23:56:06:

I agree that plain text is suboptimal, but journald's log format is such an annoying implementation of other-than-plaintext logging that the whole concept is contaminated by association.

The storage format is mostly UTF-8 and zeros with a blockchain bolted on, so storage efficiency is poor, and the supposed "compression" does almost nothing to improve it. It recently grew the ability to re-write journal files on rotation so that the filesystem can compress them (if you are using a compressing filesystem...), for something like 50% savings, but that incurs write amplification on SSDs.

The blockchain makes the logs tamper-evident if you stored the private key elsewhere ahead of time, but without a very strong suggestion that users do this almost nobody does, and most computer problems aren't enemy action anyway.

Journalctl is ponderously slow, taking several seconds to dump the log-since-boot from my machine with 2 weeks of uptime. So for interactive investigation with multiple queries, it's almost mandatory to dump the log to json and import to something else. Visidata, lnav, or even repeated grepping is faster. (This may be improved in the very latest v256 that I don't have yet.) Since the log isn't a standard structured text format, other tools can't read it directly.

It centralizes logs to a single file with a single disk quota, so there's no way to achieve log retention inversely proportional to verbosity.

The format is almost, but not quite, append-only, and somehow corruptions are possible. Luckily they are handled gracefully aside from causing journalctl to yelp every time it hits one.

Written on 29 June 2024.
« I wish systemd didn't require two units for each socket service
The systemd journal doesn't force you to not have plain text logs »

Page tools: View Source, View Normal.
Search:
Login: Password:

Last modified: Sat Jun 29 22:32:46 2024
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.