Wandering Thoughts archives

Structured log formats are not really "plaintext" logs

As sort of a follow on to how plaintext is not a great format for logs, I said something on the Fediverse:

A hill that I will at least fight on is that text based structured log formats are not 'plain text logs' as people understand them, unless perhaps you have very little metadata attached to your log messages and don't adopt one of the unambiguous encoding formats. Sure you can read them with 'less', sort of, but not really well (much less skim them rapidly).

"Plaintext" logs are a different thing than log formats that are stored using only printable and theoretically readable text. JSON is printable text, but if you dump a sequence of JSON objects into a file and call it a 'plaintext log', I think everyone will disagree with you. For system administrators, a "plaintext log" is something that we can readily view and follow using basic Unix text tools. If we can't really read through log messages with 'less' or follow the log file live with 'tail -f' or similar things, you don't have a plaintext log, you have a text encoded log.

Unfortunately, structured log formats may produce text output but often not plaintext output. Consider, for example:

ts=<...> caller=main.go:190 module=dns_amazonca target=8.8.8.8:53 level=info msg="Beginning probe" probe=dns timeout_seconds=30
ts=<...> caller=dns.go:200 module=dns_amazonca target=8.8.8.8:53 level=info msg="Resolving target address" target=8.8.8.8 ip_protocol=ip4
[...]
ts=<...> caller=dns.go:302 module=dns_amazonca target=8.8.8.8:53 level=info msg="Validating RR" rr="amazon.ca.\t17\tIN\tA\t54.239.18.172"

This is all text. You can sort of read it (especially since I've left out the relatively large timestamps). But trying to read through all of these messages with 'less' at any volume would be painful, especially if you care about the specific values of those 'rr=' things, which you're going to have to mentally decode to see through the '\t's (and other characters that may be quoted in strings).

There are text structured log formats that are somewhat better than this, for example ones that put a series of metadata labels and their values at the front then end the log line with the main log message. At least there you can look at the end of the line in things like 'tail' and 'less' to see the message, although it may not be in a consistent column. But the more labels there are, the more the message text gets pushed aside.

One of the most common example of a plaintext log format is the traditional syslog format:

Jul  1 17:58:53 HOST sshd[PID]: error: beginning MaxStartups throttling
Jul  1 17:58:53 HOST sshd[PID]: drop connection #10 from [SOMEIP]:36039 on [MYIP]:22 past MaxStartups

This is almost entirely the message with relatively little metadata (and a minimal timestamp that doesn't even include the year). This is what you need to maximize human readability with 'less', 'tail', and so on.

At this point people will note that the information added by structured logging is potentially important and it's useful to represent it relatively unambiguously. Some other people might ask if traditional Apache common log format, or Exim's log format, are 'plaintext logs'. My answer to both is that this illustrates why plaintext is not a great format for logs. True maximally readable plaintext logs are highly constrained and wind up leaving lots of information out or being ambiguous and hard to process or both. The more additional information you include in a clearly structured format, the more potentially useful it is but the less straightforwardly readable the result is and the less you have plaintext logs.

If you want to use a structured log format, where you sit on the spectrum between plaintext logs and JSON blobs appended to something depends on how you expect your logs to be used and consumed (and stored). If people are only ever going to consume them through special tools, you might as well go full JSON or the equivalent. If people will sometimes read your logs in raw format with 'less' or 'tail' or whatever, or your logs will be comingled with logs from other programs in random line-focused formats, you should probably choose a format that's more readable by eye, perhaps some version of logfmt.