Using journalctl's ability to show only one service

August 5, 2021

When I wrote about sending all syslog messages to one file, one of my reasons for it was so that I didn't have to hunt through assorted syslog files to find out just where a program's log messages went. However, even when you put all messages in one place, you still have to pick out the messages you care about from all the rest. Lately, I've realized that on systemd based Linux systems, this is working too hard and there's a much easier way using journalctl.

On a modern systemd based Linux system, the systemd journal knows which service unit a given message is associated with, even if the message was syslogged. This is separate from the idea of syslog facilities, which means that you don't have to care about what facility a program uses (or where your syslog configuration logs that facility).

Do you want to watch only messages from your DHCP daemon about DHCP activity? That's easy, with no greps needed: 'journalctl -f -u dhcpd.service'. Well, that's on Fedora. On Ubuntu, you're probably going to need to ask the journal to follow 'isc-dhcp-server.service' instead. This points out one little drawback, which is that you need to know relatively exactly what you're asking for. Journalctl can give you all of the logs for a specific binary, like /usr/sbin/dhcpd (with 'journalctl -f /usr/sbin/dhcpd'; don't accidentally use '-u'), but as far as I know it has no convenient syntax to give you all log messages from a given service that a particular PID is in.

(You can do this in two steps; first use 'systemctl status <PID>' to get the service unit that a PID is in (along with recent log data from the journal), then 'journalctl -u ...' and whatever other options. If you don't want to wait for systemd to grub around in the journal, you can also do 'systemctl status' and then search out the PID. Or you can remember 'systemctl status --lines=0 <PID>'. I wish there was a simple systemctl option to just tell us the service unit for a particular PID, although you can always get this from /proc/<PID>/cgroup.)

Another thing that I should make more use of is asking for only recent journal messages for a particular service, with '-S'. I originally thought that this required you to use relatively inconvenient time stamps, but as the journalctl and systemd.time manpages cover, you can use a convenient relative time syntax. If you want the last two days of log messages for a service, this is:

journalctl -u rsyslog.service -S -2d

Now that I've read up on this, I suspect I'm going to use both -S and journalctl more than I have in the past.

(You can use this relative time syntax with both -S and --until (aka -U), which means it's relatively straightforward to narrow in on a moderate time range of interest with some basic mental math.)


Comments on this page:

Excellent article! Somehow demonstrates that the functionality is really there, one just needs to go through the docs,like with any other piece of software.

Argument I hear often against systemd is that it does everything under the sun but complicates things with binary logs where you can't just grep them.

On that note, I'd like to add that there's also -g (--grep) option for journalctl, if your journalctl was built with matching support, which does just that :-)

By Miksa at 2021-08-06 03:44:30:

I pretty much like JournalD, but what could really endear it to me, is if they added interactive mode. Just start journalctl and then filter it to the units and timeframes that suit your needs without repeated restarting.

Let's have Microsoft port Event Viewer to Linux/systemd. Hey now...!

My life got a lot better when I started remembering the `-b` flag to journalctl to limit the output to the last boot. Since I'm generally using it on my laptop, my uptime isn't particularly long and I almost always don't care about anything that happened before the last time I powered up.

From 193.219.181.219 at 2021-08-06 15:32:50:

journalctl will automatically add .service as needed, like systemctl does, so journalctl -fu dhcpd or journalctl -u rsyslogd -S -2d will do. (I didn't know that "-2d" was valid and always used "--since=2\ days\ ago".)

There's no built-in command for "unit of this PID" that I've found. I was hoping that systemctl show -P Id $pid would work, but rather surprisingly, it treats numeric values as job IDs, rather than process IDs...

If I recall correctly, there's one caveat to using -u: messages logged by a very short-lived process are not guaranteed to be tagged with the proper _SYSTEMD_UNIT= (as there is no SO_PEERCRED for cgroups, thus journald must scrape the cgroup from /proc). Sometimes this means lines appear in journalctl -t when filtered by syslog tag (i.e. "program name") but are missing in journalctl -u.

Bonus: journalctl /dev/sda will get kernel logs for a single device, together with its parents. (But not children or siblings, which might be relevant for some USB devices. And there's no nice shortcut for network interfaces, you have to use something like journalctl -b _KERNEL_DEVICE=+pci:0000:03:00.0 for that.) I don't think I've ever used this in anger, but it's nevertheless kinda neat.

From 193.219.181.219 at 2021-08-06 15:45:20:

Hmm, speaking of kind of neat things in journalctl... Messages about core dumps logged by systemd contain quite a few metadata fields, most of which are pretty-printed by coredumpctl info, but there are also some raw values which aren't shown anywhere at all (that I know of).

For example, journalctl -n 1 -o verbose MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1 (which is the ID for "core dump" messages) will reveal the entire contents of /proc/<pid>/status, /proc/<pid>/maps, resource limits, environment variables, mount table, open FDs at time of dump. There might be a day when this reveals something useful.

Written on 05 August 2021.
« I have mixed views on new DNS top level domains (TLDs)
Some bits of how Bash and GNU Readline's "bracketed paste" mode behaves »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Aug 5 23:21:55 2021
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.