I tweeted:

That sure was a bunch of debugging because I forgot that my systemd .service file that runs scripts needed

Type=oneshot
RemainAfterExit=True

(... or it'd apparently run the ExecStop script right after the ExecStart script, which doesn't work too well.)

Let's be specific here. This was the systemd .service unit to bring up my WireGuard tunnel on my work machine, which I set up to run a 'startup' script (via ExecStart=). Because I had a 'stop' script sitting around, I also set the unit's ExecStop= to point to that; the 'stop' script takes the device down and so on.

The startup script worked when I ran it by hand, but when I set up the .service unit to start WireGuard on boot, it didn't. Specifically, although journalctl reported no errors, the WireGuard tunnel network device and its associated routes just weren't there when the system finished booting. At first I thought the script was failing in a way that the systemd journal wasn't capturing, so I stuck a bunch of debugging in (capturing all output from the script in a file, and then running with 'set -x', and finally dumping out various pieces of network state after the script had finished).

All of this debugging convinced me that the WireGuard tunnel was being created during boot but then getting destroyed by the time booting finished. I flailed around for a while theorizing that this service or that service was destroying the WireGuard device when it was starting (and altering my .service to start after a steadily increasing number of other things), but nothing fixed the issue. Then, while I was starting at my .service file, the penny dropped and I actually read what was in front of my eyes:

[Service]
WorkingDirectory=/var/local/wireguard
ExecStart=/var/local/wireguard/startup
ExecStop=/var/local/wireguard/stop
Environment=LANG=C

This .service file had started out life as one that I'd copied from another .service file of mine. However, that .service file was for a daemon, where the ExecStart= was a process that was sticking around. I was running a script, and the script was exiting, which meant that as far as systemd was concerned the service was going down and it should immediately run the ExecStop script. My 'stop' script deleted the WireGuard tunnel network device, which explained why I found the device missing after booting had finished.

The journalctl output won't tell you this; it reports only that the service started and not mention that it's stopped again and that the ExecStop script was run. If I'd looked at 'systemctl status ...' and paid attention, I'd at least have had a clue because systemd would have told me that it thought that the service was 'inactive (dead)' instead of running. If I'd had both scripts explicitly log that they were running, I would have seen in the logs that my 'stop' script was being executed for some reason; I probably should add this.

This has been a pretty useful learning experience. I know, that probably sounds weird, but my view is that I'd rather make these mistakes and learn these lessons in a non-urgent, non-production situation instead of stubbing my toes on them in production and possibly under stressful conditions.