What debugging info I want from things like SMF and systemd

December 12, 2011

I've recently been tooling around in both SMF and systemd, so I've developed some strong opinions on what startup systems like this need to provide to help working sysadmins. It will probably not surprise you to know that neither SMF nor systemd deliver entirely useful information today.

There are two sorts of service startup problems, which I'll call the simple and the complex. The simple question is 'why did this service fail to start'. To answer this sysadmins need to know exactly what was run and what happened to it, ideally including its output and log messages; if the service wasn't started because of missing dependencies, we need to know what they were. SMF and systemd both half-heartedly deliver this today.

(Neither has output that is optimized for making it clear if a service is not enabled at all, is enabled but had missing dependencies, or if it was enabled and failed to start or died later. In fact distinguishing between 'failed to start' and 'seemed to start fine, died later' is actually fairly important but doesn't tend to be reported well.)

The complex startup problems are ordering problems, such as our recent issues with ZFS pool activation and iSCSI disk discovery. To deal with these issues, you need two things: you need to know the actual order that service startup both started and finished in (you need both because in a modern system several services may be starting at once), and you need to know why, ie the service dependency graph. In fact you want several views of the service dependency graph, for example the transitive expansion of dependencies both ways for some service: 'everything that has to be up for this service to start' and 'everything that requires this service to be up'. It's also handy to be able to ask questions like 'does service X depend on service Y in some way, and if so how?'

This is mostly missing in SMF and systemd today, as far as I can see. It's possible that systemd can be coaxed into giving you the information, but if so how to do it has been carefully hidden from harried sysadmins.

(As an example of getting it incomplete, SMF will give you direct dependencies in both ways but not indirect ones, which leads to a frustrating game of 'chase the dependency'.)

Written on 12 December 2011.
« head versus sed
How not to improve your package updater application »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Dec 12 02:04:54 2011
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.