What debugging info I want from things like SMF and systemd
I've recently been tooling around in both SMF and systemd
, so I've
developed some strong opinions on what startup systems like this need to
provide to help working sysadmins. It will probably not surprise you to
know that neither SMF nor systemd
deliver entirely useful information
today.
There are two sorts of service startup problems, which I'll call the
simple and the complex. The simple question is 'why did this service
fail to start'. To answer this sysadmins need to know exactly what
was run and what happened to it, ideally including its output and log
messages; if the service wasn't started because of missing dependencies,
we need to know what they were. SMF and systemd
both half-heartedly
deliver this today.
(Neither has output that is optimized for making it clear if a service is not enabled at all, is enabled but had missing dependencies, or if it was enabled and failed to start or died later. In fact distinguishing between 'failed to start' and 'seemed to start fine, died later' is actually fairly important but doesn't tend to be reported well.)
The complex startup problems are ordering problems, such as our recent issues with ZFS pool activation and iSCSI disk discovery. To deal with these issues, you need two things: you need to know the actual order that service startup both started and finished in (you need both because in a modern system several services may be starting at once), and you need to know why, ie the service dependency graph. In fact you want several views of the service dependency graph, for example the transitive expansion of dependencies both ways for some service: 'everything that has to be up for this service to start' and 'everything that requires this service to be up'. It's also handy to be able to ask questions like 'does service X depend on service Y in some way, and if so how?'
This is mostly missing in SMF and systemd
today, as far as I can
see. It's possible that systemd
can be coaxed into giving you the
information, but if so how to do it has been carefully hidden from
harried sysadmins.
(As an example of getting it incomplete, SMF will give you direct dependencies in both ways but not indirect ones, which leads to a frustrating game of 'chase the dependency'.)
|
|