== What debugging info I want from things like SMF and _systemd_ I've recently been tooling around in both SMF and _systemd_, so I've developed some strong opinions on what startup systems like this need to provide to help working sysadmins. It will probably not surprise you to know that neither SMF nor _systemd_ deliver entirely useful information today. There are two sorts of service startup problems, which I'll call the simple and the complex. The simple question is 'why did this service fail to start'. To answer this sysadmins need to know exactly what was run and what happened to it, ideally including its output and log messages; if the service wasn't started because of missing dependencies, we need to know what they were. SMF and _systemd_ both half-heartedly deliver this today. (Neither has output that is optimized for making it clear if a service is not enabled at all, is enabled but had missing dependencies, or if it was enabled and failed to start or died later. In fact distinguishing between 'failed to start' and 'seemed to start fine, died later' is actually fairly important but doesn't tend to be reported well.) The complex startup problems are ordering problems, such as [[our recent issues with ZFS pool activation and iSCSI disk discovery ../solaris/ZFSPoolActivationII]]. To deal with these issues, you need two things: you need to know the actual order that service startup both started and finished in (you need both because in a modern system several services may be starting at once), and you need to know why, ie the service dependency graph. In fact you want several views of the service dependency graph, for example the transitive expansion of dependencies both ways for some service: 'everything that has to be up for this service to start' and 'everything that requires this service to be up'. It's also handy to be able to ask questions like 'does service X depend on service Y in some way, and if so how?' This is mostly missing in SMF and _systemd_ today, as far as I can see. It's possible that _systemd_ can be coaxed into giving you the information, but if so how to do it has been carefully hidden from harried sysadmins. (As an example of getting it incomplete, SMF will give you direct dependencies in both ways but not indirect ones, which leads to a frustrating game of 'chase the dependency'.)