== A really annoying gap in system observability

The other day I had a problem; new _gnome-session_ processes weren't
working right on one of our major login servers. They didn't quite
hang outright; instead, _strace_ showed that they seemed to spend all
of their time talking very slowly to one file descriptor instead of
responding to other Gnome processes that were trying to talk to them.

(When this happens, the other Gnome processes are not very happy and
your entire Gnome session basically hangs.)

This left me with a big question: what was on the other end of that file
descriptor?

Answering this question turned out to be absurdly difficult, and that
is the problem. At least theoretically, 'observability' of systems is
one of the next big things; everyone is burning with enthusiasm for
tools like Solaris's DTrace and Linux's SystemTap. Yet vendors (and
Linux people) have almost completely neglected basic observability tools
for tasks like simply seeing what processes are connected to.

On Linux, _lsof_ can be said to be officially supported and it was able
to tell me that the particular file descriptor was a Unix domain socket;
however it couldn't tell me what the other end was connected to, and
I'm not sure that that information is exported by the kernel. On other
Unixes like Solaris, _lsof_ isn't even officially supported by the
vendor; to the extent that it works (it's often incomplete), it works
only because people have put heroic amounts of effort into reverse
engineering portions of the Solaris kernel and obtaining information by
force and trickery.

Frankly, this is absurd. Tools like _lsof_ and _lslk_ have been a vital
part of the sysadmin arsenal for more than fifteen years. Yet it's still
the case that no one (or at least almost no one) officially supports
them and makes sure that they can get the complete information that
sysadmins need, or even makes sure that sysadmins can get the same
information through other tools.

In 2011, in the era of observability as a big thing, it should be
trivial for sysadmins to find out information like 'what is this process
talking to' or 'what is using resource X', or even 'who is using what
resources of type Y'. That it is not says sad things about vendors (and
open source developers).

=== Sidebar: how I answered my question

I used brute force. I hacked my own X environment to _strace_ my
_gnome-session_ from the moment it was started; this let me see the
moment when the relevant file descriptor was created and _connect()_'d
(it turned out to be talking to the system DBus daemon). However, this
workaround was only possible because I could start the program on demand
and it hung reliably; had I been dealing with a long-running daemon that
was malfunctioning like this, I would have been out of luck.

PS: it turns out that you do not want to restart the system DBus
daemon out from underneath _gnome-session_. If you do, all existing
_gnome-session_ processes immediately exit, taking every user's session
with them.