2014-03-03
The multiple levels of interprocess communication
In some quarters it's popular to say that things in computer programming go in cycles and history repeats itself. I'm a bit more optimistic so I like to view it as people repeatedly facing the same problems and solving them with today's tools. One of those things that keeps coming around is IPC aka interprocess communication. One way to look at IPC is to split it up into multiple levels that seem to reoccur again and again.
The first layer of IPC is simply connecting processes to each other. Since you can't get very far without this it tends to be well solved, although people keep inventing new ways of doing it with slightly different properties. Good local IPC mechanisms provide reliable authentication information about who is connecting to you and provide some way to transport credentials, access rights, or the equivalent.
(On modern Unix systems Unix domain sockets provide at least UID information and can be used to pass file descriptors around.)
The next IPC level is a common protocol format, which gets created when people get tired of making up data formats and (re)implementing encoders and decoders. Generally the high level protocol format will specify some sort of destination (in this sense, a method) plus a structured payload (which may or may not be self documenting to some degree). Sometimes you see things like sequencing and multiplexing added at this level.
(Modern examples include Google protobufs and JSON, but there are a very large number of protocol formats that people have used over the years. One early Unix example is the Sun RPC wire format they used for NFS and a number of related things.)
The next IPC level is some kind of a 'message bus', which generally exists to solve two related registration problems. In the addressing problem, programs want to talk to whatever is implementing service X (and to have a way of registering as handling X, and stopping handling X, and so on). In the broadcast problem, a program wants to broadcast information about X to any interested parties, whoever they may be. Once you have an active message bus it can also do additional things like starting a program when a request for service X comes in.
(Message bus services on Unix date at least as far back as the venerable rpcbind, which was originally invented by Sun for NFS-related addressing issues. There is a theme here.)
The final IPC level is introspection, where you can use the IPC mechanism to find out what is registered, what operations they support, and hopefully monitor traffic and other activity. Technically IPC introspection can appear without a message bus but I think this is uncommon. The power of the introspection varies tremendously from IPC system to IPC system.
You can invent ad-hoc mechanisms to solve all of these levels of problems, and people do. What drives people to 'standardize' them is a desire to avoid reinventing the wheel for each new program and each new level of the problem. Common protocol formats almost invariably come with libraries or codec generators (or both), message busses come with standard programs and libraries (instead of ad-hoc solutions like 'Unix domain sockets with specific names in directory Z'), and so on. Introspection and tools related to it solve a bunch of common problems and can often be used to enable ad-hoc interaction with services, eg through shell scripts.
(Once you have a reasonably sophisticated message bus with a lot of things going on through it, you really want some way of finding out what messages are flowing where so you can debug the total system.)