Wandering Thoughts archives

Please have symmetric option negotiations in your protocols

Suppose that you are designing a protocol where the two ends (let us call them the initiator and the target) must agree on a joint set of options, actions, or whatever that they both support, will use, or whatever. If you are doing this, I have a small request: please make the option negotiation process symmetric, where the initiator and the target do the same operations (with slightly different data).

In a symmetric option negotiation process, the initiator says something like 'I support the following things' and the target replies with 'I support the following subset of the things you support'. One sign of a symmetric exchange is that if the initiator and target are equally capable, the set of options in the messages back and forth is exactly the same; the initiator says 'I support everything', and the target says 'then lets do everything'. In an asymmetric option negotiation process, well, things don't work that neatly.

Since this all sounds very abstract, let's use a concrete example; the milter protocol. In the milter protocol the MTA and the milter exchange two sets of options. The MTA opens with a bitfield of the message modification actions it supports and a bitfield of the protocol steps it is willing to skip. The milter replies with a bitfield of the message modification actions it may use and a bitfield of the protocol steps it wants the milter to skip. The negotiation of actions is a symmetric negotiation, but the negotiation of protocol steps is an asymmetric one. We can see the asymmetry in what the MTA and the milter say; the MTA will normally say 'I could skip all of the steps I support', and the milter will normally say 'I don't want you to skip anything (except maybe the steps that I don't understand)'.

The fundamental operation in a symmetric option negotiation is binary and-ing supported options together, you can do it repeatedly, and each end can use the same code. For example, you might and together what your program wants to support, what your protocol codec library can support, and what the other side is willing to support (if there is an other end). Asymmetric option negotiation has no neatly commutable operation on the target; in the milter protocol, for example, a milter support library wants to form the 'protocol steps I want you to skip' reply with:

mta_proto & (~known_proto_steps | client_skip_list)

(We want to tell the MTA to skip anything outside the library's known protocol steps plus anything that the client code wants the MTA to skip.)

A symmetric version of this would negotiate what protocol steps the MTA should perform, and would be:

mta_proto & known_proto_steps & client_do_list

If you default mta_proto to known_proto_steps, this code is also how a library would form the MTA's initial supported protocol bitfield; if you also default client_do_list to known_proto_steps, you have covered the common case for both MTA and milter (where you wind up doing all steps in the protocol).

As you can see, symmetric option negotiation is simpler, more straightforward, easier to get right, and easier to understand and convince yourself that the code is correct. I believe that these differences matter.

Formatting information output to make it easy to manage

Suppose that you are designing systems to be managed and you are taking the straightforward approach of dumping out information when external programs ask for it. As it happens, I have some views on how you should format this information and why.

Monitoring systems have two major concerns when they're dealing with your status output. First, they want to be confident that they understand what you're printing out, ie that they are correctly parsing it. Second, they want to be sure that they understand the meaning of everything you print out. Monitoring systems care about both because a problem with either of them can cause a monitoring system to misinterpret your state.

The attraction of XML or JSON as your output format is not so much that monitoring systems don't have to write a parser, it's that they don't have to worry about whether or not they have parsed your output correctly. They don't have to debug a parser or be confidant that it can completely parse everything that you ever output, even in obscure situations.

The corollary of this is that if you want to roll your own output format, you should make it as simple and regular as possible. Simple regular formats make for easy parsers, and easy parsers are more likely to be correct, complete, and bug-free. Conversely, inventing your own complex structured output format is about the worse thing you can do (you get bonus points for things that appear only rarely).

The other concern is harder. Ultimately, understanding requires complete documentation of what your system outputs and what it means. However, you can make things easier by being consistent, for the same reason as with parsing; it makes it simpler for monitoring systems to be confidant that they haven't missed some obscure corner case in how you will report things.

(For some sorts of information you can also be redundant, for example by always flagging certain sorts of situations with a common marker. In theory this lets a monitoring system extract some information from something that it doesn't fully understand, eg if it sees the 'ERROR' field set it knows that something has gone wrong even if it doesn't fully understand what.)

Design systems to be managed

Here is something that I've come to believe strongly: if you're building a system today (whether it's just a single program or much larger), you should be designing it for what I'll call 'manageability'. By this I mean the ability of your system to integrate with other systems and to be managed by them.

Once upon a time, the state of the art in systems operation was such that systems (programs, packages, environments, etc) were basically standalone and self-contained entities. A lot of the time no one really worried about system management issues, and it was good enough that your system actually existed and worked. If you worried about these issues at all, generally you built things directly into your particular system. Thankfully, the state of the art has moved on since then. Although it's by no means universal (yet), a 'best practices' environment today will certainly have its own monitoring system, its own configuration management and automated deployment system, and so on.

This means that if you're designing a system to go into a modern environment, you should be designing it so that it can easily integrate with these systems. You should absolutely be thinking about how your system can be deployed and managed with cfengine or puppet or chef, how it can be monitored with Nagios, Catci, or Munin, and so on. Since there are a lot of these systems (not just the ones I've listed here), you shouldn't be planning to do this by writing specific plugins for your favorite tools. Instead, you should have useful public interfaces for dealing with your system.

(The minimum interface you should aim for is something to report configuration information, state information, and performance metrics.)

I don't really care what form these interfaces take, although I feel that programs with documented, machine-parseable output are usually better than libraries with an API. The important thing is that you have such interfaces, that you document them, and as much as possible that you keep them stable; whether or not you produce XML, JSON, or something else is a minor issue after that. The important thing is to think about the issue and to solve the problem of making your system transparent, not opaque.

(The problem with libraries is that they only do half of the job; you still need to write or modify other code to call the libraries.)

Please note that if you have programs that output this information but you have not made their ouput easily machine parsed or documented it, you have not actually solved the problem. As I noted in my recent ZFS grump, this is the difference between a tool and a frontend; a tool produces things that are easy for other programs to use.