On documenting (or not documenting) binary protocols

July 24, 2011

In an aside in SaneBinaryProtocols, I noted that part of a sane binary protocol was documenting it, something that the people responsible for the sendmail milter protocol had failed to do. Today I have an aside on that general issue.

The traditional excuse for not documenting your protocol, at least in the open source world, is that you don't want to commit to supporting it over the long term; by not documenting it you preserve your freedom to change it without having to worry about backwards compatibility. The problem is that in practice this doesn't actually work.

If you have your own clients and servers out there in the field, you already have programs that are speaking the old version of your protocol. Unless you can force people to upgrade everything at the same time (which is often quite unpopular), you are in practice stuck with backwards compatibility anyways so that your new server can be talked to by old clients (and sometimes so that your new clients can talk to old servers).

(Failure to have backwards compatibility simply invites people not to upgrade to your new release, which is often not one of the goals that you have.)

Documenting your protocol and having other people write clients and servers against it may make protocol upgrades take somewhat longer to become pervasive out there into the world. But it does not intrinsically create a problem for you where none existed before unless you were already annoying people by insisting on lock-step upgrades.

Also, in real life people will reverse engineer your protocol anyways if it is at all useful, regardless of what you say, and will write clients and servers against their reverse engineered version. You can yank the rug out from underneath them by changing the protocol with no backwards compatibility if you want to, but again this just encourages people not to upgrade. (It also encourages people to add backwards compatibility to third party clients and servers, making them more attractive than your own versions.)

(Yes, sometimes abrupt protocol changes with no backwards compatibility are justified, such as if you discover that the previous version had a security vulnerability that allowed bad things.)

Written on 24 July 2011.
« What good secure string expansion on Unix should look like
A little thing that irritates me about common WSGI implementations »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Jul 24 01:08:01 2011
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.