Some thoughts on creating simple and sane binary protocols
The best way to create a new simple and sane binary protocol is to not do so; create a text based protocol instead. Text based protocols have any number of advantages; they're easier to write handlers for, they can be debugged and tested by hand, semi-smart proxies are easy to write, it's easy to use network monitoring tools to trace them in live systems, and so on. And protocols like HTTP and (E)SMTP prove that they are viable at large scale and high traffic volumes. Really, your situation is probably not an exception that requires an 'efficient' binary protocol.
But suppose that you've determined that you need a (new) binary protocol for some reason. Because you're nice, you want to make one that irritates programmers as little as possible, ie that is as easy as possible to write protocol encoders and decoders for. Having looked at a number of binary protocols and just recently written a codec for sendmail's milter protocol, I have a few opinions on what you should do.
(Beyond the obvious one of 'document it', which the sendmail people skipped.)
First, wrap your various structures, bitstreams, or whatever in a simple packet format. The important bit of such a format is that packets have a common fixed-size header that includes the packet size and then the remaining variable sized data. Having the size up front allows the decoder to know very early on if it has all of the data that it needs for the packet; this simplifies further decoding and enables various sorts of error checks. You want the packet header to be fixed size so that it is easy to unconditionally read and decode.
Second, build your messages out of as few primitive field types as possible and make those primitive types as simple as possible to decode and encode. In my view, the simplest field types are fixed sized fields, then (fixed-size) length plus data, and then bringing up the rear are delimited fields (where there is some end marker that you have to scan for). If you create complex encoded field types, expect programmers to hate you.
(In general, creating a field type that can't be encoded with either
memcpy() or a printf-like formatter is probably a mistake.)
Finally, have only a single field that determines the rest of the message's format, and put this field at a fixed early point in the packet. In other words, you have a fixed set of structures (or messages) that are encoded into your binary protocol and then some marker of which message this is. Avoid variable format messages, where how you decode the message depends in part on the message contents; for example, a specification like 'if field A has value X, field B is omitted' creates a variable format message. Variable format messages require conditional encoding and decoding, which complicates everyone's life. By contrast a fixed format message can be decoded to a list of field values based only on knowing the field types and their order (and it can be encoded from such a list in the same way).
(If you have to have variable format messages, the closer you stick to this approach the better. Recursive sub-messages are one obvious approach.)
A simple protocol like this can be described in a way that enables quite simple and relatively annoyance free encoding and decoding in modern high level languages. But that's another entry, since this one is already long enough.