OpenSSH has decent basic support for opportunistic connection sharing, but most of this is that ssh will look for an existing master connection before making a new one. It could do better and enable a true opportunistic connection sharing mode.

There's two problems with ssh's current support for this, one big and one small. First, being the master connection can keep your ssh running when it would otherwise exit. Second, the master connection exits immediately when there's no active sessions running over it; there's no option to make it wait around for a bit in case you're going to open another session soon.

(You can sort of fake your way around both of these, but in either case there are drawbacks and limitations.)

The second problem is at least theoretically easy to solve; add a ssh setting to delay closing the master connection down for some number of seconds after the last session closes. (Hopefully this is allowed by the protocol. If not, ssh might have to hold a do-nothing session open behind the scenes.)

The first problem doesn't have a clear, obvious solution, but I see two approaches. The brute force solution is to add an option where ssh runs a command if there is no master connection, and then tries again. The elegant solution is to have a switch that makes ssh fork and detach a child process to handle the master connection once it's set up.

(Much like the -f switch, you could only push the master connection into the background after all of the necessary authentication and so on had been done.)

Sidebar: my theory on how to fake your way around these

There's no good way to fake a connection close delay; the only thing you can really do is decide that you're willing to have the master connections sit around forever (or until you kill them by hand), at which point you just have them run sleep with an absurdly long timeout or the like.

The best way to deal with the second problem is always try to start connection masters, but run them with '-o "ControlMaster auto"' so that if a master connection already exists, the new would-be connection master silently converts itself into just a session over the connection. Unfortunately, for this to work well you want the connection master sessions to run a command that will exit after not too long; otherwise your machines will wind up running an ever growing collection of very long sleep's or the like.