Wandering Thoughts archives

How to deprecate bits of your program

Since this appears necessary, here is how to deprecate some bit of your program in a way that makes sysadmins hate you as little as possible.

• first, add a warning to your documentation and a 'compatibility switch' that causes your program to use the old behavior that you are deprecating. Of course, the compatibility switch currently does nothing since the old behavior is the current behavior, but now you've let people start explicitly specifying that they need the old behavior.

  If you are changing the behavior of your program (instead of just stopping supporting something), you should also add a switch that enables the new behavior.

(If you are not planning on having a compatibility switch at all, you lose. Flag days make sysadmins hate you with a burning hate, because there is nothing we love quite as much as having to update all sorts of other programs the moment we upgrade an OS.)

• wait until this new version of your program makes it into many or all of the popular Linux distributions and any other OS that it's commonly used on. This is not just things like Ubuntu and Fedora and Debian; you really need to wait for the long term supported, slow updating distributions like Ubuntu LTS, Red Hat Enterprise (or CentOS, if you prefer to think of it that way), and so on.

  (You need to consider Ubuntu LTS a different distribution than Ubuntu for this, because users of Ubuntu LTS may well not update their systems until the next LTS release comes out.)

  I tend to think that you should wait a minimum of a year no matter what, although given the update schedule of some of these distributions you're probably going to have to anyways.

• now you can release a version that prints warnings about the old behavior and suchlike. This version must have a way of specifically enabling the new behavior (if there is one as such).

• wait a distribution update cycle again.

• finally you can release a version that drops the old behavior, although you have to keep the now vestigial switch that enables the 'new' behavior (even though it now does nothing).

  (If you want to remove it, wait another update cycle. You saw that one coming.)

In less words: don't have any flag days. Always make sure that sysadmins and developers can prepare for a change ahead of time. Let them suppress warnings before warnings are printed and start using new behavior before it becomes mandatory (and then don't break the mechanism for this).

More accidental BitTorrent on our network

We've recently had another interesting case of 'accidental BitTorrent' on our network, different from the previous times. This time around it was through our VPN server, and just from that alone you can probably guess what happened.

Suppose that you are a user, sitting at home running BitTorrent in the background on your home machine while you do other things. You need to access some restricted on-campus thing; our generally supported way to do this is to fire up a VPN connection so that you are 'inside' as far as our various access controls are concerned. So you do so and browse the library, or get at our Samba server, or whatever.

And all the time that you're doing this, all of your network traffic is flowing over the VPN and out our network to the outside world. Including that BitTorrent client you have running in the background. Whoops.

Like the first time, this is not the user's fault; I can't expect even our users to all understand network routing and the fine details of how our VPN behaves. (Yes, this is obvious to us, but that's because we spend all of our time immersed in these things.)

This makes a nice illustration of something else as well: the perils of a mismatch between the user's view of what's happening and what's really going on. If the user's view is that they're using the VPN to access some inside resource, how would they even think about what's happening with their traffic to regular things? At a conceptual level it's not even related to what they're doing.

Sorting out what 'passive ftp' is

(Because I keep forgetting this and having to straighten it out again in my own head, it's time to write down the difference between regular ftp and passive ftp).

Ftp has to transport two different things: commands (and status responses to them), and the actual ftp data. In a modern protocol, these two things would be multiplexed together over the same TCP connection, but ftp dates from when the Internet was both a much simpler environment and a much more limited one, so it uses separate connections for each job. In particular, it has one command connection and then every time it needs to transfer data it makes a new TCP connection, shoves the data over it, and closes the connection again.

It would be nice to report that modern passive ftp changes this and multiplexes commands and data over the same single TCP connection. Alas, it does not; the only thing that changes in passive ftp is who calls who to create the data TCP connections:

• In regular ('active') ftp, the client listens on a port (that it tells the server about) and the server makes an outgoing connection to that port. This connection is always from port 20 (ftp-data) on the server.

• In passive ftp, the server listens on a port (that it tells the client about) and the client makes an inbound connection to that port. There are no specific port numbers involved on either side, although many ftp servers can restrict the port range they use to make the life of their firewall easier.

Passive ftp is easier for clients that are behind firewalls because they only make outgoing connections to the ftp server. However, the firewall has to either understand the ftp protocol or allow clients to make connections to arbitrary high-numbered ports on outside machines.

Sidebar: the ftp bounce attack

I said that the ftp client told the ftp server what port it was listening on. Actually, that's a bit of a lie; the ftp client tells the server both the port and the IP address that it's listening on.

(So does the server, when it's giving a port to the client.)

You can see what happens next. It used to be that ftp servers were perfectly happy to connect to whatever IP address and port your client told them to and dump things into it. For example, 'port 25 on this IP address here'. This only got worse if the ftp server allowed you to upload files to it that you could then re-'download' to your target.

Building true ssh opportunistic connection sharing

OpenSSH has decent basic support for opportunistic connection sharing, but most of this is that ssh will look for an existing master connection before making a new one. It could do better and enable a true opportunistic connection sharing mode.

There's two problems with ssh's current support for this, one big and one small. First, being the master connection can keep your ssh running when it would otherwise exit. Second, the master connection exits immediately when there's no active sessions running over it; there's no option to make it wait around for a bit in case you're going to open another session soon.

(You can sort of fake your way around both of these, but in either case there are drawbacks and limitations.)

The second problem is at least theoretically easy to solve; add a ssh setting to delay closing the master connection down for some number of seconds after the last session closes. (Hopefully this is allowed by the protocol. If not, ssh might have to hold a do-nothing session open behind the scenes.)

The first problem doesn't have a clear, obvious solution, but I see two approaches. The brute force solution is to add an option where ssh runs a command if there is no master connection, and then tries again. The elegant solution is to have a switch that makes ssh fork and detach a child process to handle the master connection once it's set up.

(Much like the -f switch, you could only push the master connection into the background after all of the necessary authentication and so on had been done.)

Sidebar: my theory on how to fake your way around these

There's no good way to fake a connection close delay; the only thing you can really do is decide that you're willing to have the master connections sit around forever (or until you kill them by hand), at which point you just have them run sleep with an absurdly long timeout or the like.

The best way to deal with the second problem is always try to start connection masters, but run them with '-o "ControlMaster auto"' so that if a master connection already exists, the new would-be connection master silently converts itself into just a session over the connection. Unfortunately, for this to work well you want the connection master sessions to run a command that will exit after not too long; otherwise your machines will wind up running an ever growing collection of very long sleep's or the like.

How I use ssh's connection sharing feature

Following on the basic overview, here's how I use OpenSSH's connection sharing feature.

First, since I hate programs that don't exit when I want them to, I don't let ssh automatically create connection masters. Instead I always explicitly create them myself, so that I can do so in a way that harmlessly parks them in the background. So while I set a ControlPath (to a value mentioned in the previous entry), I leave ControlMaster set to no, the default.

When I want to create a master ssh, I have a script that boils down to:

ssh -M -f -o "BatchMode yes" $HOST exec sleep 120

You might wonder about the 'sleep 120'.

One of the peculiarities of how I use connection sharing is that so far I always do it to hosts that I also have persistent sessions to. These persistent sessions pin the master ssh in place whether or not the master ssh itself is still doing anything, so the sleep does two things here. First, it keeps the master ssh going long enough for my real sessions to start, and second, it means that if and when I later tear down all of my real sessions to the host, the master ssh will also exit because the sleep has long since finished.

(Ideally there would be a ssh option for 'keep running N seconds after the last session closes to see if a new subordinate shows up'. Then I could set that to 120 seconds or so and just use '/bin/true' as the command that the master ssh runs.)

In actual practice I don't create master ssh instances by hand. Instead I do it from my X startup file, which (vastly simplified) winds up looking like:

ssh-master host1
sleep 1  # give it time to come up
ssh host1 x-app-1 &
ssh host1 x-app-2 &
....

(And then anything else that I want to do on host1 later will also get the benefits of using the existing connection.)

Among other things, this approach has the benefit that it's easy to add. I didn't need to change anything that my X startup script was already doing; I just inserted an additional command early on. (And then took it out again the first time I tried it, when it turned out to cause various problems.)

The basics of ssh's connection sharing feature

Vaguely recent versions of OpenSSH have a feature where you can have multiple independent sessions running over what is actually one connection, even though you ran several separate ssh commands. The primary advantage this offers is that new ssh sessions start significantly faster, since they don't have to go through all of the cryptography and authentication of a full ssh connection. (As a side benefit, there's fewer processes running on either side, since you no longer need a ssh and an sshd for each active session.)

When I first tried this, it didn't work too well; it would slowly eat resources and the connection would lock up every so often. These days it seems to work fine between at least Ubuntu 8.04 and Fedora 11.

Connection sharing has two parts; there is a single master ssh and then some number of subordinate ssh sessions that reuse the master's connection instead of making their own. There are two .ssh/config settings that control this behavior:

• ControlMaster determines whether a ssh becomes a connection master (and what it does if a would-be subordinate tries to talk to it).
• ControlPath tells both master and subordinates how to find each other. The value I use for this is:

  /u/cks/.ssh/control/cs-%r@%h:%p

  (You should also use %l in there somewhere if you have a shared home directory; the machine I use this from doesn't, so I don't bother.)

One of the drawbacks of a master ssh is that it doesn't exit until the last subordinate exits. Thus, automatically creating ssh masters can be a bad idea; imagine the annoyance of, say, your first login session somewhere not exiting when you ^D it, because there's still a subordinate ssh active. Master processes can be explicitly created by using ssh's -M command line option, which is equivalent to specifying either yes or auto for the ControlMaster setting (depending on how many -M's you use).

(Although it's unlikely to matter in most circumstances, another drawback of this connection sharing is that there is only a single ssh and sshd doing all of the encryption and decryption, which can be slow. If you have multiprocessor machines of some sort and are doing multiple high-bandwidth bulk transfers, you're probably better off with entirely separate connections.)

There are three situations where I think that connection sharing is likely to be a significant benefit: if your link is slow (so that the back and forth exchanges of the ssh protocol take a significant amount of time due to packet delays), if either machine is quite slow (so that all of the computation involved in a full ssh connection takes quite a while), or if you create and throw away sessions quite a lot, so you want them to be as cheap and as fast to start as possible.

(As it happens, the last one describes my typical usage pattern.)