Wandering Thoughts archives

My preliminary views on mosh

Mosh is sort of a more reliable take on ssh that supports network disconnections, roaming, and other interruptions. I've heard about it for a while and recently Paul Tötterman asked me what I thought about it in a comment on my entry on SSH connection sharing and network stalls. The short version is that so far I haven't been interested in it for a collection of reasons, which I'm going to try to run down in the honest order.

First off, mosh solves a problem that I basically don't have. Mosh sounds great if I was trying to SSH in to our servers from a roaming, periodically suspended laptop, or facing a terribly unreliable network, or just dealing with significant network latency. But I'm not; essentially all of my use of ssh is from constantly connected static machines with fixed IP addresses and good to excellent networking to the targets of my ssh'ing.

Next, using mosh instead of ssh is an extra step. Mosh is not natively installed on essentially anything I use, either clients or especially servers. That means that before I can even think of using mosh, I need to install some software. Having to install software is a pain, especially for more exotic environments and places where I don't have root. If mosh solved a real problem for me it would be worth overcoming this, but since it doesn't, I don't feel very motived to go to this extra work.

(In the jargon, you'd say that mosh doesn't fix a pain point.)

Then there's the problem that mosh doesn't support critical SSH features that I use routinely. At work I do a lot with X11 forwarding while at home I rely on ssh agent forwarding to one machine. This narrows mosh's utility significantly in either environment, so I could only use it with selected machines instead of using it relatively pervasively. Narrow usage is another disincentive to use as it both lowers even the potential return from using mosh and increases the amount of work involved (since I can't use mosh pervasively but have to switch back and forth somehow). There are some hand-waving coping measures that could reduce the pain here.

Finally, down at the bottom (despite what I wrote in my reply comment) is that I have much less trust in the security of mosh's connection than I do in the security of SSH connections. Mosh may be secure but as the people behind it admit in their FAQ, it hasn't been subject to the kind of scrutiny that OpenSSH and the SSH v2 protocol have had. SSH has had longer scrutiny and almost certainly far more scrutiny, just because of all of the rewards of breaking OpenSSH somewhere.

If I'm being honest, nervousness about mosh's security wouldn't stop me from using it if it solved a problem for me. Since it doesn't, this nervousness is yet another reason to avoid mosh on general principles.

(It may surprise people to hear this but I'm generally quite conservative and lazy in my choice of tools. I tend not to experiment with things very often and it usually (although not always) takes a lot of work to get me to give something a try. Sometimes this is a bad thing because I quietly cling to what turns out to be an inferior alternative just because I'm used to it.)

The 'cattle' model for servers is only a good fit in certain situations

To start with, let me define my terms. When I talk about 'cattle' servers, my primary definition is expendable servers that you don't need to care about when something goes wrong. A server is cattle if you can terminate it and then start a new one and be fine. A server is a pet if you actually care about it in specific staying alive.

My contention is that to have cattle servers, you either need to have a certain service delivery model or be prepared to spend a lot of money on redundancy and (HA) failover. This follows from the obvious consequence of the cattle model: in order to have a cattle model at all, people can't care what specific server they are currently getting service from. The most extreme example of not having this is when people ssh in to login or compute servers and run random commands on them; in such an environment, people care very much if their specific server goes down all of a sudden.

One way to get this server independence is to have services that can be supplied generically. For example, web pages can be delivered this way (given load balancers and so on), and it's often easy to do so. A lot of work has gone into creating backend architectures that can also be used this way (often under the goal of horizontal scalability), with multiple redundant database servers (for example) and clients that distribute DB lookups around a cluster. Large scale environments are often driven to this approach because they have no choice.

The other way to get server independence is to take what would normally be a server-dependent thing, such as NFS fileservice, and apply enough magic (via redundancy, failover, front end load balancer distribution, and so on) to turn it into something that can be supplied generically from multiple machines. In the case of NFS fileservers, instead of having a single NFS server you would create an environment with a SAN, multiple fileservers, virtual IP addresses, and transparent failover (possibly fast enough to count as 'high availability'). Sometimes this can be done genuinely transparently; sometimes this requires clients to be willing to reconnect and resume work when their existing connection is terminated (IMAP clients will generally do this, for example, so you can run them through a load balancer to a cluster of IMAP servers with shared backend storage).

(These categories somewhat overlap, of course. You usually get generic services by doing some amount of magic work to what initially were server-dependent things.)

If you only have to supply generic services or you have the money to turn server-dependent services into generic ones, the cattle model is a good fit. But if you don't, if you have less money and few or no generic services, then the cattle model is never going to fit your operations particularly well. You may well have an automated server setup and management system, but when one fileserver or login server starts being flaky the answer is probably not going to be 'terminate it and start a new instance'. In this case, you're probably going to want to invest much more in diagnostics and so on than someone in the cattle world.

(This 'no generic services' situation is pretty much our situation.)

SSH connection sharing and erratic networks

In the past I've written about SSH connection sharing and how I use it at work. At home, though, while I've experimented with it I've wound up abandoning SSH connection sharing completely because it turns out that SSH connection sharing has a big drawback in the sort of networking environment I have at home.

Put simply, with connection sharing, if one SSH session to a host stalls or dies they all do. With connection sharing, all of your separate-looking sessions are running over one underlying SSH transport stream and one underlying TCP connection. In a reliable networking environment, this is no problem. In a networking environment where you sometimes experience network stalls or problems, this means that the moment you have one, all of your connections stall and you can't make new ones. You get to wait out TCP retransmission timeout delays and so on, for everything. And if there's an interruption that's long enough to cause an active session to close itself, you lose everything, including the inactive sessions that would have otherwise survived.

It turns out that this is really annoying and frustrating when (or if) it happens. On the university network at work, it basically never comes up (especially since most of the machines I do this to are on the same subnet as my office workstation), but from home the network stalls, dropped ACKs, and outright brief connection losses happened too often for me to keep my sanity in the face of this. The minor savings in latency for new connections weren't worth the heartburn when something went wrong.

(The most irritating things were and are generally retransmit timeouts. When the network comes back you can easily get into a situation where your new SSH session is perfectly active but the old session that had some output and thus a transmit timeout during the network interruption is just sitting there silently waiting for the TCP timeout to expire and a retransmit to kick off. This can happen on either end, although often it will happen on your end because you kept typing even as the network path was failing.)

Solving our authenticated SMTP problem by rethinking it

Part of our mail system is a mail submission machine. Perhaps unlike many places, this machine has never done authenticated SMTP and as a result has never accepted connections from the outside world; to use it, you have to be 'inside' our network, either directly or by using our VPN (and at that point it just accepts your email). Recently this has been more and more a pain point for our users as it becomes more and more common for devices to move between inside and outside (for example, smartphones).

Unfortunately, one reason we haven't supported authenticated SMTP before now is that it's non-trivial to add to our mail submission machine. There are two tricky aspects. The first is that as far we can see, any easy method to add authentication support to our Exim configuration requires that our mail submission machine be rebuilt to carry our full /etc/passwd (and /etc/shadow). The second is that the mail submission machine still has to support unauthenticated SMTP from internal machines; among other things, all of our servers use it as their smarthost. This requires a somewhat messy and complex Exim configuration, and being absolutely sure that we're reliably telling apart internal machines from external machines and not accidentally allowing external machines to use us without SMTP authentication (because that would make one of our most crucial mail machines into an open relay and get it blacklisted).

(Right now the mail submission machine has a strong defense in that our external firewall simply doesn't allow outside people to connect to it. It has its own access guard just in case, but its accuracy is less important. In the new world we'd have to open up access on the firewall and then count on its Exim configuration to do all the work.)

Exim can use a local Dovecot instance for authentication, but that doesn't help the mail submission machine directly; to run a local Dovecot that did useful authentication, we'd still need a full local /etc/passwd et al. But then we had a brainwave: we already have a Dovecot-based IMAP server.

Rather than try to modify the mail submission machine's Exim configuration to add authenticated SMTP for some connections, we can turn the problem around and do it on the IMAP server instead. The IMAP server already has Dovecot and our full /etc/passwd; all it needs is to have Exim added with a configuration that only does authenticated SMTP. Sure, we wind up with two mail submission machines, but this way we don't have to mix the two somewhat different mail submission roles and we get a much simpler change to our existing machines. People also get a somewhat simpler IMAP client configuration (and one that's probably more normal), since now their (outgoing) mail server will be the same as their IMAP server.

(The actual Exim configuration on our IMAP server can be just a slight variation on the existing mail submission Exim configuration. Insisting on SMTP authentication all the time is an easy change.)

As a side benefit, testing and migration is going to be pretty easy. Nothing is trying to talk SMTP to the IMAP server today, so we can transparently add Exim there then have people try out using it as their (outgoing) mail server. If something goes wrong, the regular mail submission machine is completely unaltered and people can just switch back.

Using an automounter doesn't always help with bad NFS servers

Suppose, not entirely hypothetically, that you have a NFS fileserver that is locking up every so often, that you statically mount its filesystems on your IMAP server, your mail server, your general login server, your web server, and so on, and that as a result when it locks up many of your servers wind up grinding to a halt. Clearly the workaround is to switch from static NFS mounts to an automounter, right?

Not so fast. Switching from static NFS mounts to using an automounter probably won't help you here. The problem is that an automounter only helps with inactive, rarely used filesystems, because these are the only sort of filesystems that it doesn't have mounted. If you have machines that are naturally using these filesystems all the time, as people read their email and serve up their home pages and so on, there's no practical difference between an automounter setup and static NFS mounts. The filesystems are mounted and active all the time and the moment the NFS fileserver goes away you're going to start have problems, as process after process tries to read from them and stalls out.

(In this situation all that switching to an automounter will do is add more moving parts to your system.)

An automounter is a decent solution to unreliable but infrequently used NFS servers, which is the situation it was designed for. It's unfortunately not a particularly effective way to deal with frequently used fileservers that become unreliable, but then nothing is; if you need filesystems and they're not responding, you have problems no matter what.

(You only have two real solutions: make your fileservers reliable or make as many machines as possible not need filesystems from the unreliable fileservers. The latter may require extreme measures like multiple IMAP servers, each one of which only talks to one fileserver.)

The puzzle of packets to your host that your host doesn't respond to

Today we tried to replace an old machine by having a newly built version of it take over from it. We built the new version using a temporary name and IP address, then at the transition time shut down the old version, reconfigured the new version to use the real name, primary IP address, and IP aliases, and rebooted it so it would come up with the new configuration. Unfortunately, when it came up it had a very weird problem: machines on the local network could talk to all of its IP addresses, but machines on other networks could only talk to its primary IP address, not any of the IP aliases. The other IP aliases didn't respond to packets.

To make it more mysterious, during the troubleshooting attempt my coworker ran tcpdump on the server itself and actually saw his pings to an IP alias coming in but not being answered:

www.cs # tcpdump -i eth0 "host workstation.cs"
08:16:22.929594 IP workstation.cs -> support.cs: ICMP echo request, id 18707, seq 33, length 64
08:16:23.929546 IP workstation.cs -> support.cs: ICMP echo request, id 18707, seq 33, length 64

Then after a while (but not a short while) the problem went away; you could ping and otherwise talk to the IP aliases from machines on other networks. Oh, and we could reproduce this (we did it when failing back to the old version, which made us very alarmed).

(There's no firewall involved here, just to cover that.)

What's going on here is the inverse of something I've seen before with outgoing traffic:

You can't tell if packets are really going to your machine without checking the destination Ethernet address. The destination IP alone is not good enough.

Sure, these packets look like they're going to our server. But actually they aren't; they're being sent to the Ethernet address of the old version of the server, not the Ethernet address of the current one. The new version of the server is seeing them for two reasons. First, the switches on the network have aged out the Ethernet address to port association for the old Ethernet address, so the switches have to flood these packets to all ports. Second, tcpdump is running in its default promiscuous mode so it's picking up this flooded traffic (and displaying only the IP level information). The kernel knows better and is quietly ignoring these packets just like it ignores all sorts of other random crud that shows up on the network port.

(If we weren't running tcpdump with the interface in promiscuous mode, the packets probably would be ignored at the hardware level and not even reach the kernel.)

The reason that the packets had the old Ethernet address is that our top level router was caching the IP to Ethernet address association for a surprisingly long time. Hosts on the local network were directly re-ARPing for the IP aliases and getting the new server's Ethernet address, so they could talk to it, but packets from other networks went through the router and the router just used the old Ethernet address it had cached. As for traffic to the server's primary IP working, we think that the Ethernet address for the server's primary IP was getting updated on the router because the server generates outgoing traffic from that IP address, forcing the router to update. The problem went away after a while because the router timed out its cached Ethernet address information, re-ARPed, and finally had the correct new Ethernet addresses.

(Once we went searching on the Internet, we discovered that this is known behavior of our particular make of router. Fortunately there's a way to forcefully purge such a cached entry; unfortunately we're going to have to remember to do this on any migration or manual failover of any machine that has IP aliases. And it's a good thing we're not trying to do automated failover of IP aliases between machines.)

I should document my test plans and their results

Every so often I rebuild one of our servers because, for instance, we're moving it from Ubuntu 10.04 to Ubuntu 14.04 (seeing as 10.04 is about to stop being supported). When I do this, of course I need to test that the new version of the server (and its services) works properly. So far this has been a basically ad-hoc process and I usually haven't actually written down very much about it. But after a recent experience or two, I've come around to the obvious realization that I should change that.

So, the simple statement: I should actively document how to test our services and the results of doing those tests. Not just as lab notes to myself that I might scribble down while I do this (and then keep), but as some degree of formal documentation. There's several things that such documentation is good for, in my view.

First, it tells us at least some tests that we want to do on the next version (because there will be a next version) and how to do them or at least how they were successfully done the last time around. These will wind up being tests both for basic functionality of the service and to cover any special problem points I found during the current build (even if in theory we've worked around them, it's good to test specifically for them for the same reason that you add tests for program bugs to your test suite).

Put simply, if I've gone to all of the work to come up with how to test the current (new) version, writing it down saves me having to redo all of that work the next time around, lets me document any surprise bear traps I found and worked around, and lowers the chances that next time around I'll miss something important. People are fallible, so the more help I give my future self the better.

(As a side benefit this will document the things that we can't test short of the machine actually being in production, and thus want to explicitly test or monitor when we do put it into production.)

Explicitly documenting the results of all of the tests (even to say 'all of these go/no go tests passed') serves both as a checklist and as a record for the future. If we run into problems later on, we can look back to see what we definitely did and did not test for. If we missed a test, well, we can add it. If we did a test and it passed at the time, we know that something changed since then (or the test was not sufficient, or the test environment was not realistic enough).

Some of our tests are performance tests, generally to see if something was good enough. Documenting the results of these tests (and their specific circumstances and methods) is especially useful for retrospective analysis and for comparing the results with other, future tests. What sort of initial IO performance did we get on our new iSCSI backends, for example, and under what specific conditions? What did we consider was 'good enough' 10G network performance, measured with what tools, and how were the machines connected to each other? And so on and so forth. It's one thing to know that we assessed the results as good enough and another thing entirely to be able to go back, see the specific results, and then compare them to what we're getting with new software or new hardware or the like.

(This is important partly because we don't routinely do performance tests; about the only time we do it is when we're building out a new system and want to make sure that it works well enough, whatever that is. Thus, testing the new system is our single best chance to capture performance information and the exact tests we did. If we actually document all of this, in detail.)

Finally and obviously, if we later run into problems with something we can go back to assess both what tests we did and what tests we could reasonably add to cover the problems. If we don't document the tests we did, we're really up in the air about whether we should have caught the problem before we went into production. This is the same theory as preserving your completed checklists so that later you can revisit them to spot what you missed and should do differently next time around.

(This elaborates a lot on a tweet of mine. And yes, part of why I'm writing this is to push myself to actually do this.)

Our brute force solution for port isolation without port isolated switches

We have a problem: a number of our areas don't have anywhere near the network jack density that people need. Sometimes this is easy to deal with; if everyone in an office just needs one particular sandbox network we can just deploy a dumb switch to add more ports for it. But sometimes people need either multiple networks in one office or, more crucially, more ports on one of our port isolated networks, which we really want to stay that way (even between machines in the same office). And that's the real problem, because we just haven't found any reasonably priced fanless small switches that have good support for port isolation (as well as VLANs; we need both in the same switch).

(Some offices are used by only a single person so it's only sort of dangerous if their infected laptop starts attacking their desktop. But other offices have multiple people, for example grad students, and there we really don't want person A's laptop trying to infect person B's laptop.)

So we've adopted a brute force switch intensive solution to the problem. The 8-port fanless switches we use may not support port isolation but they do fully support VLANs, so we put together what is basically a virtual patch panel system. Each port on an office switch is in a separate VLAN, all of which are forwarded over the uplink port. In the wiring closet, this uplink plugs into another switch that then breaks out each of those VLANs to a separate port again. We then plug each port on the wiring closet forwarding switch into an appropriate port in our regular network infrastructure for whatever network is needed. Effectively the wiring closet switch is a patch panel; it patches through its network ports on a 1 to 1 basis to network ports in office(s).

We initially considered a more complicated and obvious version of this where the wiring closet forwarding switch carried some of our non port isolated VLANs as real VLANs (passed to the office switches) and only did the 1:1 port patch panel thing for the port isolated networks. After trying to plan it out we decided that the potential moderate improvement in port density on the forwarding switch simply wasn't worth the extra complexity, including in keeping track of what ports were on what VLAN(s), what exact configuration a replacement 8-port switch would need for any particular office, and so on. Pure port forwarding has the great virtue of being completely generic.

(Doing actual VLANs on the office switches and the port forwarding switch might slightly improve overall network bandwidth if office machines talk to each other on non port isolated networks. In practice this is not the usage pattern on most of our networks; if there are office machines, they're usually talking to servers that are located elsewhere.)

We could have used more 8-port switches as the wiring closet switches, but for various reasons we wound up deciding to use 24-port ones instead (partly because our 24-port switches are designed to be rack mounted while the little 8-port ones generally aren't). A 24-port switch neatly supports three 8-port office switches with no wasted ports, but it does mean we have three varieties of 8-port office switches (one for which set of seven per-port VLANs it uses). In practice this is not a problem; we label the 8-port switches based on whether they are set up for A ports, B ports, or C ports. Then you just make sure you always connect an A port switch to the A port uplink on the 24-port switch.

(And if you get it wrong you find out right away because nothing works since the VLAN numbers don't match up. The 24-port switch is sending the 8-port switch traffic for (eg) VLANs 1 through 7, while the 8-port switch expects and is sending traffic for VLANs 15 through 21. Both sides drop each other's unexpected VLANs and nothing gets through.)

My view of the difference between 'pets' and 'cattle'

A few weeks ago I wrote about how all of our important machines are pets. When I did that I did not strongly define how I view the difference between pets and cattle, partly because I thought it was obvious. Subsequent commentary in various places showed me that I was wrong about this, so now I'm going to nail things down.

To me the core distinction is not in whether you hand-build machines or have them automatically configured. Obviously when you have a large herd of cattle you cannot hand-build them, but equally obviously the current best practice is to use automated setups even for one-off machines and in small environments. Instead the real distinction is how much you care about each individual machine. In the cattle approach, any individual machine is more or less expendable. Does it have problems? Your default answer is to shoot it and start a new one (which your build automation and scaling systems should make easy). In the pet approach each individual machine is precious; if it has problems you attempt to nurse it back to health, just as you would with a loved pet, and building a new one is only a last resort even if your automation means that you can do this rapidly.

If you don't have build automation and so on, replacing any machine is a time consuming thing so you wind up with pets by default. But even if you do have fast automated builds, you can still have pets due to things like them having local state of some sort. Sure, you have backups and so on of that state, but you go to hand care because restoring a machine to full service is slower than a plain rebuild to get the software up.

(This view of pets versus cattle is supported by, eg, the discussion here. The author of that email clearly sees the distinction not in how machines are created but in significant part in how machines with problems are treated. If machines are expendable, you have cattle.)

It's my feeling that there are any number of situations where you will naturally wind up with a pet model unless you're operating at a very big scale, but that's another entry.