Wandering Thoughts archives

Why we'll continue to have local compute servers in the future

On the surface and ignoring some issues, one of the easiest things to offload into cloud computing should be our compute servers. But in practice in our environment it isn't so; as a result I expect that we'll be running some degree of compute servers for years to come.

The great advantage that local compute servers have is that the incremental cost for a grad student to use them is zero. The grad student and/or their professor does not have to find some money (even spare change money) in the budget to buy N hours of compute time on the cloud of your choice; they just log in and start running things. This may not run anywhere near as fast as it could in a cloud but in a university environment this doesn't matter. What matters is that no one has to find any money, anywhere, to have their grad student do some computing.

(This has additional effects. For instance, you don't have to worry that one grad student in your group is going to accidentally run something unusually compute intensive and drain the group's computing budget. Of course a professor could get around that by setting individual computing budgets for their grad students, but then they have to spend time coming up with these budgets and then listening to grad students beg for more compute time. This is an instance of the general thing that money in universities always involves bureaucracy.)

In a university, it doesn't matter that the total cost of ownership for some compute servers might be higher than just getting the same computing in the cloud. What matters is that you can buy the compute servers with one-time money and then just run them for free (generally we can ignore the other ongoing costs for various reasons), whereas using cloud computing requires ongoing injections of money over time. One time money is much easier to get and once the department has spent it on a compute server, everyone in the department can use it regardless of their current funding levels.

(Varying funding levels is the other elephant in the woodpile. My sense is that it is politically much easier to buy a compute server for the department, even if it is funded out of someone's grant, than it would be to transfer money from flush people to lean people in order to fund the latter's cloud computing. The net monetary effect could be the same but the perception would likely be very different.)

Can we really use the cloud?

One of the things I think about every so often is whether we can make much use of cloud computing (by which I mean external cloud providers, not trying to set up some sort of large-scale university cloud or virtualization environment). Although I may be suffering from a lack of imagination, so far I keep coming up mostly empty.

The problem is that most of our services are internally facing; they're things we provide to our local users. I see the cloud as great for externally facing services (things like websites that the outside world uses) and things that sit behind them to support them, but this is mostly not what we have. Shipping local file storage and so on off to the cloud seems a little counter-productive and also bandwidth (and latency) limited. Something like IMAP email could in theory be hosted in the cloud instead of locally but currently it's entangled with our local file storage; detaching it would have various awkward consequences that would be annoying to the users.

(One argument is that people are already implicitly moving to the cloud as it is, by shifting email to GMail and storing stuff in Dropbox and so on. But even if they are, I don't think we can displace high quality services like GMail and Dropbox with our own efforts.)

One issue for remoting some services off into the cloud is how we deal with interactions with local fileservers and login services. Local fileservers are important for local heavy-duty computing, so how do we bridge them and either remote servers in the cloud using the local fileservice or even remote fileservice (for example if we pushed our web server into the cloud)? I suppose we could nail up IPSec tunnels and the like, but that feels both fragile and worrisome from a security perspective (assuming that NFS even works very well over WAN latencies).

(I'm assuming that issues of cost, network reliability, and any legal problems can all be magically dealt with. This is not necessarily realistic (and some of them definitely are concerns), but the general enthusiasm for clouds basically handwaves them all anyways.)

The easy way to wind up with multiple subnets on a single (V)LAN segment

In theory, a nice proper network is supposed to have only a single IP(v4) subnet running over any given network segment. But this is not actually fully required; if you're sufficiently perverse you can run multiple subnets over the same physical network. Some people are now asking how on earth you would ever get into such a crazy situation. Well, sit back, I have a story for you.

Suppose that you use private subnets and in particular you put each group in your organization in its own subnet (around here, a group is a research group). Often when you start doing this, you will give a group a /24 (because that's the simple approach). But the thing about groups is that sometimes they, well, grow. Some energetic professor will get a grant for some equipment here and another one will get a grant for a cluster there and before you know it, a /24 just isn't enough space.

In a perfectly ideal world you would have allocated all those initial /24s such that they could grow substantially, at least into /22s. In a less than ideal world you just allocated the /24s sequentially and there is no room to expand them. No problem; private IP address space is capacious, so you can just start over by giving a group, say, a /16. But this new /16 is not contiguous with the group's old /24; you can't just expand the segment's subnet mask and be done.

In an ideal world you could arrange with the group to have a flag day for the changeover between their old subnet and their new subnet; on that day, all machines would go down, all IP addresses would be changed, and the group would completely migrate from their /24 to their /16. In the real world the group is going to laugh politely at you when you propose this, because they (rightfully) have much more important things to do with their time than go through a large disruption just for your convenience. Instead you're forced into the obvious step: you just add the new /16 to their existing network segment alongside the old /24. The group will put new machines in the /16 and migrate old machines from the old /24 to the new /16 at their convenience (generally very slowly).

This does have some small drawbacks. The largest one is that all of the group's traffic between their two subnets is making an unnecessary round trip through your router (and if your router is a firewall, through your firewall rules; you'll wind up making a special exemption for their internal traffic). Hopefully there won't be much of it; if there is, you can sometimes use it to motivate the group into moving some machines.

(The obvious workaround for heavy-traffic machines is to give them a second IP address alias on the other subnet, so they know that they can reach it directly.)

Sidebar: easy IP address assignment in the new /16

We generally suggest to groups that the low /24 of the new /16 be reserved for a one to one mapping of machines in the old /24. Groups don't necessarily have to convert (or dual-home) machines this way, often they renumber them during a move in order to (re)organize their network, but it makes it easy to do a quick conversion or a dual-homing; just change the network without changing the host IP.

How we make Exim cut off bounce loops

Under certain circumstances it's possible to get bounce loops; a message bounces, the bounce for the message bounces, the bounce of the bounce then bounces, and you repeat endlessly (possibly until something explodes). When this happened to us, we decided to fix our Exim configuration so that it would detect and suppress these bounce loops (or at least as many of them as possible).

How we do it is a close relative of how we make Exim discard bounces of spam. First, we created a custom bounce message by setting bounce_message_file to an appropriately formatted file. This lets us add headers to bounce messages and those headers can refer to headers in the message bouncing. So we added a very special header:

X-CS-Bounce-Count: I$h_x-cs-bounce-count:

I have bolded the little bit of magic. What this does is count how many bounce-of-bounces we've seen by giving every such bounce one more 'I' than the previous bounce had (if there's no previous bounce the header is empty and we start out with one I).

We could handle detecting and discarding messages with too high a bounce count entirely in an Exim router, but it turns out that it is much easier and more convenient to do most of the work in an Exim filter. So what we have is a simple router that runs all messages through an Exim filter (although now that I look at it, we should make the router conditional on the presence of the header because there's no point otherwise). Thus we get the router:

eat_looping_bounces:
    debug_print = "R: eat_looping_bounces for $local_part@$domain"
    driver = redirect
    file = <dir>/exim-loop-filter
    allow_filter
    allow_freeze
    user = Debian-exim
    no_verify
    no_expn

(This router should be among your very first routers to insure that it runs before any other router that could conceivably generate a bounce.)

The Exim filter itself (with comments removed) is simply:

logfile /var/log/exim4/discardlog
if ${strlen:$h_x-cs-bounce-count:} is above 4
then
    logwrite "$tod_log junked looping-bounce $message_id from <$sender_address> to <$local_part@$domain> subject: $h_subject:"
    seen finish
 endif

(We allow more than one bounce just in case, as a safety measure. When I put all of this together I didn't want to sit down and go through the work to carefully make sure that we could never wind up with a bounce counter greater than one in a legitimate situation.)

Unlike our discarding of bounces of spam we don't carefully guard this router (and this filter) with conditions to make sure that we're really truly dealing with local bounces. The thing about bounce loops is that they can easily involve outside machines as well, so we want to squelch them whenever they pass through our mail machine.

Sidebar: how bounce loops happen

Some of my readers may now be wondering how on earth you get a bounce loop, since bounces are supposed to be sent using a null sender address (often written as '<>') and all messages to the null sender (bounces included) are just discarded (to stop exactly this sort of loop). The sad answer is that not all programs that resend messages are careful to preserve the null sender address on email they send out. In particular this includes the mail forwarding done by our version of procmail when it's run by a user from their .forward; instead it winds up changing the sender address to the user's email address (because it actually resubmits the email, exactly as if the user had sent it in the first place). This creates an immediate loop if the destination of the forwarding ever refuses some piece of email; the bounce of the refusal goes to the user, their .forward and procmail re-forwards it to the destination, the destination refuses it again, the system generates a new bounce to the user, repeat endlessly.

(The incident that saw us discover this issue managed to somehow multiply the email as it looped the bounces around. The result was quite dramatic.)

Why you should never use '/bin/sh -c ...' in configuration files

Every so often some sysadmin has a problem configuring program A to run a command in just the right way. Maybe they decide they need to run multiple commands, or they need a complex command line, or there's any number of other reasons. So they decide to get out the big guns to solve their problem:

some_option = /bin/sh -c "CMD ARG 'arg 2' $somevar ....."

WRONG. This is a terrible idea (as I mentioned before).

The problem with solving your problems with /bin/sh this way is that you have just thoroughly destroyed absolutely everything that program A is possibly doing to let you securely run programs. It can be the most careful program in the world, it can do absolutely everything right, and you have just totally defeated all of its efforts. There is almost no way to make this sort of command invocation secure in the face of malicious input, and even trying moderately hard requires determined and careful manual effort on your part. That $somevar variable expansion? That's a land mine waiting to go off unless you both quote it and then make sure that it doesn't have quote-breaking characters inside it.

Fundamentally this use of /bin/sh is exactly the sort of string pasting anti-pattern that got everyone all of those SQL injection attacks of the last N years. Everyone learned their lesson (we hope) when it came to putting user input in SQL statements (and that lesson is escape absolutely everything, carefully use placeholders); we should not be repeating the same mistake when it comes to shell command lines. Sadly, people do all the time.

(They get away with it in large part for the same reason that people got away with bad SQL habits; most input isn't malicious or accidentally dangerous.)

Theoretically, one answer to this problem is very carefully quoting every variable expansion and the like that goes into the shell command line. But that's a terrible answer in practice because it demands constant attention and vigilance on the part of everyone who writes this sort of configuration stanza (and it requires program A to support full and completely correct shell quoting in the first place). A much better answer, one that is much more resilient in the face of inevitable human error, is to let program A do its job of securely running simple commands and put all of the complexity in a shell script.

(I'm not even going to try to inventory the quoting bear traps that are waiting in such a shell command line. I'm not sure I could get them all and I don't want to encourage people by providing a list of workarounds and so on.)

Update: as correctly pointed out in comments the real SQL fix is to use proper SQL placeholders, which totally separate the SQL statement from the parameters and thus make it totally safe. See the comments for more.

The mythology of spending money on things, or not doing so

A commentator on my entry on considering FreeBSD instead of Illumos wrote in part (about paying Oracle for Solaris 11 instead of going with FreeBSD):

Unless your data is not really worth to you $1-2k per year and then you are willing to run it on other platforms. In this case FreeBSD is fine choice.

When I read this (and similar things in other bits of the comment) it made me viscerally angry. Whether or not the commentator realized it this is FUD, and FUD of a particularly insidious kind for sysadmins.

There are two things wrong with this statement. The first is the idea that paying money is necessary to get quality results. In reality paying money is almost uncorrelated with results, for many reasons including that salespeople for most vendors have a lot of incentives to make a sale but almost no incentives to make you happy. You do not need to pay money (especially lots of money) to get a good storage system and paying (lots of) money is no guarantee that the resulting storage system will be any good. There are many existence proofs of this out in the world, and similarly many existence proofs for other software.

The second is the extremely toxic attitude that if you aren't paying money (or aren't willing to pay money) that your data 'is not really worth' much to you, that you aren't taking things seriously, and so on. This is a combination of a guilt trip ('if you actually cared, you'd spend the money') and a common scare tactic ('that you didn't spend money will demonstrate to people that you didn't really care and didn't do it right'). It's also invoked as a way of dismissing people who don't spend money; clearly they aren't really serious about whatever they're doing (and if it's you, implicitly you aren't).

Good sysadmins are very sensitive to the idea that we aren't being serious about data that's entrusted to us. This sort of FUD either works great (at getting us to spend money) or drives us into paroxysms of anger at the quiet insinuations it contains. The latter is especially likely if we are being sold what we consider to be snake oil.

(It sometimes also contains the insinuation that we haven't realized that we could spend money or that it might be worthwhile to do so. This is generally not at all the case; if we are not spending money, we've made our best attempt at a rational analysis of the situation and thus have good reasons for our decision. The same is true for spending money, but it's less common for people to suggest that you shouldn't have spent money for something.)

(I had other reactions to the other portions of their comment.)