Wandering Thoughts archives

Why I increasingly think we're unlikely to ever use Docker

Apart from my general qualms about containers in our environment, I have increasingly wound up thinking that Docker itself is not a good fit for our environment even if we want to use some form of service containerization for various reasons. The problem is access to data.

Our current solution for services gaining access to service data is NFS filesystems from our central fileservers. Master DNS zone files, web pages and web apps for our administrative web server, core data files used by our mail gateway, you name it and it lives in a NFS filesystem. As far as I can tell from reading about Docker, this is rather against the Docker way. Instead, Docker seems to want you to wrap your persistent data up inside Docker data volumes.

It's my opinion that Docker data volumes would be a terrible option for us. They'd add an extra level of indirection for our data in one way or another and it's not clear how they allow access from different Docker hosts (if they do so at all). Making changes and so on would get more difficult, and we make changes (sometimes automated ones) on a frequent basis. In theory maybe we could use (or abuse) Docker features to either import 'local' filesystems (that are actually NFS mounts) into the containers or have the containers do NFS mounts inside themselves. In practice this clearly seems to be swimming upstream against the Docker current.

It's my strong view that much software has ways that it expects to be used and ways that it doesn't expect to be used. Even if you can make software work in a situation it doesn't expect, it's generally not a good idea to do so; you're almost always buying yourself a whole bunch of future pain and heartburn if you go against what the software wants. The reality is that the software is not a good fit for your situation.

So: Docker does not appear to be a good fit for how we operate and as a result I don't think it's a good choice for us.

(In general the containerization stories I read seem to use some sort of core object store or key store as their source of truth and storage. Pushing things to Amazon's S3 is popular, for example. I'm not sure I've seen a 'containerization in a world of NFS' story.)

Everything that does TLS should log the SSL parameters used

I'll start with my tweets:

Every server and client that makes SSL connections should have an option to log the protocols and ciphers that actually get used.
Having logs of SSL protocols/ciphers in use by your actual users is vital to answering the question of 'can we safely disable <X> now?'

As we've seen repeatedly, every so often there are problems uncovered with TLS ciphers, key exchange protocols, and related things. That's certainly been the pattern in the past and a realistic sysadmin has to conclude that it's going to happen again in the future too. When the next one of these appears, one of the things you often want to do is disable what is now a weak part of TLS; for instance, these days you really want to get away from using RC4 based ciphers. But unless you have a very homogenous environment, there's always an important question mark about whether any of your users is unlucky enough to be using something that (only) supports the weak part of TLS that you're about to turn off.

That's the large part of what logging TLS key exchange and cipher choice is important for. If you have such logs, you can say more or less right away 'no one seems to actually need RC4' or 'no one needs SSLv3' or the like, and you can turn it off with confidence. You can also proactively assess your usage of TLS elements that are considered deprecated or not the best ideas but aren't actually outright vulnerable (yet). If usage of problematic elements is low or nonexistent, you're in a position to preemptively disable them.

The other part of logging TLS connection information is that it lets you assess what level of security your users are actually negotiating and what the popular options are. For example, could you tell right now how many of your users are protected by TLS forward security? How widespread is support for and use of elliptic curve cryptography as opposed to older key exchange protocols? And so on and so forth.

(This can also let you assess something about the age of client software and its TLS code, since only new software is likely to be using the latest ciphers and so on. And ancient cipher choices are a good sign of old client software.)

Client logging for things like outgoing SMTP mail delivery with TLS is also important because it tells you something about how picky you can be. If you drop usage of RC4, for example, are you going to be unable to negotiate TLS with some mail servers you deliver mail to regularly, or will you be basically unaffected? How many MTAs do you try to deliver to that have too-small Diffie-Hellman parameters? There are tradeoffs here, but again having information about actual usage is important for making sensible decisions.

Don't support shortened versions of your domain names in email addresses

I'll start with the basic background: the university here is divided up into a lot of departments and divisions and groups and so on. Most of them have subdomains under our top level domain, for instance cs.utoronto.ca (and then hosts under their subdomain), and many of them run (or ran) their own email systems, creating email addresses like <user>@cs.toronto.edu.

We have been providing email to people for what is now quite a long time. Way back at the beginning of that time, so long ago that DNS itself was very shiny and new, we decided to let people send email using shortened versions of all of the university's subdomains. Rather than having to send email to '<user>@ece.toronto.edu', you could send it to just '<user>@ece' and the mail system would work out where it should really go. This saved a bunch of people a bunch of typing back in the days when you had to type all of the email addresses out and was much appreciated.

Of course people got used to using these short domain names. Of course people put them everywhere, because they were generally supported; you could put them in .forward files, you could put them in simple mailing lists, and so on. And as a result of this, of course we've never depreciated this feature.

If you guessed that supporting this feature in modern MTAs is a slowly increasing amount of work, you're correct; it absolutely is. If you guessed that it kind of goes wrong sometimes, you're also correct (I discovered a case today). So I have a simple suggestion: if you're tempted to do this because it sounds like a neat convenience feature, don't. Forcing everyone to use fully canonicalized domain names will make your life much simpler and will avoid a certain amount of pain.

(It's probably going to get worse, since DNS keeps growing more and more weird top level domains. Sooner or later we're going to have a serious collision between a new one and one of our own subdomains.)

The hard thing about this is that I don't think we were wrong to support shortened versions of the university domains way back when, and depreciating them at any time since then would undoubtedly have been a lot of work and not have been appreciated at all. But I still think that in the modern world it would be better if we weren't supporting them any more. In a world with MUAs that have address autocompletion and so on, I think they're not necessary and they do cause problems.

(When I say that 'we' did this way back when, I mean the people who were sysadmins here in those days, not me personally.)

Sidebar: one way to have a collision heartburn

Suppose that you have a university subdomain, let's call it aa for illustration; this subdomain has a wildcard A record for its own reasons. There's also a aa TLD (here it would be a country), and one of your users tries to send email to '<user>@misspelled.aa'. With the best of intentions, your MTA is almost certainly going to wind up trying to deliver this email to the aa subdomain wildcard A record instead of rejecting it as 'no such domain exists'.

(If the target of the A record doesn't respond to SMTP connections, your MTA will be retrying it for some time (especially if you've configured high timeouts for in-university addresses).)

What SSH keys in your .ssh/config will be offered to servers

Recently I've become aware that some things about specifying keys in .ssh/config don't work quite the way I absently thought they did. Let's start with a simple situation:

Host *
  # encrypted & usually loaded into
  # ssh-agent
  IdentityFile /u/cks/.ssh/ids/key-ed2

Host github.com
  IdentitiesOnly
  IdentityFile /u/cks/.ssh/ids/github

When I set up my .ssh/config, this configuration looked like it would offer Github only my Github specific key. It doesn't; it offers key-ed2 as well (in fact it offers key-ed2 first). What's happening is that IdentityFile is cumulative, and so every IdentityFile in every matching Host stanza is another key that will be offered. When you connect to github.com, both the 'Host *' and the 'Host github.com' stanzas match, so the IdentityFile directives from both are used.

(Where this matters is that servers have a limit on how many keys you can offer them before they drop your connection. This can make it important to send them the right key first or at least very early.)

This can make IdentitiesOnly into an essentially meaningless directive. If the only key we ever load into our ssh-agent is the key-ed2 generic key, that's what happens here; it's always a candidate IdentityFile key so it will never be excluded. IdentitiesOnly only matters if you load extra keys from outside your .ssh/config (or at least from outside a 'Host *').

If ssh-agent is not running or does not have keys loaded, you can get around this by reordering your .ssh/config. Without ssh-agent, keys are offered to servers in the order of IdentityFile directives. If the 'Host github.com' stanza is before 'Host *', the Github specific key will be offered first (and then accepted). Unfortunately this doesn't work if you have keys loaded into ssh-agent, as keys from ssh-agent are always tried first before keys from .ssh/config (in the order they were loaded into ssh-agent, not the order in .ssh/config). This is the case even with IdentitiesOnly specified.

The rather awkward and not very scalable hack fix for this is to specifically exclude some destinations from the generic 'Host *' stanza:

Host * !github.com !elsewhere.com ...
  ...

As long as you specify IdentitiesOnly in the host-specific Host stanza, things work right (and the order in .ssh/config doesn't matter). This is clearly not scalable if you have very many of these hosts; every time you add a new one, you need to remember to add it to the 'Host * ...' exclusion as well or things will go subtly wrong. And if you have truly generic parameters other than IdentityFile, you'll need a separate 'Host *' stanza for them.

The other workaround is to only use your general keys through ssh-agent and strip them out of .ssh/config (here we'd delete the IdentityFile line for key-ed2). This costs you the ability to use them with ssh when ssh-agent is not running (or doesn't have them loaded), but it makes IdentitiesOnly do what we want it to.

As far as I know there's no other way around this, although I'd love to be wrong. The whole situation is kind of irritating and I wish there was a version of IdentitiesOnly that meant 'only identities from this specific Host/Match stanza'. It's also irritating that such an important issue as key choice and key ordering is not discussed explicitly in the OpenSSH client documentation.

(Of course this also means that OpenSSH is free to change any or all of this on you in future versions, because they never made any official promises about how it worked. It's almost all 'undocumented behavior' and we're on our own.)

When SSH needs you to decrypt your keys

Here's something nice but vaguely surprising that I just learned about how SSH handles encrypted keys. Start with the following in your .ssh/config:

Host *
  # encrypted key:
  IdentityFile /u/cks/.ssh/ids/key-ed2

Here's the question: suppose that you're not running ssh-agent and you try to ssh off to a host where you have an account that doesn't use this SSH key. Do you get prompted for the passphrase for your encrypted key? After all, your ssh client is going to try to use it to authenticate you, since it's your default key.

My naive expectation was that you would, but it turns out that I'm wrong. As you can see with 'ssh -v' (although it's not quite explicit), the SSH protocol is that the client first asks the server 'can I authenticate with this public key?' before proceeding to actually do so. When you have what I've casually called an encrypted SSH key, only the private key is actually encrypted; the public key is stored in the clear and your SSH client can send it off to the server without having to ask you for a passphrase. If the server declines the public key, that's it. Only if the server tells ssh 'sure, now prove you know the private key' does ssh need you to decrypt it and thus asks you for the key's passphrase.

In short, ssh only needs to know your key passphrase if it's about to authenticate you with it.

This has two consequences. The first is that if you get prompted to decrypt a SSH key, that's a good sign; you're about to be accepted if you can do so (well, normally, unless the server is playing funny games). This is unlike plain password prompts, where the server will demand a password even if you have no access and the remote login doesn't even exist.

The second is that it is relatively harmless to have additional encrypted SSH keys set up in your .ssh/config and offered to remote hosts. Specifically, you won't have your ssh commands interrupted to provide passphrases only to find out that you did it for nothing.

(Until I found this out just now it was a concern of mine, since I have a mix of encrypted general keys and narrow purpose unencrypted keys in .ssh/config and sometimes try things that ssh off to 'narrow purpose' destinations when I don't have ssh-agent up and active.)

Sysadmin use of email is often necessarily more or less interrupt driven

One of the things that people commonly do with virtual screens is to put their email client off in a separate virtual screen so they can ignore it and avoid having it interrupt them. As I mentioned when I wrote up my virtual screen usage, I don't try to cordon off email this way. Fundamentally this is because as a sysadmin, I feel my use of email is necessarily interrupt driven.

Allowing email to interrupt me certainly can derail my chain of thought when I'm coding or working on a hard problem. But at the same time it's absolutely necessary, because that email may carry news of an emergency or a high priority issue that I need to handle more or less right away. I almost never have the option of ignoring even the possibility of such things, so almost all of the time I have to allow email to interrupt me. The best I can do is contrive low distraction email monitoring so that when I'm in a flow state it distracts me as little as possible.

So I can stop there, right? No, not so fast. What this really argues is that email is a bad way of receiving high priority information like alerts. Because it mixes high priority information with much less important messages, I have to allow even unimportant things to interrupt me at least a bit just so I can figure out whether or not I can ignore them. If alerts and priority items came in through another channel, I could readily ignore email during high focus times.

(There are always going to be days where all I do is fiddle around with stuff and swat things as they come up; on those days, I'd read and handle everything right away.)

Of course the problem is that there is no good other channel today, at least for us. Oh, with work you can build such a thing (possibly outsourcing parts of it to companies who specialize in it), but there's very little in the way of a canned out of the box solution. Plus there's the problem of getting people use your new 'urgent things' channel when they have an urgent thing and of course not using it when they don't have an urgent thing (with the associated issue of having people know whether or not their issue is urgent).

(Life is likely somewhat easier if you can assume that everyone has a smartphone, perhaps by issuing them one, but that is not something that's true in our environment.)