Wandering Thoughts archives

Thinking about the different models of supplying computing

It's the time of year when new graduate students show up here, so one of the things on my mind has been the various ways that computers can be supplied to people in an environment like ours. There are at least three that come to mind.

First is the 'bring your own device' model where every incoming graduate student (or professor) is expected to bring their own computer (probably a laptop) and, as a corollary, to look after it. Perhaps we'd supply some niceties like external screens to hook up to them. The BYOD approach is popular partly because any number of people are going to do this anyways.

Then there is the 'hardware only' model, where we hand a computer to every new graduate student but make no attempt to manage or control it beyond that; the graduate student can run whatever they want in whatever configuration they want. Probably we'd preinstall some OS in a recommended configuration just for convenience (and many grad students would leave it as-is). Lots of people like this model for its freedom and similarity to the BYOD experience (at least until the OS install blows up in their face).

The final model is managed desktops, where we both supply hardware and maintain the OS installed on it. On the one hand, we guarantee that it works right; on the other hand, people lose the freedom to run whatever they want and have to generally live with our choices. 'We don't support that' will probably get said a lot.

(Note that these are not necessarily a good set of options for any environment other than our peculiar one.)

As you might suspect, in practice right now we have a mix of all three options. The historical evolution of our environment is that we started out providing fully managed computing because computing was too expensive for any other answer, but over time the decrease in computing costs (especially compared to staff costs) has caused more and more people to shift towards BYOD and 'here, have a box'.

(I will skip a discussion of trying to do managed Windows installs and just say that we were and are primarily a Unix shop without much expertise in that area. This leads to non-technical issues beyond the scope of this entry.)

I'm mulling this over partly because how computing get supplied to people has a big impact on what services they're interested in consuming from us (and how). For one obvious example, in the days when we provided serial terminals on everyone's desk, having Unix servers for people to log in to was a big deal and they were very important. Today an increasing number of people here have probably only used our login servers to change their password.

(Since we're a Computer Science department, you could actually argue that we should actively push people to interact with Unix because Unix is an important part of effective, practical computing and so something they should be learning. But that's another debate entirely.)

Why I wind up writing my own (sysadmin) tools

I have a tendency to write my own versions of tools, like bulk IO measurement tools, versions of netcat, and network bandwidth reporters. Historically there have been two official reasons for this and a third unofficial one.

First, when I write a tool myself I know exactly what it's doing and what its output means. This has historically been a problem with other people's tools, including system supplied ones (eg). If I'm going to wind up reading the source of a program just to be sure I really understand what it's telling me, I may not be saving much time over writing my own.

(Also, if I write it myself the tool can generally wind up doing exactly what I want in the way that I want it. Other people's tools may have various sorts of rough edges and little annoyances for me.)

Second, because finding and testing and investigating the existing potential options is generally a pain in the rear. People have written a very many tools for doing disk IO benchmarking, for example, and as an outsider it is all a big mess. I'm honest enough to admit that for almost anything I want, there probably are tools out there that would meet my needs and make me happy; the problem is finding them and sorting out the wheat from the chaff. It's unfortunately easy for it to be less work and less frustration to write my own, or at least to feel like it is or will be.

(The frustration primarily comes from investing time into things that turn out to not be useful despite all of the effort, especially if they have active irritations. And most programs have active irritations somewhere.)

The third, unofficial reason is that programming is fun. When it's going well, it's a pure shot of the 'solving problems' drug that I get only in moderation when I'm doing less glamorous activities like building and testing new systems (and never mind the humdrum daily routine; the problems I solve there are all very small). It's especially fun when I contrast it with the slogging work of searching the Internet for a pile of programs that might perhaps do what I want, getting copies of all of them, testing each out, and throwing most of them away. That's not solving problems, that's shuffling files around.

One view on practical blockers for IPv6 adoption

I recently wound up reading Russ White's Engineering Lessons, IPv6 Edition (via), which is yet another meditation by a network engineer about why people haven't exactly been adopting IPv6 at a rapid pace. Near the end, I ran across the following:

For those who weren't in the industry those many years ago, there were several drivers behind IPv6 beyond just the need for more address space. [...]

Part of the reason it's taken so long to deploy IPv6, I think, is because it's not just about expanding the address space. IPv6, for various reasons, has tried to address every potential failing ever found in IPv4.

As a sysadmin, my reaction to this is roughly 'oh god yes'. One of the major pain points in adding IPv6 (never mind moving to it) is that so much has to be changed and modified and (re)learned. IPv6 is not just another network address for our servers (and another set of routes); it comes with a whole new collection of services and operational issues and new ways of operating our networks. There are a whole host of uncertainties, from address assignment (both static and dynamic) on upwards. Given that right now IPv6 is merely nice to have, you can guess what this does to IPv6's priority around here.

Many of these new things exist primarily because the IPv6 people decided to solve all of their problems with IPv4 at once. I think there's an argument that this was always likely to be a mistake, but beyond that it's certainly made everyone's life more complicated. I don't know for sure that IPv6 adoption would be further along if IPv6 was mostly some enlarged address fields, but I rather suspect that it would be. Certainly I would be happier to be experimenting with it if that was the case.

What I can boil this down to is the unsurprisingly news that large scale, large scope changes are hard. They require a lot of work and time, they are difficult for many people to test, and they are unusually risky if something goes wrong. And in a world of fragile complexity, their complexity and complex interactions with your existing environment are not exactly confidence boosters. There are a lot of dark and surprising corners where nasty things may be waiting for you. Why go there until you absolutely have to?

(All of this applies to existing IPv4 environments. If you're building something up from scratch, well, going dual stack from the start strikes me as a reasonably good idea even if you're probably going to wind up moving slower than you might otherwise. But green field network development is not the environment I live in; it's rather the reverse.)

Two factor authentication and emergency access to systems

One of the things that makes me hesitant to whole heartedly embrace two factor authentication for system administration is how to deal with emergency system access in an exceptional situation. A lot of 2FA systems seem to involve daemons and central services and other things with a lot of moving parts that are potentially breakable, especially in extreme situations like partial network failures or machine failures. If you require all of this to be working before you can get in as a sysadmin or as root, you may have serious problems in emergencies. But if you create some bypass for 2FA, you're at least weakening and perhaps defeating the protections 2FA is supposed to be giving you; an attacker just needs to compromise your back door access instead of your regular access.

Some organizations guard sufficiently sensitive information and important systems that the answer is 'in an emergency, we go to the machine room and boot from recovery media for manual access'. This does not particularly describe us, for various reasons; we really would like to be more resilient and recoverable than that (and we are towards that now, since we're not using 2FA or LDAP or the like).

It looks like some 2FA systems can be configured to use purely local resources on the machine, which at least gets you out of relying on some central server to be up and talking to the machine. Relying on the local 2FA software to be configured okay and working right is probably no worse than things like relying on the overall PAM configuration to be working; you can blow your own foot off either way and at least you can probably test all the behavior in advance of a crisis.

The other side of emergency system access is network access in situations where you yourself do not have both your 2FA and a system that's capable of using it. This won't be an issue in a place where all sysadmins are issued laptops that they carry around, but again that's not our situation. Closely related to this is the issue of indirect system access, where instead of logging in directly from your desktop to the fileserver you want to log in to another machine and copy something from it to the fileserver. Your desktop has access to your good 2FA device but the other machine doesn't, no more than it has access to your SSH keypairs.

(I suspect that the answer here is that you set the system up so you can authenticate either with a 2FA protected SSH keypair or with your password plus a 2FA code. Then your desktop uses the 2FA protected keypair and on other machines you fall back to password plus 2FA code. Maybe you have multiple password plus code setups, one using a 2FA fob and one using a SMS message to your cellphone.)

What I want out of two factor authentication for my own use

Locally, we don't currently use two factor authentication for anything (for reasons that in large part boil down to 'no one has money to fund it'). I don't know if we can change that, but I'd like to at least move in that direction, and one part of any such move is trying to think about what I want out of a two factor authentication system. Since this is a broad area, I'm focusing on my own use today.

Right off the bat, any two factor authentication system we deploy has to be capable of being used for only certain accounts (and only for certain services, like SSH logins). The odds that we'll ever be able to deploy something universally is essentially nil; to put it crudely, no one is going to pay for two-factor authenticators for graduate students (and we can't assume that all grad students will have smartphones). Similarly it's unlikely that we'll ever be able to get all our services to be 2FA capable.

For my own use, what I care about is SSH access and what I want out of 2FA is for it to be as convenient as SSH with an unlocked keypair (while being more secure). I log in to and out of enough of our machines every day that having to enter a 2FA password or challenge every time would be really irritating. At the same time I do want to have to unlock the 2FA system somehow, since I don't want to reduce this down to 'has possession of a magic dongle'. It'd be ideal if some SSH authentication could be configured to require explicit 2FA with a password I have to type as well as whatever 'proof of object' there is (ideally more or less automated).

(Oh, and all of this needs to work from Linux clients to Linux and OmniOS servers, and ideally OpenBSD servers as well.)

Based on some Internet searching today, it seems the way many people are doing this is with a Yubikey Neo from Yubico. They're even cheap enough that we might be able to buy at least one to experiment with (it helps that they don't require expensive software licenses to go with them). If there are other reasonably popular alternatives, they don't seem to come up in my Internet searches.

(A smartphone based solution is not a good one for me because I don't have a smartphone.)

Why I increasingly think we're unlikely to ever use Docker

Apart from my general qualms about containers in our environment, I have increasingly wound up thinking that Docker itself is not a good fit for our environment even if we want to use some form of service containerization for various reasons. The problem is access to data.

Our current solution for services gaining access to service data is NFS filesystems from our central fileservers. Master DNS zone files, web pages and web apps for our administrative web server, core data files used by our mail gateway, you name it and it lives in a NFS filesystem. As far as I can tell from reading about Docker, this is rather against the Docker way. Instead, Docker seems to want you to wrap your persistent data up inside Docker data volumes.

It's my opinion that Docker data volumes would be a terrible option for us. They'd add an extra level of indirection for our data in one way or another and it's not clear how they allow access from different Docker hosts (if they do so at all). Making changes and so on would get more difficult, and we make changes (sometimes automated ones) on a frequent basis. In theory maybe we could use (or abuse) Docker features to either import 'local' filesystems (that are actually NFS mounts) into the containers or have the containers do NFS mounts inside themselves. In practice this clearly seems to be swimming upstream against the Docker current.

It's my strong view that much software has ways that it expects to be used and ways that it doesn't expect to be used. Even if you can make software work in a situation it doesn't expect, it's generally not a good idea to do so; you're almost always buying yourself a whole bunch of future pain and heartburn if you go against what the software wants. The reality is that the software is not a good fit for your situation.

So: Docker does not appear to be a good fit for how we operate and as a result I don't think it's a good choice for us.

(In general the containerization stories I read seem to use some sort of core object store or key store as their source of truth and storage. Pushing things to Amazon's S3 is popular, for example. I'm not sure I've seen a 'containerization in a world of NFS' story.)

Everything that does TLS should log the SSL parameters used

I'll start with my tweets:

Every server and client that makes SSL connections should have an option to log the protocols and ciphers that actually get used.
Having logs of SSL protocols/ciphers in use by your actual users is vital to answering the question of 'can we safely disable <X> now?'

As we've seen repeatedly, every so often there are problems uncovered with TLS ciphers, key exchange protocols, and related things. That's certainly been the pattern in the past and a realistic sysadmin has to conclude that it's going to happen again in the future too. When the next one of these appears, one of the things you often want to do is disable what is now a weak part of TLS; for instance, these days you really want to get away from using RC4 based ciphers. But unless you have a very homogenous environment, there's always an important question mark about whether any of your users is unlucky enough to be using something that (only) supports the weak part of TLS that you're about to turn off.

That's the large part of what logging TLS key exchange and cipher choice is important for. If you have such logs, you can say more or less right away 'no one seems to actually need RC4' or 'no one needs SSLv3' or the like, and you can turn it off with confidence. You can also proactively assess your usage of TLS elements that are considered deprecated or not the best ideas but aren't actually outright vulnerable (yet). If usage of problematic elements is low or nonexistent, you're in a position to preemptively disable them.

The other part of logging TLS connection information is that it lets you assess what level of security your users are actually negotiating and what the popular options are. For example, could you tell right now how many of your users are protected by TLS forward security? How widespread is support for and use of elliptic curve cryptography as opposed to older key exchange protocols? And so on and so forth.

(This can also let you assess something about the age of client software and its TLS code, since only new software is likely to be using the latest ciphers and so on. And ancient cipher choices are a good sign of old client software.)

Client logging for things like outgoing SMTP mail delivery with TLS is also important because it tells you something about how picky you can be. If you drop usage of RC4, for example, are you going to be unable to negotiate TLS with some mail servers you deliver mail to regularly, or will you be basically unaffected? How many MTAs do you try to deliver to that have too-small Diffie-Hellman parameters? There are tradeoffs here, but again having information about actual usage is important for making sensible decisions.

Don't support shortened versions of your domain names in email addresses

I'll start with the basic background: the university here is divided up into a lot of departments and divisions and groups and so on. Most of them have subdomains under our top level domain, for instance cs.utoronto.ca (and then hosts under their subdomain), and many of them run (or ran) their own email systems, creating email addresses like <user>@cs.toronto.edu.

We have been providing email to people for what is now quite a long time. Way back at the beginning of that time, so long ago that DNS itself was very shiny and new, we decided to let people send email using shortened versions of all of the university's subdomains. Rather than having to send email to '<user>@ece.toronto.edu', you could send it to just '<user>@ece' and the mail system would work out where it should really go. This saved a bunch of people a bunch of typing back in the days when you had to type all of the email addresses out and was much appreciated.

Of course people got used to using these short domain names. Of course people put them everywhere, because they were generally supported; you could put them in .forward files, you could put them in simple mailing lists, and so on. And as a result of this, of course we've never depreciated this feature.

If you guessed that supporting this feature in modern MTAs is a slowly increasing amount of work, you're correct; it absolutely is. If you guessed that it kind of goes wrong sometimes, you're also correct (I discovered a case today). So I have a simple suggestion: if you're tempted to do this because it sounds like a neat convenience feature, don't. Forcing everyone to use fully canonicalized domain names will make your life much simpler and will avoid a certain amount of pain.

(It's probably going to get worse, since DNS keeps growing more and more weird top level domains. Sooner or later we're going to have a serious collision between a new one and one of our own subdomains.)

The hard thing about this is that I don't think we were wrong to support shortened versions of the university domains way back when, and depreciating them at any time since then would undoubtedly have been a lot of work and not have been appreciated at all. But I still think that in the modern world it would be better if we weren't supporting them any more. In a world with MUAs that have address autocompletion and so on, I think they're not necessary and they do cause problems.

(When I say that 'we' did this way back when, I mean the people who were sysadmins here in those days, not me personally.)

Sidebar: one way to have a collision heartburn

Suppose that you have a university subdomain, let's call it aa for illustration; this subdomain has a wildcard A record for its own reasons. There's also a aa TLD (here it would be a country), and one of your users tries to send email to '<user>@misspelled.aa'. With the best of intentions, your MTA is almost certainly going to wind up trying to deliver this email to the aa subdomain wildcard A record instead of rejecting it as 'no such domain exists'.

(If the target of the A record doesn't respond to SMTP connections, your MTA will be retrying it for some time (especially if you've configured high timeouts for in-university addresses).)

What SSH keys in your .ssh/config will be offered to servers

Recently I've become aware that some things about specifying keys in .ssh/config don't work quite the way I absently thought they did. Let's start with a simple situation:

Host *
  # encrypted & usually loaded into
  # ssh-agent
  IdentityFile /u/cks/.ssh/ids/key-ed2

Host github.com
  IdentitiesOnly
  IdentityFile /u/cks/.ssh/ids/github

When I set up my .ssh/config, this configuration looked like it would offer Github only my Github specific key. It doesn't; it offers key-ed2 as well (in fact it offers key-ed2 first). What's happening is that IdentityFile is cumulative, and so every IdentityFile in every matching Host stanza is another key that will be offered. When you connect to github.com, both the 'Host *' and the 'Host github.com' stanzas match, so the IdentityFile directives from both are used.

(Where this matters is that servers have a limit on how many keys you can offer them before they drop your connection. This can make it important to send them the right key first or at least very early.)

This can make IdentitiesOnly into an essentially meaningless directive. If the only key we ever load into our ssh-agent is the key-ed2 generic key, that's what happens here; it's always a candidate IdentityFile key so it will never be excluded. IdentitiesOnly only matters if you load extra keys from outside your .ssh/config (or at least from outside a 'Host *').

If ssh-agent is not running or does not have keys loaded, you can get around this by reordering your .ssh/config. Without ssh-agent, keys are offered to servers in the order of IdentityFile directives. If the 'Host github.com' stanza is before 'Host *', the Github specific key will be offered first (and then accepted). Unfortunately this doesn't work if you have keys loaded into ssh-agent, as keys from ssh-agent are always tried first before keys from .ssh/config (in the order they were loaded into ssh-agent, not the order in .ssh/config). This is the case even with IdentitiesOnly specified.

The rather awkward and not very scalable hack fix for this is to specifically exclude some destinations from the generic 'Host *' stanza:

Host * !github.com !elsewhere.com ...
  ...

As long as you specify IdentitiesOnly in the host-specific Host stanza, things work right (and the order in .ssh/config doesn't matter). This is clearly not scalable if you have very many of these hosts; every time you add a new one, you need to remember to add it to the 'Host * ...' exclusion as well or things will go subtly wrong. And if you have truly generic parameters other than IdentityFile, you'll need a separate 'Host *' stanza for them.

The other workaround is to only use your general keys through ssh-agent and strip them out of .ssh/config (here we'd delete the IdentityFile line for key-ed2). This costs you the ability to use them with ssh when ssh-agent is not running (or doesn't have them loaded), but it makes IdentitiesOnly do what we want it to.

As far as I know there's no other way around this, although I'd love to be wrong. The whole situation is kind of irritating and I wish there was a version of IdentitiesOnly that meant 'only identities from this specific Host/Match stanza'. It's also irritating that such an important issue as key choice and key ordering is not discussed explicitly in the OpenSSH client documentation.

(Of course this also means that OpenSSH is free to change any or all of this on you in future versions, because they never made any official promises about how it worked. It's almost all 'undocumented behavior' and we're on our own.)

When SSH needs you to decrypt your keys

Here's something nice but vaguely surprising that I just learned about how SSH handles encrypted keys. Start with the following in your .ssh/config:

Host *
  # encrypted key:
  IdentityFile /u/cks/.ssh/ids/key-ed2

Here's the question: suppose that you're not running ssh-agent and you try to ssh off to a host where you have an account that doesn't use this SSH key. Do you get prompted for the passphrase for your encrypted key? After all, your ssh client is going to try to use it to authenticate you, since it's your default key.

My naive expectation was that you would, but it turns out that I'm wrong. As you can see with 'ssh -v' (although it's not quite explicit), the SSH protocol is that the client first asks the server 'can I authenticate with this public key?' before proceeding to actually do so. When you have what I've casually called an encrypted SSH key, only the private key is actually encrypted; the public key is stored in the clear and your SSH client can send it off to the server without having to ask you for a passphrase. If the server declines the public key, that's it. Only if the server tells ssh 'sure, now prove you know the private key' does ssh need you to decrypt it and thus asks you for the key's passphrase.

In short, ssh only needs to know your key passphrase if it's about to authenticate you with it.

This has two consequences. The first is that if you get prompted to decrypt a SSH key, that's a good sign; you're about to be accepted if you can do so (well, normally, unless the server is playing funny games). This is unlike plain password prompts, where the server will demand a password even if you have no access and the remote login doesn't even exist.

The second is that it is relatively harmless to have additional encrypted SSH keys set up in your .ssh/config and offered to remote hosts. Specifically, you won't have your ssh commands interrupted to provide passphrases only to find out that you did it for nothing.

(Until I found this out just now it was a concern of mine, since I have a mix of encrypted general keys and narrow purpose unencrypted keys in .ssh/config and sometimes try things that ssh off to 'narrow purpose' destinations when I don't have ssh-agent up and active.)