Some good practices for handling OpenSSH keypairs

January 30, 2016

It all started with Ryan Zezeski's question on Twitter:

Twitter friends: I want to better manage my SSH keys. E.g. different pairs for different things. Looking for good resources. Links please.

I have opinions on this (of course) but it turns out that I've never actually written them all down for various reasons, including that some of them feel obvious to me by now. So this is my shot at writing up what I see as good practices for OpenSSH keypairs. This is definitely not the absolutely best and most secure practice for various reasons, but I consider it a good starting point (but see my general cautions about this).

There are some basic and essentially universal things to start with. Use multiple SSH keypairs, with at least different keypairs for different services; there is absolutely no reason that your Github keypair should be the keypair that you use for logging in places, and often you should have different keypairs for logging in to different places. The fundamental mechanic for doing this is a .ssh/config with IdentityFile directives inside Host stanzas; here is a simple example.

(My personal preference is to have different keypairs for each separate machine I'll ssh from, but this could get hard to manage in a hurry if you need a lot of keypairs to start with. Consider doing this only for keypairs that give you broad access or access to relatively dangerous things.)

Encrypt all of your keys. Exactly what password(s) you should use are a tradeoff between security and convenience, but simply encrypting all keys stops or slows down many attacks. For instance, the recent OpenSSH issue would only have exposed (some of) your encrypted keys, which are hopefully relatively hard to crack.

Whenever possible, restrict where your keys are accepted from. This is a straightforward way to limit the damage of a key compromise at the cost of some potential inconvenience if you suddenly need to access systems from an abnormal (network) location. In addition, if you have some unencrypted keys because you need some automated or unattended scripts, consider restricting what these keys can do on the server by using a 'command=' setting in their .ssh/authorized_keys line; an example where we do this is here (see also this). You probably also want to set various no-* options, especially disabling port forwarding.

At this point we're out of truly universal things, as the path splits depending on whether you will access all of your keys via ssh-agent or whether at least some of them will be handled only by ssh (with passphrase challenges every time you use them). There is no single right answer (and covering the issues needs another entry), but for now I'll assume that you'll access all keys via ssh-agent. In this case you'll definitely want to read the discussion of what identities get offered to a remote server and use IdentitiesOnly to limit this.

If you need to ssh to hosts that are only reachable via intermediate hosts, do not forward ssh-agent to the intermediate hosts. Instead, use ProxyCommand to reach through the intermediates. This is sometimes called SSH jump hosts and there are plenty of guides on how to do it. Note that modern versions of OpenSSH have a -W argument for ssh that makes this easy to set up (you no longer need things like netcat on the jumphost).

(There are some cases that need ssh agent forwarding, but plain 'I have to go through A to get to B' is not one of them.)

With lots of keys loaded, your ssh-agent is an extremely large basket of eggs. There are several things you can do here to reduce the potential damage of an attacker gaining access to its full authentication power, although all of them come with convenience tradeoffs:

  • Set infrequently used or dangerous keys so that you'll have to confirm it before they can be used, by loading them with ssh-add's -c 'have ssh-agent ask for confirmation' argument.

  • Treat some keys basically like temporary sudo privileges by loading them into ssh-agent with a timeout via ssh-add's -t argument. This will force you to reload the key periodically, much as you have to sudo and then re-sudo periodically.

  • Arrange to wipe all keys from ssh-agent when you suspend, screenlock, or otherwise clearly leave your machine; my setup for this is covered here.

    (This is good practice in general, but it becomes really important when access to ssh-agent is basically the keys to all the kingdoms.)

You'll probably want to script some of these things to make them more convenient; you might have an 'add-key HOST' command or the like that runs ssh-add on the right key with the right -c or -t parameters. Such scripts will make your life a lot easier and thus make you less likely to throw up your hands and add everything to ssh-agent in permanent, unrestricted form.

(Also, check your ssh_config manpage to see if you have support for AddKeysToAgent. This can be used to create various sorts of convenient 'add to ssh-agent on first use' setups. This is not yet in any released version as far as I know but will probably be in OpenSSH 7.2.)

PS: You probably also want to set HashKnownHosts to yes. I feel conflicted about this, but it's hard to argue that it doesn't increase security and most people won't have my annoyances with it.

PPS: My personal views on SSH key types are that you should use ED25519 keys when possible and otherwise RSA keys (I use 4096 bits just because). Avoid DSA and ECDSA keys; the only time you should generate one is if you have to connect to a device that only supports DSA (and then the key should be specific to the device).


Comments on this page:

""" There are some basic and essentially universal things to start with. Use multiple SSH keypairs, with at least different keypairs for different services; there is absolutely no reason that your Github keypair should be the keypair that you use for logging in places, and often you should have different keypairs for logging in to different places." """

Er, why? The site you're connecting to never sees your private key, just your public one. A keypair per device makes sense (I archive private keys in pass) since a device could become compromised. An ssh keypair identifies an account on a machine. But your public key is fine to be, well, public. The public key portion doesn't identify anything.

I do think it would be nice to have a central place to track all my authorized_keys files so if a device was compromised or I regenerated a device's keypair I could have some sort of script run off and update all those files. Though the problem then is that things like github and bitbucket don't use authorized_keys files so it's a little complicated.

By Alex at 2016-01-30 07:32:58:

Also NIST just released their final guide about this:

http://nvlpubs.nist.gov/nistpubs/ir/2015/NIST.IR.7966.pdf

And mwlucas presentation is well worth watching:

https://m.youtube.com/watch?v=fnkG9_jy2qc

 -Alex
By mtk@acm.org at 2016-01-30 09:08:35:

what is the problem with ssh-agent forwarding?

Kevin:

The public key portion doesn't identify anything.

Yes it does. It identifies you. It can be used to find out that the person who has GitHub account X also has the system account Y on machine Z. This is relevant in both privacy and opsec terms (e.g. that simple fact right there might be enough information to launch a social engineering attack).

By cks at 2016-01-30 16:46:41:

Kevin Lyda: CVE-2016-0777 shows a large reason why you should have multiple SSH keys. There are circumstances where the remote end can compromise a key or you can lose a key, and when this happens you want that key to have as little access as possible.

(Also, while I believe that SSH keypairs are immune from man in the middle attacks I don't know enough to be absolutely sure of that. If there is a MITM attack possible here under some circumstances, using the same public key for everything is a great help for the attacker.)

mtk: There are at least two problems with ssh-agent forwarding. The first is that it gives an attacker who has compromised your account on the remote machine the ability to authenticate with all keys that your ssh-agent holds, protected only by whatever confirmation requirements you've put on some keys. The second is that it gives such an attacker a direct path to ssh-agent which they can use to mount direct attacks on ssh-agent itself. Ideally ssh-agent doesn't have any buffer overflow vulnerabilities or the like, but then ideally ssh itself wouldn't have had CVE-2016-0777 either.

(OpenSSH itself has no support for limiting what ssh-agent keys are available remotely. Although people have written programs for this, they have some drawbacks because they have to work outside OpenSSH. And they still retain the 'more or less direct access to ssh-agent' issue.)

I wonder if it would be possible to leverage ProxyCommand to dynamically load and unload keys from the ssh-agent. I.e.

Host *
     ProxyCommand sshProxyWrapper.sh %h %p

Where the sshProxyWrapper.sh script does the following:

ssh-add ~/.ssh/keys/${1}
nc ${1} ${2}
ssh-add -d ~/.ssh/keys/${1}

This is all pseudo code. You may have some odd things to deal with, like the fact that you can't use STDOUT / STDERR to ask for the key pass phrase. So, you'd probably only want to do this in X where you can spawn a new window to ask for the pass phrase.

This would still expose keys loaded into the agent when agent forwarding is in use, but it would reduce the number of keys exposed.

You could also probably limit the length of time the key could be used down to a number of seconds (30 ~ 90?).

You might also be able to leverage the LocalCommand to unload the key from the ssh-agent -after- using it to successfully authentic, and re-add it with the -c option, thus requiring confirmation before it's used again (if using forwarding).

If you stop to think about it, there's quite a bit that can be done to limit key exposure. All of them require a good understanding of the work flow and deciding what you want to do where.

I have recently been experimenting with SSH certificates. The general idea is that you build a CA key ( ssh-keygen -f /etc/ssh/ca ) and lock it away on a machine you trust as a CA. (I usually use the TPM on my laptop).

Then for every new machine that you generate a key on, you copy the public key to your CA machine, and run:

ssh-keygen -s /etc/ssh/ca \
   -I "$(whoami)@$(hostname --fqdn) user key" \
   -n "$(whoami)" \
   -z "$(date +%s)" \
   sshkey.pub

This generates a sshkey-cert.pub file, which ssh will present along with your public key to servers.

Then, instead of putting in the .ssh/authorized_users file a copy of the public key, you put:

echo "cert-authority,principals=$(whoami) $(cat /etc/ssh/ca.pub)" >>~/.ssh/authorized_keys

Now, any key that is signed by the CA, and that has the principal of $(whoami) can login. Thus you can easily rotate keys for individual hosts regularly — possibly even daily, and you don't have to remember all the locations that need to have the key updated. People can, unfortunately, still track you via the CA public key.

Of course, this has just turned the problem into managing the rotation of the CA, but that's a key that is much easier to protect.

There's lots of other stuff you can do here, I've documented much of it in http://www.lorier.net/docs/ssh-ca

If you're worried about your private key (and you should be), I'd suggest using a YubiKey 4:

https://www.yubico.com/products/yubikey-hardware/yubikey4/

Caveat: I work for Yubico.

It does require a little bit more configuration initially; creating a gpg key, writing it to the YubiKey and configuring gpg-agent.

Once done, it's friction less though, especially the nano version.

Written on 30 January 2016.
« What SSH identities will be offered to a remote server and when
The tradeoffs of having ssh-agent hold all of your SSH keys »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sat Jan 30 01:54:16 2016
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.