Chris's Wiki :: blog/sysadmin/SSHConfigIdentities

What SSH keys in your .ssh/config will be offered to servers

July 10, 2015

Recently I've become aware that some things about specifying keys in .ssh/config don't work quite the way I absently thought they did. Let's start with a simple situation:

Host *
  # encrypted & usually loaded into
  # ssh-agent
  IdentityFile /u/cks/.ssh/ids/key-ed2

Host github.com
  IdentitiesOnly
  IdentityFile /u/cks/.ssh/ids/github

When I set up my .ssh/config, this configuration looked like it would offer Github only my Github specific key. It doesn't; it offers key-ed2 as well (in fact it offers key-ed2 first). What's happening is that IdentityFile is cumulative, and so every IdentityFile in every matching Host stanza is another key that will be offered. When you connect to github.com, both the 'Host *' and the 'Host github.com' stanzas match, so the IdentityFile directives from both are used.

(Where this matters is that servers have a limit on how many keys you can offer them before they drop your connection. This can make it important to send them the right key first or at least very early.)

This can make IdentitiesOnly into an essentially meaningless directive. If the only key we ever load into our ssh-agent is the key-ed2 generic key, that's what happens here; it's always a candidate IdentityFile key so it will never be excluded. IdentitiesOnly only matters if you load extra keys from outside your .ssh/config (or at least from outside a 'Host *').

If ssh-agent is not running or does not have keys loaded, you can get around this by reordering your .ssh/config. Without ssh-agent, keys are offered to servers in the order of IdentityFile directives. If the 'Host github.com' stanza is before 'Host *', the Github specific key will be offered first (and then accepted). Unfortunately this doesn't work if you have keys loaded into ssh-agent, as keys from ssh-agent are always tried first before keys from .ssh/config (in the order they were loaded into ssh-agent, not the order in .ssh/config). This is the case even with IdentitiesOnly specified.

The rather awkward and not very scalable hack fix for this is to specifically exclude some destinations from the generic 'Host *' stanza:

Host * !github.com !elsewhere.com ...
  ...

As long as you specify IdentitiesOnly in the host-specific Host stanza, things work right (and the order in .ssh/config doesn't matter). This is clearly not scalable if you have very many of these hosts; every time you add a new one, you need to remember to add it to the 'Host * ...' exclusion as well or things will go subtly wrong. And if you have truly generic parameters other than IdentityFile, you'll need a separate 'Host *' stanza for them.

The other workaround is to only use your general keys through ssh-agent and strip them out of .ssh/config (here we'd delete the IdentityFile line for key-ed2). This costs you the ability to use them with ssh when ssh-agent is not running (or doesn't have them loaded), but it makes IdentitiesOnly do what we want it to.

As far as I know there's no other way around this, although I'd love to be wrong. The whole situation is kind of irritating and I wish there was a version of IdentitiesOnly that meant 'only identities from this specific Host/Match stanza'. It's also irritating that such an important issue as key choice and key ordering is not discussed explicitly in the OpenSSH client documentation.

(Of course this also means that OpenSSH is free to change any or all of this on you in future versions, because they never made any official promises about how it worked. It's almost all 'undocumented behavior' and we're on our own.)

Comments on this page:

By Ewen McNeill at 2015-07-10 05:15:53:

ISTR discovering that the key specified on the command line (-i) will be used first (or at least early enough to work around some host's max-auth-attempts limitations) -- although obviously that only helps when you have the key right there, not via a chain of agent forwarding.

If you are not running ssh agent (and/or don't have the keys loaded into the agent) then I'm pretty sure that ssh will also try the default ssh key names (~/.ssh/id_rsa, ~/.ssh/id_dsa) although possibly towards the end (I've never checked the precise order, as it's never mattered to me). So if you have general use keys and don't always use ssh-agent, then those could be good choices of names for the general use keys (leaving other names for specific use keys). My general use keys are called exactly that.

Personally I almost always have ssh-agent running, and use agent proxy forwarding to every host I trust (eg, I run or know/trust the people running it), and have done for 15+ years. I also mostly don't use host-specific keys, aside from special task purposes (eg, "always run command FOO on remote host). So the main key ordering symptom I see is running out of auth attempts (for which a one-off work around is to overwrite the ssh agent environment variables for that one connection attempt -- and try to avoid loading more than two keys into the agent as that usually leaves at least one other auth attempt for "something else").

Ewen

PS: In case you're not aware I'm pretty sure you can ssh-add on a remote host into an agent on your local machine, providing ssh agent forwarding is active. And that key will stay there until deleted or the agent is restarted. So, eg, adding a work key into your home agent is a viable option, give or take security policies.

By cks at 2015-07-10 14:26:46:

In some experimentation, it looks like the default .ssh/id_* identities are only tried if your .ssh/config doesn't itself specify any IdentityFile keys for whatever host you're connecting to (either through a generic 'Host *' directive or via a host specific one).

Based on my testing, the order that SSH tries keys is:

keys from ssh-agent, with the ones offered possibly restricted by IdentitiesOnly

the key from a -i argument, if any

key(s) from .ssh/config OR the default .ssh/id_* keys if there are no keys from .ssh/config.

If you have usable ssh-agent keys at all there appears to be no way to stop them from being offered first (and in the order they were added to ssh-agent, regardless of the order that they would be offered without ssh-agent).

So if you're offering too many keys and need to order them so you can pass authentication, -i only helps if you have too many keys that match in .ssh/config but those keys are not in ssh-agent. I'm not sure I can come up with a scenario for this that isn't better handled by ordering Host and Match blocks in .ssh/config so that the most specific one matches first, insuring that the right key is the first one tried.

In short, this stuff is complicated and badly documented and doesn't really work great. I suppose that the core problem is that all acceptable keys from ssh-agent are tried first.

By Ewen McNeill at 2015-07-10 18:07:11:

Thanks for investigating the order in detail. As you say, it's not well documented and seems somewhat arbitrary. I guess the rationale for trying the ssh agent keys first is that if they're accepted they "just work" (no user intervention required). But it seems unfortunate since one use for specifying a key on the command line is to get a specific pre-recorded action from ~/.ssh/authorized_keys; apparently anything wanting that behaviour should explicitly unset SSH_AUTH_SOCK.

Ewen

By Evaryont at 2015-07-10 19:19:12:

I seem to remember a fork of ssh-agent that would only offer specific keys based on the hostname of the connection, which is passed to the agent. This way you can create a configuration file matching keys to hosts and then add every key to the agent without worry.

Not sure if the work is worth it for you in this case, though.

Written on 10 July 2015.

«	When SSH needs you to decrypt your keys

OpenBSD and the impact of culture