There are a fair number of guides out on the Internet for how to set up a Yubikey that holds a SSH key for you, like this one. For me, the drawback of these is that they're all focused on doing this through GPG, a full set of PGP keys, and gpg-agent. I don't want any of that. I have no interest in PGP, I'd rather not switch away from ssh-agent to gpg-agent, and I definitely don't want to get a whole set of PGP keys that I have to manage and worry about. I just want the Yubikey to hold a SSH key or two.

Fortunately this turns out to be quite possible and not all that complicated. I wound up finding and using two main references for this, Wikimedia's Yubikey-SSH documentation and then Thomas Habets' Yubikey 4 for SSH with physical presence proof. Also useful is Yubico's official documentation on this. I'm doing things somewhat differently than all of these, and I'm going to go through why I'm making the choices I am.

(I've done all of this on Fedora 24 with a collection of packages installed. You need the Yubico tools and OpenSC; I believe both of those are widely available for at least various Linux flavours. FreeBSD appears to have the necessary Yubico PIV tool in their ports, presumably along with the software you need to talk to the actual Yubikey.)

The first step is to change the default Yubikey PIN, PUK, and management key. You won't be using the PUK and management key very much so you might as well randomly generate them, as Wikimedia advises, but you'll be using the PIN reasonably frequently so you should come up with an 8-character alphanumeric password that you can remember.

# In a shell, I did:
key=$(dd if=/dev/urandom bs=1 count=24 2>/dev/null | hexdump -v -e '/1 "%02X"')
puk=$(dd if=/dev/urandom bs=1 count=6 2>/dev/null | hexdump -v -e '/1 "%u"'|cut -c1-8)
pin=[come up with one]
# Record all three values somewhere

yubico-piv-tool -a set-mgm-key -n $key
yubico-piv-tool -a change-pin -P 123456 -N $pin
yubico-piv-tool -a change-puk -P 12345678 -N $puk

Changing the management key is probably not absolutely required, because I don't think an attacker can use knowledge of the management key to compromise things. Even increasing the retry counters requires more than just the management key. I may wind up resetting my Yubikey's management key back to the default value for convenience.

(We don't need to do anything with ykpersonalize, because current Yubikeys come from the factory with all their operating modes already turned on.)

Next we'll create two SSH keys, one ordinary one and one that requires you to touch the Yubikey 4's button to approve every use. Both will require an initial PIN entry in order to use them; you'll normally do this when you load them into your ssh-agent.

SSH keypair creation goes like this:

• tell the Yubikey to generate the special touch-always-required key.

  yubico-piv-tool -k $key -a generate --pin-policy=once --touch-policy=always -s 9a -o public.pem
  

  Note that the PIN and touch policies can only be set when the key is generated. If you get them wrong, you get to clear out the slot and start all over again. The default key type on the Yubikey 4 is 2048-bit RSA keys, and I decided that this is good enough for me for SSH purposes.

  A Yubikey 4 has four slots that we can use for SSH keys; these are the PIV standard slots 9a, 9c, 9d, and 9e. In theory the slots have special standardized meanings, but in practice we can mostly ignore that. I chose slot 9a here because that's what the Wikipedia example uses.

  (A Yubikey 4 also has a bunch of additional slots that we can set keys and certificates in, those being 82 through 95. However I've been completely unable to get the Fedora OpenSSH and PKCS#11 infrastructure to interact with them. It would be nice to be able to use these slots for SSH keys and leave the standard slots for their official purposes, but it's not possible right now.)

• Use our new key to make a self-signed certificate. Because we told the Yubikey to require touch authentication when we use the key, you have to touch the Yubikey during the self-signing process to approve it.

  yubico-piv-tool -a verify-pin -P $pin -a selfsign-certificate -s 9a -S '/CN=touch SSH key/' --valid-days=1000 -i public.pem -o cert.pem
  

  I don't know if the Yubikey does anything special once the self-signed certificate expires, but I didn't feel like finding out any time soon. SSH keypair rollover is kind of a pain in the rear at the best of times.

  (We don't really need a self-signed certificate, since we only care about the keypair. But apparently making a certificate is required in order to make the public key fully usable for PKCS#11 and OpenSSH stuff.)

• Load our now-generated self-signed certificate back into the Yubikey.

  yubico-piv-tool -k $key -a import-certificate -s 9a -i cert.pem
  

• Finally, we need to get the SSH public key in its normal form and in the process verify that OpenSSH can talk to the Yubikey. The shared library path here is for 64-bit Fedora 24.

  ssh-keygen -D /usr/lib64/opensc-pkcs11.so -e
  

  This will spit out a 'ssh-rsa ...' line that is the public key in the usual format, suitable for adding to authorized_keys and so on.

  (Also, yes, configuring how to do PKCS#11 things by specifying a shared library is, well, it's something.)

The process for creating and setting up our more ordinary key is almost the same thing. We'll set a different touch policy and we'll extract the SSH public key from the public.pem file instead of using ssh-keygen -e, because ssh-keygen gives you no sign of which key is which once you have more than one key on the Yubikey. We'll use slot 9c for this second key. You could probably use any of the other three slots, but 9c is the slot I happened to have used to test all of this so I know it works.

yubico-piv-tool -k $key -a generate --pin-policy=once --touch-policy=never -s 9c -o public.pem
yubico-piv-tool -a verify-pin -P $pin -a selfsign-certificate -s 9c -S '/CN=basic SSH key/' --valid-days=1000 -i public.pem -o cert.pem
yubico-piv-tool -k $key -a import-certificate -s 9c -i cert.pem

ssh-keygen -i -m PKCS8 -f public.pem

Note that you absolutely do not want to omit the --pin-policy bit here. Otherwise you'll inherit the default PIN policy for this slot and things will wind up going terribly wrong when you try to use this key through ssh-agent.

The ssh-keygen invocation here came from this Stackexchange answer, which also has what you need to extract this information from a full certificate. This is a useful thing to know, because you can retrieve specific certificates from the Yubikey with eg 'yubico-piv-tool -a read-certificate -s SLOT', and you can see slot information with 'yubico-piv-tool -a status' (this includes the CN data we set up above, so it's useful to make it distinct).

With all of this set up, you can now add your Yubikey keys to ssh-agent with:

ssh-add -s /usr/lib64/opensc-pkcs11.so

You'll be prompted for your PIN. After it's accepted, you can use the basic Yubikey SSH key just as you would any other SSH key loaded into ssh-agent. The touch-required key is also used normally, except that you have to remember to touch the Yubikey while it's flashing to get your attention (fortunately the default timeout is rather long).

In an ideal world, everything would now be smooth sailing with ssh-agent. Unfortunately this is not an ideal world. The first problem is that you currently have to remove and re-add the PKCS#11 SSH agent stuff every time you remove and reinsert the Yubikey or purge your ssh-agent keys. More significantly, various other things can also break ssh-agent's connection to the Yubikey, forcing you to go through the same thing. One of these things is using yubico-piv-tool to do anything with the Yubikey, even getting its status. So if you do a SSH thing and it reports:

sign_and_send_pubkey: signing failed: agent refused operation

What this means is 'remove and re-add the PKCS#11 stuff again'. Some of the time, doing a SSH operation that requires your PIN such as:

ssh -I /usr/lib64/opensc-pkcs11.so <somewhere that needs it>

will reset things without the whole rigmarole.

You don't have to use the Yubikey keys through ssh-agent, of course; you can use them directly with either ssh -I /usr/lib64/opensc-pkcs11.so or by setting PKCS11Provider /usr/lib64/opensc-pkcs11.so in your .ssh/config (perhaps only for specific hosts or domains). However the drawback of this is that you'll be challenged for your Yubikey PIN every time you use a Yubikey-hosted SSH key (this happens regardless of what the setting of --pin-policy is). Using an agent means that you're only challenged once every so often. Of course in some circumstances, being challenged for your PIN on every use may be a feature.

(I have a theory about what's going on and going wrong in OpenSC, but it's for another entry. Ssh-agent has its own bugs here too, and it's possible that using gpg-agent instead would make things significantly nicer here. I have no personal experience with using gpg-agent as a SSH agent and not much interest in experimenting right now.)

While I haven't tested more than two SSH keys, I believe you could fill up all four slots with four different SSH keys just in case you wanted to segment things that much. Note that in general there's no simple way to tell which specific SSH key you're being requested to authorize; all keys share the same PIN and if you have more than one key set to require touch, you can't tell which key you're doing touch-to-approve for.

(Also, as far as I know the PKCS#11 stuff will make all keys available whenever it's used, including for ssh-agent. You can control which keys will be offered to what hosts by using IdentitiesOnly, but that's a limit imposed purely by the SSH client itself. If you absolutely want to strongly control use of certain keys while others are a bit more casual, you probably need multiple Yubikeys.)

Sidebar: working with PKCS#11 keys and IdentitiesOnly

The ssh_config manpage is very specific here: if you set IdentitiesOnly, keys from ssh-agent and even keys that come from an explicit PKCS11Provider directive will be ignored unless you have an IdentityFile directive for them. Which normally you can't have, because the Yubikey won't give you the private key. Fortunately there is a way around this; you can use IdentityFile with only the public key file. This is a rare case where doing this makes perfect sense and is the only way to get what you want if you want to combine Yubikey-hosted keys with selective identity offering.