The question of how to do non-annoying multi-factor authentication for SSH

April 26, 2021

Suppose, hypothetically, that you have access to a general multi-factor authentication (MFA) system such as Duo (with the choice of MFA system not being under your control), and that you would like to use this for secure SSH logins to your collection of (Ubuntu) servers. This is generally easy by itself, with pretty much any MFA system having a PAM module that adds a second factor challenge to your regular SSH authentication. Unfortunately the result of a straightforward MFA integration with SSH logins is going to be quite annoying for some people to use, because every time they log in to any machine they will have to pass an MFA challenge as well as their regular login authentication. If you only log in to a few machines every so often, this is okay. If you're frequently logging in and out of multiple machines, you're going to be irritated.

(One consequence of this is that it encourages everyone to stay logged in to all of your machines all of the time. This is not necessarily what you want for good security.)

The general industry solution for this seems to be short term SSH user certificates that are issued through some MFA/SSO gateway; Smallstep has a writeup of this with links to other people who have done it. I have a number of questions about how well this works on Windows and how easy it is to teach people to use it (and to troubleshoot problems), plus you also have to run your own MFA/SSO gateway and signing infrastructure (which is critical both for security and for people being able to log in to any of your systems).

(I'm going to assume that Windows 10 has a decent SSH agent story these days. The SSH user certificate stories usually load your newly issued certificate into the agent, although that leaves me with other questions.)

The general simple solution seems to be putting the MFA somewhere other than SSH logins; common approaches are MFA VPNs and MFA remote desktop connections. You still require (single factor) authentication on SSH logins as usual, but you only allow them from MFA-protected sources instead of from anywhere (either the general Internet for people like us or an organizational intranet). This solution also lets you have MFA for SSH logins from outside a trusted area without forcing MFA for all SSH logins, which otherwise gets complicated with the MFA PAM module solution. The remote desktop solution has the drawback that people need a remote desktop at work to connect to.

The natural Unix and SSH expert solution is a variant of this. You have an MFA protected SSH jumphost and people establish a shared connection to it that they authenticate and let sit. With an existing connection master, they can run ssh in other windows and sessions and it will piggy-back on the already authenticated master connection to the jumphost, then pass onward to perform regular SSH authentication to the hosts past the jumphost. Your other hosts are not accessible via SSH except from the jumphost or the internal network. I'm not sure how well connection sharing works with current Windows SSH clients, and in general it's subject to issues if the single TCP connection has problems or is ever broken.

A less nice but easier to explain version of this is that people log in to the jumphost and start screen or tmux sessions there that they use to log in to whatever machines they want. This version is resilient against the person to jumphost connection being broken, but has the drawback that people who reasonably want to log in to different machines in different windows are faced with multiple MFA challenges.

It would be nice if there was some easy to use way that would get SSH on the clients to interact nicely with web based MFA SSO. How I'd like it to work is that if you already have a SSO authenticated browser session (which you probably do), your SSH clients transparently inherit this single signon without you having to do anything special. However, I don't think there are any such systems without additional software.

PS: In some high security environments, the answer is that every SSH login must be protected by MFA no matter what and that people will have to live with that inconvenience. This doesn't describe our environment, and if we tried to treat it that way I'm pretty certain we'd find people bypassing us in various ways.


Comments on this page:

By AnonCanuck at 2021-04-27 01:11:41:

having recently fallen down this rabbit hole here are some things I learned:

The windows 10 ssh client doesn't support ControlMaster connection sharing. Putty supports connection sharing.

The windows 10 ssh client supports ssh-signed certs. Putty does not.

Both putty and windows 10 ssh support jump hosts. Putty via plink. Windows 10 ssh client needs to be updated to the lastest release of the ssh client (which is not in the 20h2 version of windows 10).

WSL solves many ssh problems, but is not available on all versions of windows 10.

The easiest to implement 2FA ssh is the token + pam module. By far.

Smallstep implements ssh-signed certs. This is good, but you need an oauth system, and you need to manage mapping Oauth users to unix users, which can be a big admin burden.

hashicorp vault, and cyberark conjur, are building blocks to make an ssh-signed cert system, not a complete system.

RedHat IdM / FreeIPA is a pretty nice system. It works well in our environment with a bit of ssh scripting to make joining the Linux systems to the IPA domain / realm mostly painless. We have passwordless kerberos auth working on most systems. (Centos 7, 8, and Ubuntu 20.04)

What we implemented:

ssh jump host with password or kerberos auth, pubkeyauth disabled, and lots of logging.

This is almost 1.5 FA. It is a hardened system. The logging gives a good view into who is using the systems. It gives us a single point to immediately cut users off. It allows us to enforce user policies that might not be enforced on systems where we have granted sudo access to users. It is a nice simple system. IIRC to a base install we add 3 config files, (sshd_config, rsyslog.conf, sudoers,) a firewall, and run ipa-client-join.

After a lengthy review we determined that 2FA/MFA for regular users had a high administrative cost with minimal benefit. MFA looks good, but an attacker has too many other ways to abuse the system trusts (in our environment). Our effort is better spend elsewhere. We will likely implement 2FA for sudo as the effort:benefit ratio is better.

From 193.219.181.219 at 2021-04-27 07:27:51:

(I'm going to assume that Windows 10 has a decent SSH agent story these days. The SSH user certificate stories usually load your newly issued certificate into the agent, although that leaves me with other questions.)

Windows ships mostly standard OpenSSH ssh-agent, with one major difference that it has persistent storage (the Registry, encrypted using DPAPI) instead of having to be reloaded every time. I believe it supports SSH certificates.

(WSL users of course have just regular Linux ssh-agent. There are various GitHub projects which allow bridging various agent protocols, e.g. I think I've seen one that acts as fake Pageant for PuTTY to access a WSL ssh-agent, etc.)

I'm not sure how well connection sharing works with current Windows SSH clients, and in general it's subject to issues if the single TCP connection has problems or is ever broken.

It's not yet available on Windows OpenSSH, possibly because the OpenSSH mechanism is a bit more Unix-specific than merely carrying packets over a socket (which Windows could deal with); I think it actually involves passing the tty fd to the control master?

PuTTY has support for connection sharing, though (and I think I've heard rumors that OpenSSH developers kinda want to follow their method) both for the graphical client and for the CLI plink.

Hmm, if you need connection sharing just for the jumphost, then you can probably use PuTTY's plink as the ProxyCommand for your Windows OpenSSH connections...

It would be nice if there was some easy to use way that would get SSH on the clients to interact nicely with web based MFA SSO. How I'd like it to work is that if you already have a SSO authenticated browser session (which you probably do), your SSH clients transparently inherit this single signon without you having to do anything special. However, I don't think there are any such systems without additional software.

I've seen projects which tried, but unfortunately it involves digging deep into the browser's cookie storage (and knowing which browser to dig into, Firefox, Chrome, snap'd Firefox, flatpak'd Chrome...), which not only has the problem of dealing with undocumented likely-to-change data formats, but also is exactly what malware would do.

I would say Kerberos is close, though it works in the opposite direction: getting Kerberos credentials outside the browser allows them to be inherited by the browser and used via mod_auth_gssapi or similar. Unfortunately it's extremely cumbersome to set up on Windows if the machine isn't already part of Active Directory (which home PCs won't be...)

---

I think Duo and similar MFA systems are really cumbersome; maybe Smallstep's tools are better but I haven't looked into those yet. So far I generally prefer hardware-backed keys (smartcards, TPM devices, etc.).

The hardware PIN code acts as the 2nd factor, but you can also configure recent OpenSSH server versions to require password and public-key authentication to achieve MFA as well. Assuming you have some way to verify that the key is stored in hardware and not ~/.ssh/id_rsa, of course...

For example, I have a Yubikey NEO and I'm using its PIV smartcard mode to hold a self-signed X.509 certificate that acts as a hardware SSH key. Windows has built-in drivers for PIV cards, so one can just carry around PuTTY-CAC on a USB stick (it's a maintained patch that adds CAPI/CNG support to PuTTY) if one often needs to log in from not-quite-trusted locations.

For Windows OpenSSH, the situation is more annoying, as it only speaks PKCS#11 and there's no decent PKCS#11-to-CAPI bridge; fortunately Yubikey themselves provide a PKCS#11 module that directly accesses the smartcard, so I can use ssh -I libykcs11.dll to connect.

(Windows OpenSSH may also support the "sk-" FIDO-backed key format that OpenSSH has recently added. I haven't tried this type yet at all.)

As another example, my personal server that I'm hosting on university's LAN and I've ended up using as a jumphost has a TPM installed, again storing an SSH key in hardware. It's probably not very good that I'm using a personal box for this purpose, but at least the key can't be used without the PIN and it can't be stolen.

By Carl at 2021-04-27 09:51:00:

Sounds like Tailscale might help but it’s a big change to your network topology.

By cks at 2021-04-27 10:55:33:

Tailscale is nifty but there's almost no chance we could get all of the people who SSH in to our systems to switch over to it (assuming it's even possible on their clients), and I'm not sure it would satisfy MFA authentication requirements without a fair amount of extra tooling (although I haven't looked at its features there).

The solution my university has arrived at, for better or worse, is to pawn this off on the VPN. Anything SSH-able (like the HPC cluster), along with some other similarly sensitive things (like edit access to the website CMS), is restricted to access from the on-campus network. The VPN is configured to use 2FA, so you have to do that first before accessing anything else. On the plus side, once you're on the VPN, you can SSH to 100 machines if you want, with no additional 2FA challenges.

The MFA/SSO gateway things all strike me as a case of "those who didn't understand Kerberos are condemned to reinvent it poorly".

It's a shame that Kerberos hasn't kept up-to-date with advances in encryption technology because the workflow was very neat.

IE on Windows (still, I think?) supports providing NTLM sessions (which are now Kerberos?) over HTTP auth, using their own mechanism. I seem to remember that you need to explicitly enable it for specific URL domains tho'.

Anyway, OpenSSH has good GSSAPI support and you can still use password or private key auth so Kerberos can be deployed gradually if you can overcome the risks of running an Internet facing KDC.

Written on 26 April 2021.
« Maybe a local, on-machine caching DNS resolver should be standard (for us)
Firefox's slow takeover of the address bar's space »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Apr 26 22:57:04 2021
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.