Two factor authentication and emergency access to systems
One of the things that makes me hesitant to whole heartedly embrace two factor authentication for system administration is how to deal with emergency system access in an exceptional situation. A lot of 2FA systems seem to involve daemons and central services and other things with a lot of moving parts that are potentially breakable, especially in extreme situations like partial network failures or machine failures. If you require all of this to be working before you can get in as a sysadmin or as root, you may have serious problems in emergencies. But if you create some bypass for 2FA, you're at least weakening and perhaps defeating the protections 2FA is supposed to be giving you; an attacker just needs to compromise your back door access instead of your regular access.
Some organizations guard sufficiently sensitive information and important systems that the answer is 'in an emergency, we go to the machine room and boot from recovery media for manual access'. This does not particularly describe us, for various reasons; we really would like to be more resilient and recoverable than that (and we are towards that now, since we're not using 2FA or LDAP or the like).
It looks like some 2FA systems can be configured to use purely local resources on the machine, which at least gets you out of relying on some central server to be up and talking to the machine. Relying on the local 2FA software to be configured okay and working right is probably no worse than things like relying on the overall PAM configuration to be working; you can blow your own foot off either way and at least you can probably test all the behavior in advance of a crisis.
The other side of emergency system access is network access in situations where you yourself do not have both your 2FA and a system that's capable of using it. This won't be an issue in a place where all sysadmins are issued laptops that they carry around, but again that's not our situation. Closely related to this is the issue of indirect system access, where instead of logging in directly from your desktop to the fileserver you want to log in to another machine and copy something from it to the fileserver. Your desktop has access to your good 2FA device but the other machine doesn't, no more than it has access to your SSH keypairs.
(I suspect that the answer here is that you set the system up so you can authenticate either with a 2FA protected SSH keypair or with your password plus a 2FA code. Then your desktop uses the 2FA protected keypair and on other machines you fall back to password plus 2FA code. Maybe you have multiple password plus code setups, one using a 2FA fob and one using a SMS message to your cellphone.)