The tradeoffs of having
ssh-agent hold all of your SSH keys
Once you are following the initial good practices for handling SSH keys, you have a big decision to make: will you access all of your encrypted keys via ssh-agent, or will at least some of them be handled only by ssh? I don't think that this is a slam dunk decision, so I want to write down both sides of this (and then give my views at the end).
The first and biggest thing that might keep you from using ssh-agent for everything is if you need to use SSH agent forwarding. The problem with agent forwarding is twofold. First and obviously, it gives the remote host (where you are forwarding the agent to) the ability to authenticate with all of the keys in ssh-agent, protected only by whatever confirmation requirements you've put on them. This is a lot of access to give to a potentially compromised machine. The second is that it gives the remote host a direct path to your ssh-agent process itself, a path that an attacker may use to attack ssh-agent in order to exploit, say, a buffer overflow or some other memory error.
(Ssh-agent is not supposed to have such vulnerabilities, but then ssh itself wasn't supposed to have CVE-2016-0777.)
In general, there are two advantages of using ssh-agent for everything. The first is that ssh itself never holds unencrypted private keys (and you can arrange for ssh to have no access to even the encrypted form, cf). As we saw in CVE-2016-0777, ssh itself is directly exposed to potentially hostile input from the network, giving an attacker an opportunity to exploit any bugs it has. Ssh-agent is one step removed from this, giving you better security through more isolation.
The second is that ssh-agent makes it more convenient to use encrypted
keys and therefor makes it more likely that you'll use them. Without
ssh-agent, you must decrypt an encrypted key every time you use it,
ie for every
rsync and so on. With ssh-agent,
you decrypt it once and ssh-agent holds it until it expires (if
ever). Some people are fine with constant password challenges, but
others very much aren't (me included). Encrypted keys plus ssh-agent
is clearly more secure than unencrypted keys.
The general drawback of putting all your keys into ssh-agent is that ssh-agent holds all of your keys. First, this makes it a basket with a lot of eggs; a compromise of ssh-agent might compromise all keys that are currently loaded, and they would be compromised in unencrypted form. You have to bet and hope that the ssh-agent basket is strong enough, and you might not want to bet all of your keys on that. The only mitigation here is to remove keys from ssh-agent on a regular basis and then reload them when you next need them, but this decreases the convenience of ssh-agent.
(I suppose any attacker code can also attempt to dump the memory of the ssh-agent process, since that's going to contain keys in decrypted form. But it might be easier for an attacker to get access to the ssh-agent socket and then talk to it.)
Similar to this, unless you have confirmations on key usage, you yourself can easily do potentially dangerous operations without any challenges. For example, if you have your Github key loaded, publishing something on Github is only a 'git push' away; there is no password challenge to give you a moment of pause. Put more broadly, you've given yourself all the capabilities of all of the keys you load into ssh-agent; they are there all the time, ready to go.
(You can mitigate this in various ways (cf), but you have to go out of your way to do so and it makes ssh-agent less convenient.)
My personal view is that you should hold heavily used keys in
ssh-agent for convenience but that there are potentially good reasons
for having less used or more dangerous keys kept outside of ssh-agent.
For example, if you need your Github key only rarely, there is
probably no reason to have it loaded into ssh-agent all the time
and it may be easier to never load it, just use it directly via
ssh. There is a slight increase in security exposure here, but
it's mostly theoretical.