The high costs of true security paranoia in the face of compromises
A while back I wrote about paranoia versus optimism in dealing with security compromises. One of the factors driving optimism is the sheer cost of doing otherwise in a decent-sized environment such as, well, ours.
Suppose that we take the paranoid approach; if we detect that an account has been compromised on a machine, we make the assumption that there is an invisible rootkit. So we take the machine down, build a replacement from scratch, and do forensics on the potentially compromised machine to detect how correct our paranoia was. Better safe than sorry, right? And we can (or should) be able to build new machines relatively fast.
Except that we have a security domain, not just isolated machines. If you're going to be paranoid enough to assume an undetected root compromise, you have to also assume that the intruder has subverted your login logs and gained undetected access to all of the other machines in the same security domain, and then of course used their undetected root compromise on them in turn.
So under the paranoid assumptions, an access compromise of any user accessible machine implies a compromise of every user accessible machine, and a non-root compromise implies a root compromise. The net result is that a compromised user account means a total compromise of our entire environment and thus a total shutdown and rebuild of all machines.
(By the way, even this level of paranoia isn't sufficient. How do you
know that your intruder hasn't left themselves backdoors into people's
accounts by one of the many, many means of doing so? Better audit
everyone's SSH keys, everyone's crontabs and
Firefox plugins directories, and, well, you'll be there for a while.)
This is clearly absurd and unworkable (at least in most environments). Thus, we need to back off the paranoia somewhere. At some point you have to have optimism; the only question is where to draw the line, and this question is both a technical one and an economic one.
(Note that even having magical complete and trustworthy login logs doesn't help very much in an active environment, especially if you have not detected the compromise immediately. Since the attacker has root, all logins from that machine to others are under their control, regardless of what user they were done as. In most environments this rapidly snowballs into a mesh of untrustable logins that flood-fill to all of your user accessible machines in that security domain.)