Things I want to remember during a security incident
We're having a little security incident right now. I thought about writing an entry about some details and specific thoughts, but then realized I had some more important things to write about, things that I need to keep in mind to keep grounded during our handling of the incident.
The first thing I want to remember is that hindsight is too easy. We live in a noisy world in the present, but when we look back at the past it's easy to pick out the thread of signal that we now know is there. Then you can sit there beating yourself up with the thought that you should have seen X or realized Y at the time. Our past selves were not incompetent idiots, not matter what it may look like now, and we made rational decisions at the time. Playing what-if games and blaming ourselves is both wrong and a bad idea.
The second thing I want to remember is that not all possible responses to our incident are worth doing. We could spend a steadily increasing amount of time analyzing what happened, hardening our systems, increasing our monitoring, adding this and that, and so on; all of them would increase the odds of stopping further incidents. But any and all of them will take time, time that will have to come from other work. At some point the right answer is 'more work to stop another incident is less important than what we were doing before'. However bad it may sound and feel, we'll need to simply live with the possibility of another incident happening (or there being undetected aspects to this one) and to move on.
(And then if (when) there is another incident, we don't beat ourselves up about this choice even though we can't say 'we did the best we could to prevent it'.)
I've been reading various John Allspaw writings about all of this for some time and it has done a lot to change and shape my views on all of this. But it's one thing to read all of this stuff and nod along intellectually and another thing entirely to have to try to live through it and put it into practice despite all of the inconvenient squishy human emotions running around.
(And I should read some of his stuff again, eg bits from this, so that I can do some sort of proper, useful postmortem writeup. It's probably past time that we did a real postmortem, although of course that takes time too.)