Wandering Thoughts archives

2010-12-15

Always remember that people make mistakes

One very important thing to remember when trying to design practical security systems is that people make mistakes. Always. Even under the best of circumstances and the best of intentions, sooner or later someone will do accidentally do something wrong.

If your security system breaks explosively when people makes mistakes, your system is wrong in practice. Regardless of how mathematically pure it is, you have not designed something with real security. Real security needs to cope with things going wrong and people making mistakes, because that's what actually happens.

(There are all sorts of mitigation and coping strategies, depending on what the overall design goals are for your security system.)

You cannot fix this fact. You cannot exhort users to not make mistakes; it doesn't work. You cannot threaten users to get them to not make mistakes; it doesn't work, you can't make it work, and the side effects of trying this are extremely unpleasant. You can't even make it so strongly in people's self-interest to not make mistakes that they won't make mistakes; it still doesn't work. People just make mistakes.

Perhaps you're convinced that your system and environment is an exception. If so, please consider aviation's 'controlled flight into terrain', which is the dry technical term for 'a highly trained pilot with their life on the line spaced out and flew their plane into the ground'. Pilots kill themselves (and other people) in CFIT accidents every year. This happens in basically the best situation possible; commercial pilots are highly trained, they've got pretty much the best motivation possible to not do this, and there are huge support structures and millions of dollars invested in avoiding these accidents. Given that commercial pilots still fly planes into the ground, your system is not going to do better.

PS: obviously this applies to more than just security systems. It's just that security systems are the most common place for people to appeal to shiningly perfect math and dismiss actual reality as an annoying inconvenience. By now, most other computing subfields are willing to acknowledge actual human behavior and design accordingly.

Sidebar: how many mistakes is too many

It's sensible to say that you can't cope with too many mistakes at once, although ideally you will have some modeling to assess how likely this is. Please do not make this merely some handwaving math about low percentages multiplied together; for a start, mistakes are not necessarily independent events.

tech/UsersMakeMistakes written at 01:50:39; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.