How self-signed certificates are a problem for browsers
I was a little harsh on Firefox 3's handling of self-signed SSL certificates in the last entry, and in fairness I need to say that this is a genuinely hard problem for browsers, at least from a security perspective. Let us talk about just how this is so.
First, let us start with my previous basic model of how https could work. Browsers always use SSL if possible, and they protect against straightforward man in the middle attacks by remembering the SSL certificate they saw last time, or at least how strong it was, and complaining loudly if the SSL certificate is weaker. Call this the downgrade problem, and I believe that it's pretty solved.
(What does 'solved' here mean? To me, it means that you can accept self-signed SSL certificates while giving users the security that they think they're getting.)
This still leaves what I will call the initial connection problem.
Suppose that the user types a https URL in that resolves to a site with
a self-signed certificate, and the web browser has no SSL certificate
information for the site on file; what do you do? Since the user
explicitly typed in the
https bit, they may well expect that you will
give them full SSL protection, including genuine identity verification.
(Before people bring up SSH's handling of this, let us admit that only the paranoid actually validate those SSH host key signatures that OpenSSH tells you about; everyone else just says 'yes' automatically.)
One might hope that this was an uncommon case, but consider how often people empty their browser's caches for various reasons. And privacy concerns mean that if the user selects 'forget everything', we do need to forget the remembered SSL certificate information; clever hashing can't save us.
A related risk is using other people's computers, including public computers, and connecting to https sites; here an attacker might have preloaded their man in the middle self-signed certificates as the website's certificate, so the browser would see a self-signed certificate that matched the one on file. The counter argument here is that in practice there are lots of far worse things that can be done to public machines.