Wandering Thoughts archives

Pragmatic issues with hash verifiers for email messages

Suppose that you have a 'sender stores message' email system, and that you protect yourself against the sender rewriting the message after it's initially sent by sending and verifying a secure hash of the message before you show the message to the user. To follow up a recent entry, here are the social and technical issues that I can see with this hash protection scheme.

First and largest, you have a social issue; users are going to react badly to 'this message is (minorly) corrupt so I will show you nothing at all'. Users want you to recover as much of their messages as you can and show them as much as you can, but doing so weakens or destroys your protection against on the fly targeted spam and phish, client exploits, and so on.

(Trying to sell this to users as message integrity checking is going to be even less successful, because I think users care even less about that.)

Next, this scheme is more complicated than it looks because you don't want to have a 'unitary' message hash, a simple single hash that covers the entire message. The problem with a single hash is that it requires you to download the entire message before you can verify the hash and show any of it to the user, which has very bad delays with large messages (or slow network connections or both). You can deal with this by hashing the message in sections, but this probably leaves you with more client attack surface exposed (especially if you make the mistake of making the sections be the various MIME parts).

Finally, this doesn't help your security issues if clients can be fingerprinted at all; it just forces the attacker to send two messages. You use the first message to find out the client fingerprint (and to verify that you have a live user), at which point you can craft your customized exploit message and send it to the user.

Sidebar: A sectional hash scheme

I think that the best sectional hash scheme would be one that simply breaks the message up into blocks of so many Kbytes each and hashes each block separately. You still need a single final hash for the message, but you can create this by hashing the list of section hashes. Then you embed the list of section hashes early in the message header, so that the client can verify pieces independently.

On digital signatures and client security issues

One of the technical problems with 'sender stores messages' schemes for email-like things is that it lets a malicious sender generate messages on the fly for up to the moment customized malware (and this gets much, much worse if the protocol helpfully includes things that you can use to fingerprint specific client software).

It is tempting to deal with this problem by making the identification for messages include a hash verifier or address, in the way that current version control systems identify files by their hashes. This would mean that while a malicious server could still try to send you different content, your mail client would detect a hash mismatch and reject the message as corrupted.

(Let us set aside a number of pragmatic problems with this idea, some of them social and some of them technical, and the fact that it would be relatively weak protection in general.)

But this has not solved the problem; it has merely reduced the attack surface. You still have to trust that your client will not have bugs in the code that runs up to the point where it rejects a message for not having the right hash (and you have to trust your client to not have hash verification bugs). Depending on the message access protocol, this may amount to a significant amount of code, some of it complex.

This is of course not a problem unique to 'sender stores messages' email systems; it is a general issue with anything that uses hash verification or other digital signatures in an attempt to reject malicious input. Hash verification can reduce your risks, but it does not magically make them all go away (and it adds new risks, like 'did the people who wrote the libraries you're using understand the proper way to verify signatures').

Git and 'rewriting history'

Every so often, someone says that git is bad because it lets you rewrite history (usually they have other reasons too). This almost invariably irritates me because it is so completely wrong at a deep level.

Git has no support for 'rewriting history' in any meaningful sense. In fact, git's design means that by its very nature you cannot rewrite history. Since a commit is identified by the hash of its contents, you cannot change a commit once it is made. Since commits identify their parents, the tree of commits, which is exactly the history, is immutable. It is quite literally impossible to alter the history of commits to, say, introduce another one in the middle, change a commit's text, or turn a series of commits into one; your altered history will produce commits with the wrong hash values and nothing will accept them.

(This is not unique to git; it is true of any VCS that uses content hashes for identifiers, and is a very useful property.)

What git does that is un-traditional for a version control system is that it lets you create new history, or more specifically it provides convenient support for doing this (you can create new history in any version control system by hand). It is in this new history, and only in this new history, that you can do things like introduce a new commit, compact a series of commit into one, or revise the text of a commit. Even this terminology is inaccurate, because you are not doing anything to the old commits that are part of the old history; instead you are making new commits in your new history that have the desired contents and relationships.

This new history doesn't overwrite the old history (it can't), but what you can do is delete the old history; at that point, the new history is the only history and you have, in some sense, 'rewritten history'. But not really, because the cracks show immediately; for example, people who've pulled the old history from your repository will not transparently migrate to using your new history.

(The ability to delete history is by no means unique to git; it is in fact a common request and feature.)

There are some version control systems that do let you truly rewrite history, where you can change a commit while preserving its identifier. Ironically, they are old and generally non-distributed.

Why I don't expect ARM-based netbooks to be a success

I saw yet another story recently on how low-power netbooks are finally going to lead to ARM-based machines successfully challenging the x86 hegemony in general purpose computers. As the story goes, netbook makers don't need much CPU power or to run Windows, but they do need very low power consumption, and so they will turn to Linux-based ARM machines.

This is a fine theory and a fine story, but I don't think it's going to work out in practice. Why is really simple to explain, so let's start with Flash. Specifically, the fact that ARM-based netbooks aren't going to have Flash.

And Flash is just an example. The general problem for people trying to make non-x86-based general purpose computers is that so very many pieces of what people expect to be present (and fast) on such machines are currently x86-only binary blobs, or if they have alternative implementations the alternatives are significantly slower. Flash is the most obvious tip of the iceberg, because I don't think you're going to sell many netbooks that can't watch Youtube videos at full speed.

(I suspect that this is especially the case as many things are adding JIT systems for additional speed; my impression is that most of those JITs primarily or exclusively target x86 code generation.)

You can sell ARM-based machines, and lots of people do. But they aren't sold as general purpose machines, they're sold as PDAs or cellphones or the like, with significantly different software requirements.

This x86 focus can be overcome, but it is probably going to take a significant development effort, which means significant amounts of time and money. For at least some pieces, such as Flash, it's also going to require you to convince commercial software companies that there is a large enough ARM-based market to make it worth porting their software.

(And since netbooks are a low-cost, low-margin product, it's hard to see the makers of ARM-based netbooks coming up with the time and the money necessary.)