Wandering Thoughts archives

There is a surprising amount of old SSL code in running MTAs

SSL has a number of old, dark corners. One of them is 'SSL2 handshake compatibility', where clients that want to be able to talk to a SSLv2 server must start their conversation out with a (special) SSLv2 ClientHello. Of course this means that a SSL v3 or TLS server that wants to be able to talk to such clients needs to accept the SSLv2 ClientHello and then upgrade the connection to SSLv3 or TLS afterwards, in a coordinated dance with the client. Now, SSLv2 has been dead for a long time now, but of course SSL libraries don't like removing features so SSLv2 client support lingered on for years and years after SSLv2 servers had probably all vanished, and along with it lingered SSL2 handshake compatibility. All of this is generally invisible in server logs, which will just reflect that SSLv3 or more likely TLS was negotiated in the end.

Except if you run a Go-based TLS server, that is, because Go doesn't support SSL2 handshake compatibility. As Filippo Valsorda notes, the browsers and browser software that supports this is now really, depressingly old and most people won't care about them anyways. Unfortunately, HTTPS is not the only TLS based protocol in the world and browsers (and browser libraries) are not the only things making TLS connections. In particular, SMTP supports TLS, which means that MTAs thus talk TLS.

I have a Go based SMTP server with TLS support and it logs TLS setup errors, so I actually have an opportunity to see how many sending MTAs try to do SSL2 handshake compatibility. It turns out that there were more than I expected, although the good news is that the number of such servers has fallen over time and that some of the really eyebrow raising parties seem to have stopped doing this.

(When I started my sinkhole SMTP server in the spring and summer of 2014, some of Yahoo's mail servers were still doing SSL2 handshake compatibility for outgoing SSL connections. That seems to have changed since then.)

I've done various casual checks on the machines trying to do this with my server, and they've turned up a wide variety of apparent sending MTAs, many of them running on Unix machines (unless someone has ported eg qmail to Windows). At least some of them are running MTA code that seems very old by now (eg), although this doesn't say anything definitive about how old their SSL libraries are.

At one level most of this is not really surprising. We've always known that there are very old servers out there and old servers mean old SSL libraries with old behavior, which this very definitely is. On another level I am surprised by just how common this seems to be. My little SMTP sinkhole is not really a hotspot of activity, yet practically the day I recently started logging these messages again I was getting some sending MTAs that were trying to do this.

Unfortunately this also means that if you care about maximal SSL support for MTAs and similar server side software, you probably want to consider still supporting SSL2 handshake compatibility. Users clearly update browsers, IMAP clients, and so on far more frequently than people update server side software like MTAs and OS SSL libraries (and OSes, for that matter).

(I also suspect that visible programs like browsers, IMAP clients, and so on are somewhat more likely to explicitly tell their SSL libraries to turn off now obsolete features like SSL2 support, SSL2 handshake compatibility, and so on. I expect that much MTA code just goes with the SSL library defaults, which means that if the library is conservative about what it deprecates the MTA goes along with that.)

Where cryptographic hashes come into TLS certificates

There has recently been a bit of commotion about whether or not to deprecate 'SHA1 certificates' for TLS, or at least how fast to do so. If you know something about TLS in general, this may sound a bit peculiar, because TLS certificates are based on public key cryptography (and public key signatures, where the CA signs your certificate and public key to say that they approve of it). SHA1 is not a public key system (that would be eg RSA), it is a well known and now old cryptographic hash.

Where SHA1 and other cryptographic hashes come into the picture is that CAs do not actually sign your certificate and public key. Directly signing lots of data with a public key is a very expensive and time-consuming thing, so what most everyone does instead is they take a cryptographic hash of the data to be 'signed' and it is that cryptographic hash that is directly signed through public key cryptography itself. Since SHA1 hashes are only 160 bits (20 bytes), you can see that this represents a very big savings over what is probably hundreds of bytes of certificate data and public key information.

(This is a savings not just when the CA generates the signature but also every time it is validated, which is on every TLS connection. Well, almost every TLS connection. On the other hand, I found a SHA1 certificate and according to the detailed information in Firefox the 'Certificate Signature Value' is 256 bytes, the same as with a SHA256 certificate. This is probably less than the full certificate data, but it's clearly a lot more than just the SHA1 hash.)

Since the CA is only actually signing the SHA1 hash, not the full certificate and public key, you can 'break' their signature if you can create a second certificate that has the same SHA1 hash. With the same hash, the CA's certificate signature applies just as much to your malicious certificate as it does to the original certificate that the CA actually signed. TLS certificate validation will happily accept your malicious certificate instead and so on.

(I think that a sufficiently determined client could detect your malicious certificate under some circumstances, by looking up the SHA1 hash in the growing certificate transparency system and seeing that the details don't match.)

So when people talk about 'SHA1 certificates' or 'SHA256 certificates', this is really a shorthand for 'certificates that are hashed and signed using SHA1'. There is nothing intrinsic about the certificate data itself that is tied to the hashing algorithm used, so I believe that a CA could do things like re-issue a certificate it originally signed using SHA1 as a SHA256 certificate.

(It's probably obvious, but I should say it for the record: the choice of hashing algorithm for certificate verification doesn't affect the security of your public and private keys, so it is (or would be) perfectly safe to get a certificate re-issued this way with the same keys. In some situations this may make certificate changeover easier.)

At a mechanical level, you generally set the hashing algorithm when you generate a CSR (in OpenSSL, via eg 'openssl req -new -sha256 ...'). However, basically all of the information in a CSR is advisory except for your public key, and I know that there are at least some CAs that basically take your CSR, extract the public key and maybe a few other pieces, and give you back a certificate that they put together themselves. These CAs are likely to ignore your choice of hashing algorithms too.

(Since no new SHA1 certificates are supposed to be issued after December 31st, pretty soon all CAs should either ignore you asking for a SHA1 hash or reject your CSR.)

A surprise about cryptographic signatures

I won't say that I know a lot about cryptography, but I know a certain amount. What this really means is that every so often I get the opportunity to be really surprised about something in cryptography. The most recent incident came about from reading Andrew Ayer's Duplicate Signature Key Selection Attack in Let's Encrypt. I had to read the article more than once before I really understood the problem, but here is the cryptographic thing that really startled me, boiled down:

A digital signature is not necessarily tied to a message.

As Ayer puts it:

Digital signatures guarantee that a message came from a particular private key. They do not guarantee that a signature came from a particular private key, [...]

By extension (and Ayer mentions this later), a signature does not uniquely identify a message; many pairs of messages and keys may result in the same signature. The specific vulnerability that Ayer exploited is that in RSA, if you have a message and a signature, you can quite easily generate a private key that produces the given signature for the message. The original Let's Encrypt protocol was vulnerable to this issue because it had you basically publish your signature of their validation message to you. Since this signature was on its own, an attacker could arrange a situation where it was also a valid signature for a different message signed with the attacker's key.

(The article is well worth reading in full, just to absorb the details of both how this works in RSA and how the specific attack worked against Let's Encrypt's original protocol.)

Until I read this article, I would not have expected this result at all. Had I been in a situation where it mattered, I wouldn't even have thought about the assumptions I was making about how a message, a signature, and a private key were connected; I probably would have just assumed that a signature was inextricably tied to both the message and the private key. Nope. Not so at all.

The direct lesson I take away from this is that anything involving a signature floating around on its own is dangerous, and if I ever design any sort of validation protocol I should avoid it. The indirect lesson is yet another useful reminder that I do not know enough about cryptography to be designing anything involving cryptography. If I try to do this in any non-toy context, the things I don't even know I don't know will probably eat me for breakfast without breaking a sweat.

One limit to how much TLS can do for you against MITM attacks

One of the bits of TLS news recently has been that Kazakhstan is apparently planning to force all Internet users in the country to install a new TLS root certificate (see eg here and here) and then probably use that certificate to intercept whatever HTTPS communication they want to. This situation points out what is not so much a limitation of TLS and encryption in general as a simple absence of magic.

The obvious assumption to make is that the Kazakhstan authorities are going to end up blocking HTTPS connections that aren't willing to accept their MITM certificates. At this point, TLS is up a creek. If an attacker with control over your network link is willing to enforce a policy of 'you can only communicate if we can read your traffic', well, mainstream software has no good options. If it prioritizes secrecy over communications, you can't talk at all; otherwise, you have no secrecy. There is no magic third path.

(Non-mainstream software can attempt to smuggle traffic through various forms of steganography and concealment, but this is not really viable for popular software; the attacker will soon become aware that this smuggling is going on and block it.)

This doesn't mean that TLS is useless in this situation. TLS is forcing the Kazakhstan authorities to do this in public, in a very visible way. Without encryption or the (weak) authentication that certificate validation provides, Kazakhstan could have done this on the sly (and in fact they undoubtedly are doing it for HTTP and other plaintext traffic, just as plenty of other people are). TLS is even forcing this to be visible to software (at least software that has access to the certificate chain).

An encryption system that doesn't allow itself to be defeated this way (including TLS software that refuses this MITM certificate even if the user has accepted it) cannot 'win' on a technical level; its traffic will get blocked and it will fail to achieve both of availability and secrecy. It can only win (if it does eventually win) at the social and political level, if the resulting lack of communication is sufficiently important and painful to change the attacker's actions. Urging that software fail in the favour of secrecy here is essentially a bet that the social level will win out in the end.

(If Kazakhstan is really serious about this they're going to have to block a lot of other network protocols and connections. But they can certainly do this if they want to badly enough; after all, it's no different than what 'block by default' outbound firewalls do in many organizations. Of course they may not go this far; picking off the majority of encrypted traffic might be good enough.)

My feelings about my mechanical mini keyboard

I wrote a while back about my temptation to find a good USB mini keyboard to replace my old PS/2 mini keyboard. Well, I recently gave in to that temptation and got a mechanical mini keyboard, specifically the Matias Mini Quiet Pro, which seemed like close to the best candidate I could find. Although I've only been using it for a few weeks, I want to write my initial feelings down now before my memories of what it was like to use my old keyboard fade too much.

In general, well, the keyboard works (including in the BIOS of my machine when it boots up). It's slightly bigger than my old keyboard (which set off some other changes that are another entry) and it's significantly heavier. It is indeed quiet, just as claimed; it's probably slightly louder than my old keyboard, but not by very much. Certainly it doesn't sound like a clacky mechanical keyboard.

This is effectively my first mechanical keyboard, so I can't say anything about how it feels as compared to other mechanical keyboards. Compared to my old rubber-dome keyboard it feels subtly nicer. On my old rubber-dome keyboard, typing away for long enough could leave me with a low-level feeling that I was slamming my fingers against something hard, probably from bottoming out the keys against their hard stop (I felt it most acutely in the outer fingers of my left hand, which could spend a bunch of time banging on the control and shift keys). This was never painful or anything but it was something that I was definitely aware of. I don't really feel that with my new keyboard, and it's definitely something that I've looked for.

Because the new keyboard's a bit different in size, the actual physical keycaps are in slightly different positions from my old keyboard. This has given me a great demonstration of just how acclimatized my reflexes were (and are) to the exact position of the old keyboard's keys; although I've been getting better over time, I still keep missing keys every so often, usually when I'm just starting to type something. I've also apparently been extremely attuned to just how much pressure I needed to lift from things like the shift and control keys in order to un-shift and un-control things. This is different on the new keyboard, and as a result, well, I've been fixing a certain amount of surprise capital letters and so on. This too has been improving with time. In a few months I'll probably be just as finely tuned to the new keyboard.

(I gave that a helping hand by persuading work to get me one of them too, so I don't have to go back and forth between different keyboards at home and at work.)

The different relative locations of the various function keys has revealed that a number of my window manager key assignments are quite sensitive to the physical location of the function keys I was using. The different key positions on the Matias has moved several function keys further away from my regular hand position than they were before, making it comparatively less convenient to invoke those window manager bindings. I'm still considering how (and if) I want to shuffle key bindings around, but I'll probably wind up making changes sooner or later. Right now I'm being conservative, because I may find that I get used to it in the long run and they're not really any different than before.

(I've already added one adaptation to my environment, but that's another entry.)

PC laptop and desktop vendors are now clearly hostile parties

You may have heard of Lenovo's SuperFish incident, where Lenovo destroyed HTTPS security on a number of their laptops by pre-installing root certificates with known private keys. Well, now Dell's done it too, and not just on consumer laptops, and it turns out not just one bad certificate but several. One could rant about Dell here, but there's a broader issue that's now clear:

PC vendors have become hostile parties that you cannot trust.

Dell has a real brand. It sells to businesses, not just consumers. Yet Dell was either perfectly willing to destroy the security of business oriented desktops or sufficiently incompetent to not understand what they were doing, even after SuperFish. And this was not just a little compromise, where a certificate was accidentally included in the trust store, because a Dell program that runs on startup puts the certificate back in even when it's removed. This was deliberate. Dell decided that they were going to shove this certificate down the throat of everyone using their machines. The exact reasons are not relevant to people who have now had their security compromised.

If Dell can do this, anyone can, and they probably will if they haven't already done so. The direct consequence is that all preinstalled vendor Windows setups are now not trustworthy; they must be presumed to come from a hostile party, one that has actively compromised your security. If you can legally reinstall from known good Microsoft install media, you should do that. If you can't, well, you're screwed. And by that I mean that we're all screwed, because without trust in our hardware vendors we have nothing.

Given that Dell was willing to do this to business desktops, I expect that sooner or later someone will find similar vendor malware on preinstalled Windows images on server hardware (if they haven't already). Of course, IPMIs on server hardware are already an area of serious concern (and often security issues all on their own), even before vendors decide to start equipping them with features to 'manage' the host OS for you in the same way that the Dell startup program puts Dell's terrible certificate back even if you remove it.

(Don't assume that you're immune on servers just because you're running Linux instead of Windows. I look forward to the grim meathook future (tm jwz) where server vendors decide to auto-insert their binary kernel modules on boot to be helpful.)

Perhaps my gloomy cloud world future without generic stock servers is not so gloomy after all; if we can't trust generic stock servers anyways, their loss is clearly less significant. Smaller OEMs are probably much less likely to do things like this (for multiple reasons).

What modern version control systems are

If you read about new version control systems these days, it's very common to see them put forward as essentially the expression or manifestation of mathematics. Maybe it's graph theory, maybe it's patch theory, but the basic idea is that you build up some formal model of patching or version control and then build a VCS system that implements it. This is not restricted to recent VCSes, either; version control as a whole has long had a focus on formally correct operations (and on avoiding operations that were not formally correct).

It is my new belief that this is a terrible misunderstanding of the true role of a VCS, or at least a usable VCS that is intended for general use. Put simply, in practice a VCS is the user interface to the formal mathematics of version control, not the actual embodiment of those mathematics. The job of a good VCS is to sit between the fallible, normal user (who does not operate in the domain of formal math) and the underlying formal math, working away to convert what the user does to the math and what the math says to what the user can understand and use.

As a user interface, a VCS must live in the squishy world of human factors, not the pure world of mathematics. That's its job; it's there to make the mathematics widely usable. This is going to frequently mean 'compromising' that mathematical purity, by which we really mean 'translating what the user wants to do into good mathematics'. I put 'compromise' in quotes here because this is only a compromise if you really think that the user should always directly express correct mathematics.

(We know for sure that users will not always do so, so the only way to pretend otherwise is to spit out error messages any time what the user attempts to do is incorrect mathematics (including the error message of 'no such operation').)

Does this mean that the mathematics is unimportant? Not at all, any more than your skeleton is unimportant in determining your shape. The underlying mathematics can and should shape the user experience that the VCS puts forward (and so different formal models of version control will produce VCSes with different feels). After all, one job of a UI is to steer users into doing the right thing by making it the easy default, and the 'right thing' here is partly determined by the specific math.

PS: The exception to this view of VCSes is a VCS written as an academic exercise to prove that a particular set of version control mathematics can actually be implemented and work. This software is no more intended (or suitable) for general use than any other software from academic research.

VCS bisection steps should always be reversible

So this happened:

@thatcks: I think I just ruined my bisect run with one errant 'hg bisect --bad', because I can't see a way to recover from it in the Mercurial docs.
This is my extremely angry face. Why the hell won't Mercurial give me a list of the bisect operations I did? Then I could fix things.
Instead I appear to have just lost hours of grinding recompilation to a UI mistake. And Mercurial is supposed to be the friendly VCS.

VCS bisection is in general a great thing, but it's also a quite mechanical, repetitive process. Any time you have a repetitive process that's done by people, you introduce the very real possibility of error; when you do the same thing five times in a row, it's very easy to accidentally do it the sixth time. Or to just know that you want the same command as the time before and simply recall it out of your shell's command history except that nope, your reflexes were a bit fast off the mark there.

(It's great when bisection can be fully automated but there are plenty of times when it can't because one or more of the steps requires human intervention to run a test, decide if the result is correct, or the like. Then you have a human performing a series of steps over and over again but they're supposed to do different things at the end step. We should all know how that one goes by now.)

So inevitably, sooner or later people are going to make a mistake during the bisection process. They're going to reflexively mark the point under testing as good when it's actually bad, or mark it as bad when they just intended to skip it, or all of the other variants. It follows directly that a good bisection system that's designed for real people should provide ways to recover from this, to say 'whoops, no, I was wrong, undo that and go back a step' (ideally many steps, all the way back to the start). Bisection systems should also provide a log, so that you can see both what you did and the specific versions you marked in various ways. And they should document this clearly, of course, because stressed out people who have just flubbed a multi-hour bisection are not very good at carefully reading through three or four different sections of your manual and reasoning out what bits they need to combine, if it's even possible.

Of course, this sort of thing is not strictly speaking necessary. Bisection works just fine without it, provided that people don't make mistakes, and if people make mistakes they can just redo their bisection run again from the start. A bisection system with no log and no undo has a pleasantly mathematical sort of minimalism. It's just not humane, as in 'something that is intended to be used by actual humans and thus to cope with their foibles and mistakes'.

Overall, I suppose I shouldn't be surprised. Most version control systems are heavily into mathematical perfection and 'people should just do it right' in general.

(This is a terrible misunderstanding but that's another entry.)

On public areas on the Net and conversations therein

There have been essentially public areas on the 'net for a long time (from the time before the 'net was the Internet). In all of that time, a pattern that repeats over and over is that they get used for what I'll call closed discussions among an in crowd. These discussions happen in public (in a Usenet newsgroup, on a public mailing list or website, on IRC, on Twitter, etc) so they're not private, but they're not public in the usual sense because they're not open to outside participants to butt in on. When this closed nature is not supported by the technology of the medium (which it usually isn't), it will instead be supported by social mores and practices, including ignoring people and the equivalent of mail filters and Usenet killfiles. There may be flaming or mockery of transgressors involved, too.

(If you want an analogy, what is going on is much like a group of people having a discussion at a restaurant table with you one table over. You can hear them fine and they are in 'public', but of course very few people think it's correct to turn around and join in their discussion and doing so rarely gets a good reaction from the group.)

What this means is that a conversation taking place in nominal public is not necessarily an open invitation for outside people to comment, and if they do they may be summarily ignored or find that there are bad reactions to their words. Equally, it's wrong to assert something like 'all conversations in public must include anyone who wants to participate' or the equivalent, because this is not how things work in practice in the real world (either on the 'net or off it).

As I mentioned, people on the 'net have been doing this with public spaces for a very long time now; this behavior is not at all novel or unusual. People who are shocked, shocked to see this happening in any particular instance (especially when they are shoving themselves into other people's discussions) are at best disingenuous. Wherever they are, people make groups and then talk among themselves.

There are also genuinely open public discussions in those public areas of the 'net, which creates obvious possibilities for confusion and misunderstandings. The cues for closed discussions are not always clear and some number of closed discussions are in practice only semi-closed; if you fit in, you can join in the conversation (indeed, this is how many such discussion groups expand). One way to assess the line between good faith misunderstandings of a situation and something else is the degree of stubborn persistence exhibited by the outsider.

(Relevant, and also.)

My discovery that the USB mouse polling rate matters

I mentioned recently that I had found a PS/2 to USB converter that worked for my keyboard, but it made my PS/2 mouse kind of jerky and stuttery. Over the past few days I've discovered why that is so, which is unfortunately kind of bad news for me with this converter.

As you might have guessed from my sudden interest in Linux's mouse polling rate, my problem turns out to be that a PS/2 mouse that's run through the PS/2 to USB converter does not have a high enough polling rate. Courtesy of evhz, I can put actual numbers on this for all of the mice I have handy. With default settings for mouse polling, my scroll wheel mouse runs at 125 Hz, my PS/2 mouse claims a 90 Hz rate when connected to the PS/2 mouse port, and for the PS/2 to USB converter I get a rate that jumps back and forth between 31 Hz and 41 Hz. Increasing Linux's mouse polling to 500 Hz increases the rate of real USB mice to that, but it doesn't really change the performance of the PS/2 to USB converter; evhz still has it at an average of about 36 Hz.

This, it turns out, is nowhere near as fast as it needs to be in order to be really smooth. A 36 Hz average rate is just good enough that the mouse mostly performs reasonably well; it's not obviously terrible or anything. Instead it's just subtly jerky from time to time. I could almost imagine it or persuade myself that it wasn't there (and I did off and on). But it's not good enough.

The bad news is that this is clearly an issue in the PS/2 to USB converter itself. Linux can ask it to run at whatever high rate I want, but the converter simply doesn't pass on mouse motion that fast or frequently. I'm never going to have a satisfactory mouse experience with this converter, although it's been fine for my keyboard. The good news is that I clearly don't need a special fast USB mouse in order to get the responsiveness I'm used to, at least if evhz is correct about my PS/2 mouse running at 90 Hz natively.

(According to this PS/2 mouse reference (also), PS/2 mice do natively run at 100 Hz by default unless the host changes that. Linux probably doesn't.)

This leaves me with a number of ways forward. First, I can try other PS/2 to USB converters purely for their mouse conversion, to see if they run at higher sample rates. Second, I'm trying out a HP three button optical USB mouse. So far it seems okay but perhaps not as smooth as my PS/2 mouse, which may partly be because it's an optical mouse instead of a ball mouse and I'm using it on a fairly featureless surface. All of this is enough options that I'm not really worried about things any more; mostly I'm just going to use my PS/2 mice until I don't have a PS/2 port to plug them into.

(Getting a Contour Mouse remains kind of tempting, though. If I liked it, it would be great. That 'if' is of course the problem, given their price.)