Wandering Thoughts archives

Why I don't believe in generic TLS terminator programs

In some security circles it's popular to terminate TLS connections with standalone generic programs such as stunnel (cf). The stated reason for this boils down to 'separation of concerns'; since TLS is an encrypted TCP session, we can split TLS termination from actually understanding the data streams that are being transported over. A weakness in the TLS terminator doesn't compromise the actual application and vice versa. I've seen people harsh on protocols that entangle the two issues, such as SMTP with STARTTLS.

I'm afraid that I don't like (and believe) in generic TLS terminator programs, though. The problem is their pragmatic impact; in practice you are giving up some important things when you use them. In specific, what you're giving up is easy knowledge of the origin IP address of the connection. A generic TLS terminator turns a TLS stream into the underlying data stream but by definition doesn't understand anything about the structure of the data stream (that's what makes it generic). This lack of understanding means it has no way to pass the origin IP address along to whatever is handling the actual data stream; to do so would require it to modify or augment the data stream somehow, and it has no idea how to do that.

You can of course log enough information to be able to reconstruct this information after the fact, which means that in theory you can recover it during the live session with suitably modified backend software. But this requires customization in both the TLS terminator and the backend software, which means that your generic TLS terminator is no longer a drop in part.

(Projects such as titus can apparently get around this with what is presumably a variant of NAT. This adds a lot of complexity to the process and requires additional privileges.)

I consider losing the origin IP address for connections to be a significant issue. There are lots of situations where you really want to know this information, which means that a generic TLS terminator that strips it is not suitable for unqualified general use; before you tell someone 'here, use this to support TLS' you need to ask them about how they use IP origin information and so on. As a result I tend to consider generic TLS terminators as suitable mostly for casual use, because it's exactly in casual uses that you don't really care about IP origin information.

(You can make a general TLS terminator that inserts this information at the start of the data stream, but then it's no longer transparent to the application; the application has to recognize this new information before the normal start of protocol and so on.)

(These issues are of course closely related to why I don't like HTTP as a frontend to backend transport mechanism, as you have the same loss of connection information.)

Security capabilities and reading process memory

Today, reading Google's Project Zero explanation of an interesting IE sandbox escape taught me something that I probably should have known already but hadn't thought about and realized. So let's start at the beginning, with capability-based security. You can read about it in more detail in that entry, but the very short version is that capabilities are tokens that give you access rights and generally are subject to no further security checks. If you've got the token, you have access to whatever it is in whatever way the token authorizes. Modern Unix file descriptors are a form of capabilities; for example, a privileged program can open a file that a less privileged one doesn't have access to and pass the file descriptor to it to give access to that file. The less privileged program only has as much access to the file (read, write, both, etc) as is encoded in the file descriptor; if the file descriptor is read only, that's it.

When you design a capability system, you have to answer some questions. First, do programs hold capabilities themselves in their own memory space (for example, as unforgeable or unguessable blobs of cryptographic data) or do they merely hold handles that reference them and the kernel has the real capability? Unix file descriptors are an example of the second option, as the file descriptor number at user level is just a reference to the real kernel information. Second, are the capabilities or handles held by the process bound to the process or are they process independent? Unix file descriptors are bound to processes and so your FD 10 is not my FD 10.

My impression is that many strong capability based systems answer that user processes hold as much of the capability token as possible (up to 'all of it') and that the token is not bound to the process. This is an attractive idea from the system design perspective because it means that the kernel doesn't have to have much involvement in storing or passing around capability tokens. Again, contrast this with Unix, where the kernel has to do a whole pile of work when process A passes a file descriptor to process B. The sheer work involved might well bog down a Unix system that tried to make really heavy use of file descriptor passing to implement various sorts of security.

What I had not realized until I read the Google article is that in systems where processes hold capability tokens that are not bound to the particular process, being able to read a process's memory is a privilege escalation attack. If you can read the memory of a process with a capability token, you can read the token's value and then use it yourself. The system's kernel doesn't know any better, and in fact it not knowing is a significant point of the whole design.

This matters for two reasons. First, reading process memory is often an explicit feature of systems to allow for debugging (it's hard to do it otherwise). Second, in practice there have been all sorts of attacks that cause processes to leak some amount of memory to the attacker and often these disclosure leaks are considered relatively harmless (although not always; you may have heard about Heartbleed). In a capability based system, guarding against both of these issues clearly has to be a major security concern (and I'm not sure how you handle debugging).

(In many systems you probably don't need to use the capabilities in the process that read them out of someone's memory. If they really are process independent, you can read them out in one process, transfer them through the Internet, and pass them back in to or with a completely separate process that you establish in some other manner. All that matters is that the process can give the kernel the magic token.)

Using the SSH protocol as a secure transport protocol

I have an IPSec problem: my IPSec tunnel uses constant keys, with no periodic automatic rekeying. While IPSec has an entire protocol to deal with this called IKE, in practice IKE daemons (at least on Linux) are such a swamp to wade into that I haven't been willing to spend that much time on it. Recently I had a realization; rather that wrestle with IKE, I could just write a special purpose daemon to rekey the tunnel for me. Since both ends of the IPSec tunnel need to know the same set of keys, I need to run the daemon at either end and the ends have to talk to each other. Since they'll be passing keys back and forth, this conversation needs to be encrypted and authenticated.

The go-to protocol for encryption and authentication is TLS. But TLS has a little problem for my particular needs here in that it very much wants to do authentication through certificate authorities. I very much don't want to. The two endpoints are fixed and so are their authentication keys, and I don't want to have to make up a CA to sign two certificates and (among other things) leave myself open to this CA someday signing a third key. In theory TLS can be used with direct verification of certificate identities, but in practice TLS libraries generally make this either hard or impossible depending on their APIs.

(I've written about this before.)

As I was thinking about this it occurred to me that there is already a secure transport protocol that does authentication exactly the way I want it to work: SSH. SSH host keys and SSH public key authentication is fundamentally based on known public keys, not on CAs. I don't want to literally run my rekeying over SSH for various reasons (including security), but these days many language environments have SSH libraries with support for both the server and client sides. The SSH protocol even has 'do command' operation that can be naturally used to send operations to a server, get responses, and perhaps supply additional input.

It's probably a little bit odd to use SSH as a secure transport protocol for your own higher level operations that have nothing to do with SSH's original purpose. But on the other hand, why not? If the protocol fits my needs, I figure that I might as well be flexible and repurpose it for this.

(The drawback is that SSH is relatively complex at the secure transport layer if all that you want is to send some text back and forth. Hopefully the actual code complexity will be minimal.)

I want opportunistic, identity-less encryption on the Internet

The tech news has recently been full of reports that some ISPs are modifying SMTP conversations passing through them in order to remove encryption, or more specifically to remove a note that the server is offering it (this goes by the name of 'STARTTLS'). Some people are up in arms over it; others are saying that this is no big deal because it's not as if TLS on SMTP conversations means much in the first place. This latter viewpoint may raise some eyebrows. The reason why TLS SMTP means less than you think is that essentially no clients require signed and verified TLS certificates from the SMTP server; instead they'll almost always accept any random TLS certificate. In one set of jargon, this is called 'opportunistic' encryption; you could also call it 'unauthenticated' encryption. Its drawback is that this lack of authentication of the server means that the server could be anyone, including an attacker playing man in the middle to read your 'TLS encrypted' email.

This general issue is a long-standing dispute in Internet cryptography. On one side are people who say that opportunistic encryption is better than nothing; on the other side are people who say that it's effectively nothing because of its vulnerability to MITM attacks (and that it has other bad side effects, like causing people to think that they're more secure than they are). Once upon a time I might have been reasonably sympathetic to the second view, but these days I have come completely around to the first view.

As far as the second view goes: yes, there are many places that are perfectly happy to perform MITM attacks on you at the drop of a hat. Most such places generally don't call them MITM attacks, of course; instead they're 'captive portals' and so on. Those ISPs that are stripping STARTTLS markers are effectively doing a MITM attack. But in practice, in the real world, this is not equivalent to having no encryption at all. The big difference between opportunistic encryption and no encryption is that opportunistic encryption completely defeats passive monitoring. And in the real world, passive monitoring is now pervasive.

That is why I want as much opportunistic encryption as possible; I want that pervasive passive monitoring to get as little as possible. Sure, in theory the opportunistic encryption could be MITMd. In practice no significant amount of it is likely to be, because the resources required to do so go up much faster than with passive monitoring (and in some situations so do the chances of getting noticed and caught). Opportunistic encryption is not theoretically clean but it is practically useful, and it is much, much easier to design in and set up than authenticated encryption is.

(And this is why ISPs stripping STARTTLS matters quite a bit.)

What you're saying when you tell people to send in patches

Exerpted from a comment on my entry about my problem with reporting CentOS bugs:

You could not be more wrong about your assumptions:

[...]

• Bugs.centos.org is community help .. meaning that our users and volunteer QA team answer questions there. Not only should you report bugs there .. you should also find and fix, the report the fix there. That is how open source works. You should fix the problem, report the fix to bugs.centos.org and bugzilla.redhat.com if you can .. I mean, you are getting the software for free, right?

When I read things like this, where people tell other people 'it's open source, you should be finding and submitting patches', this is how I expect those other people are reading it. Whether you realize it or not and whether you intend it or not, telling people 'submit a patch' is actually telling people 'go away, you annoying thing'.

(It's also sending the message that your project is only really for developers, who are the people who can actually come up with those patches. Mere mortals need not apply. For that matter, developers who have other things to work on need not apply either.)

Sometimes, occasionally, this is an appropriate thing to say. If someone is showing up in your bug report system to demand that you fix a bug (or is otherwise complaining loudly that an open source community is failing to do so), sure, go ahead and tell them to go away and shut up. Mind you, telling them a softball version of 'patch or GTFO' is kind of passive-aggressive; perhaps you could be more straightforward and just say 'we're not going to fix this, if you need a fix badly enough you'll need to do it yourself'.

But as a general reply? No. As a general reply or a general suggestion it's both rude and a bridge burner. It also hangs a kind of smug open source arrogance out for bystanders to see and take note of.

Oh, as a side note, saying this sort of thing is also kind of insulting in that it suggests (by implication) that someone with the capacity to create a fix has not realized that gosh, they could do it. Of course, there are people who don't think that they're good enough to create fixes or lack the confidence to do so and who need encouragement, but such gentle encouragement is not delivered in anything even vaguely close to this 'send patches' manner (any more than encouragement to submit bug reports is, cf the comments on this entry).

PS: this is one of the rare entries when I will say the following directly and explicitly: if you're thinking of being deliberately rude in a comment, either on this entry or elsewhere, go away. Enabling and hosting rudeness is not anywhere near why I have comments here and rude comments may be subject to summary removal if I am angry enough.

Sidebar: the toxic nature of 'that is how open source works'

I want to point the following out explicitly. If, to quote the comment, 'not only should you report bugs there .. you should also find and fix, the report the fix there. That is how open source works' is actually how open source works then the direct corollary is that open source is only really for developers, who are the only people who can actually find the source of bugs and fix them. Everyone else is a hanger-on and camp follower.

To put it mildly, I think that a lot of people in the open source world would strongly disagree with the view that their open source work is only or primarily for developers.

(I can't call this view a toxic one, although I want to. It's certainly toxic for wide use of open source, but if your view is 'open source is for developers' then you've already decided that you don't care about a wide use of open source.)

The weakness of doing authentication over a side channel

Yesterday I mentioned our method of authenticating NFS client hosts; fundamentally it operates by every so often verifying that the client host knows a secret. Suppose that we had a slightly improved version of this, where the NFS fileserver holds an authenticated TCP connection open with the client and periodically exchanges authenticated and encrypted packets with it; the simple version of this would just be a SSH connection with SSH level keepalives. Is this a reasonably secure system or is it attackable?

(A system without a continuous authentication connection is trivially attackable; get the real client to authenticate once, force it off the network, and replace it with your imposter client.)

Unfortunately, yes it is. Take your attack host with two network interfaces and insert it as a bridge between a valid client and the client's normal network. Now set your host to pass SSH traffic through the bridge to the valid client (and back out) but to intercept and generate its own NFS traffic. We will now authenticate the valid client but take NFS requests from your imposter client, and the authentication channel will stay perfectly live while you're doing this.

As far as I can tell this is a fundamental weakness of doing authentication over a side channel instead of the channel that your main communication is flowing over. If the authentication is not strongly connected with the actual real conversation, an attacker can peel the two apart and pass the authentication to a valid client while handling the real conversation itself. For full security, the authentication should be an intrinsic and inseparable part of the main communication.

(There are hacks you can try if you're stuck with separate channels, like having the client observe signs of the main protocol in action and report them back over the authentication channel. If this doesn't match the server's view of the client's activity, something's up.)

PS: I'm sure this is well known in the security and protocol design community. I'm writing it down for myself, because I want to remember the logic of this after I worked it out in my head.

Practical security and automatic updates

One of the most important contributors to practical, real world security is automatically applied updates. This is because most people will not take action to apply security fixes; in fact most people will probably not do so even if asked directly and just required to click 'yes, go ahead'. The more work people have to go through to apply security fixes, the fewer people will do so. Ergo you maximize security fixes when people are required to take no action at all.

(Please note that sysadmins and developers are highly atypical users.)

But this relies on users being willing to automatically apply updates, and that in turn requires that updates must be harmless. The ideal update either changes nothing besides fixing security issues and other bugs or improves the user's life. Updates that complicate the user's life at the same time that they deliver security fixes, like Firefox updates, are relatively bad. Updates that actually harm the user's system are terrible.

Every update that does harm to someone's system is another impetus for people to disable automatic updates. It doesn't matter that most updates are harmless and it doesn't matter that most people aren't affected by even the harmful updates, because bad news is much more powerful than good news. We hear loudly about every update that has problems; we very rarely hear about updates that prevented problems, partly because it's hard to notice when it happens.

(The other really important thing to understand is that mythology is extremely powerful and extremely hard to dislodge. Once mythology has set in that leaving automatic updates on is a good way to get screwed, you have basically lost; you can expect to spend huge amounts of time and effort persuading people otherwise.)

If accidentally harmful updates are bad, actively malicious updates are worse. An automatic update system that allows malicious updates (whether the maliciousness is the removal of features or something worse) is one that destroys trust in it and therefor destroys practical security. As a result, malicious updates demand an extremely strong and immediate response. Sadly they often don't receive one, and especially when the 'update' removes features it's often even defended as a perfectly okay thing. It's not.

PS: corollaries for, say, Firefox and Chrome updates are left as an exercise to the reader. Bear in mind that for many people their web browser is one of the most crucial parts of their computer.

(This issue is why people are so angry about FTDI's malicious driver appearing in Windows Update (and FTDI has not retracted their actions; they promise future driver updates that are almost as malicious as this one). It's also part of why I get so angry when Unix vendors fumble updates.)

Making bug reports is exhausting, frustrating, and stressful

I've danced around this subject before when I've written about bug reports (and making bug reports), but I want to come out and say it explicitly: far too often, making bug reports is an exhausting experience that is frequently frustrating and stressful.

This is not because the tools for doing it are terrible, although that doesn't help. It is because the very frequent result of trying to make a bug report is having to deal with people who don't believe you, who don't take you seriously, and who often don't read, consider, and investigate what you wrote. Some of the time it involves arguing with people who disagree with you, people who feel that what you are reporting is in fact not a bug or at best a trivial issue. The crowning frustration on top of all of these experiences is that after all of your effort and the stress of arguing with people, the bug will often not be fixed in any useful fashion. By the way, that 'deal with' is often actually 'argue with' (which is about as much fun as you'd expect).

(A contributing factor to the stress is often that you really need a fix or a workaround for the bug.)

Whether or not they can articulate it, everyone who's made enough bug reports knows this in their gut. In my opinion it's a fairly big reason why a lot of people burn out on making bug reports and stop doing it; it's not that they're making carefully considered cost/benefit calculations (no matter what I've written before about this), it's that they have absolutely no desire to put themselves through the whole exercise again. The frequently low cost/benefit ratio is a post-facto rationalization that people would reach for much less if the whole experience was actually a pleasant one.

There is a really important corollary for this: if you're tempted to urge someone to make a bug report, especially a bug report that you reasonably expect may be rejected, you should understand that you're trying to get them to put themselves through an unpleasant experience.

(I think this is a big part of why I have a very strong urge to bite the heads off of people who respond to me to suggest that I should file bug reports.)

Changing major version numbers does not fix compatibility issues

So in the wake of the Bash vulnerability I was reading this Errata Security entry on Bash's code (via due to an @0xabad1dea retweet) and I came across this:

So now that we know what's wrong, how do we fix it? The answer is to clean up the technical debt, to go through the code and make systematic changes to bring it up to 2014 standards.

This will fix a lot of bugs, but it will break existing shell-scripts that depend upon those bugs. That's not a problem -- that's what upping the major version number is for. [...]

There are several issues here, but let me start with the last statement. This statement is wrong, and let me say it plainly: changing your major version number does not fix compatibility problems. Changing your version number is not some magic get out of problems card that makes your new version compatible with your old version; regardless of its version number it remains just as incompatible with the old version as before. The purpose of changing version numbers is to warn people of potential problems, and by extension to reassure them that there won't be problems for minor version changes (which makes them more likely to adopt those changes).

If you introduce breaking changes in Bash or in any program, things that people have today will break. That is what breaking changes mean. People will still have to either fix them or stay with the old version of the program and yes, the latter really is an option whether or not you like it (it is especially an option if you are proposing changes in something that is not actually your program). When you introduce breaking changes you are asking people do extra work, which they do not enjoy in the least; generally the more breaking changes you introduce the more work you create. If you want your changes to be accepted, you need to be very sure that what you are offering is worth that work. Otherwise you are going to get an unpleasant surprise.

Changing the major version number does absolutely nothing to change this dynamic and thus it does nothing at all to make breaking changes less of a problem or less of an issue. You cannot hand wave away the cost of breaking changes this way, no matter how much programmers would love to do so and how this 'we'll just change major versions' thing is an ever-popular idea in languages, libraries, systems, commercial software, tools, and so on.

(Note that being left with no real option but to do a bunch of work also drastically lowers the cost of moving to some other alternative; you no longer have the difference between 'do nothing' and 'do the migration', now it is just the difference between 'fix the existing stuff' and 'do the migration'.)

My view on using VLANs for security

I've recently read some criticism of the security value of VLANs. Since we use VLANs heavily I've been thinking a bit about this issue and today I feel like writing up my opinions. The short version is that I don't think using VLANs is anywhere close to being an automatic security failure. It's much more nuanced (and secure) than that.

My overall opinion is that the security of your VLANs rests on the security of the switches (and hosts) that the VLANs are carried on, barring switch bugs that allow you to hop between VLANs in various ways or to force traffic to leak from one to another. The immediate corollary is that the most secure VLANs are the ones that are on as few switches as possible. Unfortunately this cuts against both flexibility and uniformity; it's certainly easier if you have all of your main switches carry all of your VLANs by default, since that makes their configurations more similar and means it's much less work to surface a given VLAN at a given point.

(This also depends on your core network topology. A chain or a ring can force you to reconfigure multiple intermediate switches if VLAN A now needs to be visible at a new point B, whereas a star topology pretty much insures only a few directly involved switches need to be touched.)

Because they're configured (partly) through software instead of purely by physical changes, a VLAN based setup is more vulnerable to surreptitious evil changes. All an attacker has to do is gain administrative switch access and they can often make a VLAN available to something or somewhere it shouldn't be. As a corollary, it's harder to audit a VLAN-based network than one that is purely physical in that you need to check the VLAN port configurations in addition to the physical wiring.

(Since basically all modern switches are VLAN-capable even if you don't use the features, I don't think that avoiding VLANs means that an attacker who wants to get a new network on to a machine needs the machine to have a free network port. They can almost certainly arrange a way to smuggle the network to the machine as a tagged link on an existing port.)

So in summary I think that VLANs are somewhat less secure than separate physical networks but not all that much less secure, since your switches should be fairly secure in general (both physically and for configuration changes). But if you need ultimate security you do want or need to build out physically separate networks. However my suspicions are that most people don't have security needs that are this high and so are fine with using just VLANs for security isolation.

(Of course there are political situations where having many networks on one switch may force you to give all sorts of people access to that switch so that they can reconfigure 'their' network. If you're in this situation I think that you have several problems, but VLANs do seem like a bad idea because they lead to that shared switch awkwardness.)

Locally we don't have really ultra-high security needs and so our VLAN setup is good enough for us. Our per-group VLANs are more for traffic isolation than for extremely high security, although of course they and the firewalls between the VLANs do help increase the level of security.

Sidebar: virtual machines, hosts, VLANs, and security

One relatively common pattern that I've read about for virtual machine hosting is to have all of the VLANs delivered to your host machines and then to have some sort of internal setup that routes appropriate networks to all of the various virtual machines on a particular host. At one level you can say that this is obviously a point of increased vulnerability with VLANs; the host machine is basically operating as a network switch in addition to its other roles so it's an extra point of vulnerability (perhaps an especially accessible one if it can have the networking reconfigured automatically).

My view is that to say this is to misread the actual security vulnerability here. The real vulnerability is not having VLANs; it is hosting virtual machines on multiple different networks (presumably of different security levels) on the same host machine. With or without VLANs, all of those networks have to get to that host machine and thus it has access to all of them and thus can be used to commit evil with or to any of them. To really increase security here you need to deliver fewer networks to each host machine (which of course has the side effect of making them less uniform and constraining which host machines a given virtual machine can run on).

(The ultimate version is that each host machine is only on a single network for virtual machines, which means you need at least as many host machines as you have networks you want to deploy VMs on. This may not be too popular with the people who set your budgets.)