Wandering Thoughts archives

Finding out what TLS/SSL cryptography people actually get with your servers

One of my hobbies is slowly improving the SSL (okay, TLS) security settings on our various TLS-enabled servers, in pursuit of both better practical security with real clients and things like forward secrecy. In an ideal world things would come preconfigured with the best setups possible, but that doesn't always happen in the real world (note that those settings are from 2010, which means that they are now obsolete). Part of doing a good job of this is testing things to make sure that the server settings actually do what I want them to do, especially with real clients. There are so many SSL/TLS bits and it is easy to miss something or set configurations that look good but which will have no meaningful effect in the real world when you interact with real (and imperfect) clients. Sadly, this is more difficult than you would like. Much more difficult.

There are several things going on to get in the way. The first question is what cipher suites your server actually supports and what your specifications have turned into in practice. If you're using OpenSSL (as most people are) the way to find this out is with 'openssl ciphers -v <ciphers spec>'. As a bonus this will print out detailed information about each of the cipher suites, showing you the key exchange, certificate authentication, stream encryption, stream MAC, and which SSL/TLS standard things came from. This listing comes out in the server's preference order, most preferred first.

(If you want to see how your web server's TSL settings stack up in general, I quite like Qualys's SSL Server Test. This doesn't help for internal web servers or for things like IMAP servers.)

There's an immediate gotcha for the unwary: the server preference often doesn't matter. In the TLS handshake the client gives the server a list of what cipher suites it supports, in its preference order, and servers usually defer to the client's preferences. You want to turn this off, forcing use of the server's preference order instead of the client's.

The next question is what cipher suites any particular client supports and what cipher suite was actually picked for a conversation (which is the real test). In an ideal world clients would tell you at least the latter (either natively or with an extension). In the real world, not so much; many clients give you little or no information and you need go around behind their back to get it.

(For example Firefox won't tell you what key exchange was used for a HTTPS connection, only what stream cipher is in effect for it (I'm pleased to see that modern versions of Chrome will tell you both), and I can't see how to get Thunderbird to tell me anything about a TLS protected IMAP connection.)

Fortunately the initial TLS handshake is mostly unencrypted, which means that we can snoop in on the conversation and see the client's list of supported cipher suites as well as the server's actual choice of cipher suite. If it works for you, the most convenient tool for doing this is ssldump. If it doesn't the best general tool for this is probably Wireshark, which has full TLS/SSL protocol decoding. In theory you can use tshark to dump this in ASCII from the command line; in practice I haven't been able to get this working as nicely as the GUI.

(Since I already had to reconstruct this stuff from my cryptic notes once, it's clearly high time I wrote it down in a somewhat more comprehensible form.)

Sidebar: ssldump and Wireshark

I once had ssldump working, but these days all it does is dump the initial ClientHello message and then report 'ERROR: Length mismatch'. This is actually good enough if all you care about is knowing what cipher suites the client supports (and what its preference order is).

There is probably some clever way to use tshark options to display just the TLS parts of the packets, but at the moment the best I can do is:

tshark -i INTERFACE -R ssl.handshake -V -p -n "host IP and port WHATEVER"

This unfortunately dumps a verbose decode of everything in the TLS handshake packets, from the raw frame on up.

If you want just the ClientHello and the ServerHello, you can use '-R "ssl.handshake.type == 1 or ssl.handshake.type == 2"'.

Thinking about how I use email

I've recently wound up thinking that sysadmins probably use email a little bit differently than other people, so I've decided to write down some words on how I use email (as distinct from the tools I use for this).

A bunch of my email is archives, in a sprawling set of folders. I archive things for a variety of reasons; the major ones are more or less sentimentality (email to and from some people), for active future reference (for example, I keep a copy of the current build instructions for all of our machines), and as a genuine archive (for example, I keep a copy of all conversations with vendors). The important thing about archives is that I almost never look at them; they are not active email. As a result they're not really interesting in terms of email handling.

My active email is in my inbox and spinoffs from it. My habit is to almost always keep entire conversations, start to finish, until the whole issue has been concluded (at which point I'll delete the entire thread). Conversations that I'm involved in or might be almost always stay in my inbox; conversations that I'm just monitoring I try to shove off to a side folder (although I don't usually manage to do it right away; often I hope that the conversation is only going to be a short one, so I can delete it soon). In short and in theory, my inbox is for active things that concern me, things that I need to stay on top of (including things where I need to do something and then send a reply).

(The problem with the side folders is that I never look at them, so they usually have plenty of now-concluded conversations that I could delete. But pruning them is the kind of low priority boring work that's hard to get around to.)

In general, what stays in my inbox is messages and conversations where I want to do something as a reaction to them; reply to them, read something they're pointing me to, keep track of whether a condition has cleared, and so on. While I get a constant flow of automated status messages, alerts, and so on, I read (or skim) these and immediately delete them unless they're actually something I need to react to. I try not to use my inbox as a todo list (I have a separate system for that), but I'm not always entirely successful.

(One reason not to use my inbox as a todo list is that my mail client only shows me a few of the most recent messages. Once a message disappears off that list I'm no longer being reminded of it and poof, it's gone from my mind.)

I don't have a good solution for things that I want to get around to but that I'm not. I don't always want to delete them and they shouldn't really stay in my inbox for long, but I'm always reluctant to refile them somewhere else because I know that makes it basically sure I'll never deal with them (and it's hard to give up on the idea that someday I will etc).

I should mention this explicitly: I don't tolerate unread messages in my inbox. I always read (or skim) everything right away, whether or not I do anything with it. To make this viable, I strongly limit what I let get into my inbox; almost everything that winds up there is at least theoretically important and relevant to me.

(I almost never subscribe to mailing lists (there are better alternatives) and I've ruthlessly pruned various web site notifications and so on down to things that I actually care about. Every so often I conduct another slash and burn round; for extreme cases I 'unsubscribe' by deleting the email alias that I gave wherever it was. As for automated status messages from local stuff, I procmail away as many of the 'everything is fine' status messages as I can and I try to make systems quiet in general.)

Solve your command parsing problems by using scripts

Every so often sysadmins have a common form of problem: they need to configure program A to run program B with various arguments but program A's configuration system has some sort of baroque system for specifying commands and evaluating their arguments. What gets quoted or unquoted when, how things are handed to what, and so on are often unclear, unhelpful, or both. Everything works fine in simple cases but if you need to specify arguments for program B that have troublesome elements (like spaces) or do things like pipelines or redirections, well, you have a headache. Maybe several.

In this situation my standard advice is this: the easy way to solve all of these issues is to do all the work with a script. Instead of trying to configure program A to run a complex or troublesome command line, put the complex command line into a script and have program A run the script. Give the script whatever dynamic arguments it needs to have, but do so in a simple form; the script can glue arguments back together again or manipulate them as it needs to.

(The script should itself contain any and all constant arguments; there's no need to pass them from program A to the script. Configuration settings can go either way; it may be better to put them in program A's configuration file and pass them to the script than to embed them into the script.)

This has several benefits. First off, you are no longer wrestling with program A about how it handles command lines. Almost any program that supports running outside programs at all has solid support for simple forms such as '${run /some/script $arg1 $arg2}', and that's all you need. Any complex things you need you can do in the shell, which is probably both more powerful and more predictable than program A's configuration language (and if you run into problems, there are lots of resources for shell quoting issues).

Second, you are probably better off from a security perspective. Securely running programs in the face of potentially malicious input is somewhat of a minefield, but the simplest form of it is the easiest for programs to get correct and thus where they're most likely to have done so. Every bit of complexity you add to program A's handling of command parsing is another potential problem.

(If you care about this (and you probably should), you should test how program A behaves.)

(This is the flipside of the advice to program implementers that I put in my entry on securely running programs.)

By the way (and I am going to write more about this later), if you are ever tempted to solve your command parsing problems by doing something that looks like '${run /bin/sh -c ....}', STOP. Do not do that. Deliberately running the shell directly by hand straight from program A's configuration file makes your problems much, much worse. This is not a pattern or a solution, this is an anti-pattern. Use a script.