(Open)SSH quiet connection disconnects in theory and in practice

November 26, 2018

Suppose, not entirely hypothetically, that you are making frequent health checks on your machines by connecting to their SSH ports to see if they respond. You could just connect, read the SSH banner, and then drop the connection, but that's abrupt and also likely to be considered a log-worthy violation of the SSH protocol (in fact it is considered such by OpenSSH; you get a log message about 'did not receive identification string'). You would like to do better, in the hopes of reducing your log volume. It turns out that the SSH protocol holds out the tantalizing prospect of doing this in a protocol-proper way, but it doesn't help in practice.

The first thing we need to do as a nominally proper SSH client is to send a protocol identification string; in the SSH transport layer protocol this is covered in 4.2 Protocol Version Exchange. This is a simple CR LF delimited string that must start with 'SSH-2.0-'. The simple version of this is, say:

SSH-2.0-Prometheus-Checks CR LF

After the client sends its identification string, the server will begin the key exchange protocol by sending a SSH_MSG_KEXINIT packet (section 7.1). If you use nc or the like (I have my own preferred tool) to feed a suitable client version string to a SSH server, you can get this packet dumped back at you; conveniently, almost all of it is in text.

(In theory your client should read this packet so that the TCP connection doesn't wind up getting closed with unread data.)

At this point, according to the protocol the client can immediately send a disconnect message, as the specification says it is one of the messages that may be sent at any time. A disconnect message is:

uint32    reason code
string    description in ISO-10646 UTF-8 encoding [RFC3629]
string    language tag

How these field types are encoded is covered in RFC 4251 section 5, and also the whole disconnect packet then has to be wrapped up in the SSH transport protocol's Binary Packet Protocol. Since we're early in the SSH connection and have not negotiated message authentication (or encryption), we don't have to compute a MAC for our binary packet. If we're willing to not actually use random bytes in our 'random' padding, this entire message can be a pre-built blob of bytes that our checking tool just fires blindly at the SSH server.

In practice, this doesn't work because OpenSSH logs disconnect messages; in fact, it makes things worse because OpenSSH logs both that it received a disconnect message and a 'Disconnect from <IP>' additional message. We can reduce the logging level of the 'received disconnect' message by providing a reason code of SSH_DISCONNECT_BY_APPLICATION instead of something else, but that just turns it down to syslog's 'info' level from a warning. Interesting, OpenSSH is willing to log our 'description' message, so we can at least send a reason of 'Health check' or something. I'm a little bit surprised that OpenSSH is willing to do this, given that it provides a way for Internet strangers to cause text of their choice to appear in your logs. Probably not very many people send SSH_MSG_DISCONNECT SSH messages as part of their probing.

On the one hand, this is perfectly reasonable on OpenSSH's part. On the other hand, I think it's probably not useful any more to log this sort of thing by default, especially for services that are not infrequently exposed to the Internet.

(I was going to confidently assert that there are a lot of SSH scanners out there, but then I started looking at our logs. There certainly used to be a lot, but our logs are now oddly silent, at least on a first pass.)

Sidebar: Constructing an actual disconnect message

I was going to write out a by-hand construction of an actual sample message, but in the end I had so much trouble getting things encoded that I wrote a Python program to do it for me (through the struct module). Generating and saving such messages is pointless anyway, since they don't reduce the log spam.

Still, building an actual valid SSH protocol message more or less by hand was an interesting exercise, even if having no MAC, no encryption, and no compression makes it the easiest case possible.

(I also left out the 'language tag' field by setting its string length to zero. OpenSSH didn't care, although other SSH servers might.)

Written on 26 November 2018.
« Firefox's middle-click behavior on HTML links on Linux
Go 2 Generics: A way to make contracts more readable for people (if not programs) »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Nov 26 22:30:11 2018
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.