Link: "The History of a Security Hole" (in various *BSD kernels)
To yank my words from Twitter, Michal Necasek's The History of a Security Hole is a fascinating exploration of both the arcana of the x86 and what C can innocently do to you. Watching the code train barrel down the tracks towards its doom was decidedly compelling. There are also some useful lessons for long term software development that can be extracted here, since many of the mistakes made were entirely natural ones.
I often find this sort of stuff fascinating, so I really liked reading this entry and found I couldn't look away once things got going and mistakes piled up on top of misunderstandings. By the way, don't read this as a slam on the *BSDs; this sort of cascading misunderstanding can happen in any software, and undoubtedly has happened in non-BSD kernels as well in spots. It's simply easy to miss things in large, complex software (see eg).
An extravagant and dense piece of malware-laden email
In the process of looking through our mail system logs for my entry on phish spam with multiple tries, I stumbled over the following extravagant and apparently densely packed email message:
<ID> attachment application/zip; MIME file ext: .zip; zip exts: .jar; inner zip exts: .class .mf none; email is in-dnsbl rejected <ID> from email@example.com to <redacted>: identified virus: CXmail/JarZip-A, Mal/Jacksbot-A, Mal/Jeetrat-A, Troj/JavaBz-WB detail <id> Subject: [PMX:SPAM] [PMX:VIRUS] Kindly Quota
That's a single attachment with a relatively ordinary looking '.jar in .zip' (well, for malware), where Sophos identified no less than four different sorts of bad things. I wonder if every single .class file in that JAR had a different piece of malware in it.
(It also appears possible that Sophos identified the JAR file as a whole as being one sort of malware and then some pieces inside it as being additional sorts of malware. But all of this is opaque.)
At the time of this message, the IP address was in zen.spamhaus.org and cimexcuritibas.com was in dbl.spamhaus.org. Neither are true any more, so someone cleaned up something. We logged the message headers, but none of them have anything interesting except that the message was DKIM-signed by cimexcuritibas.com with a valid signature.
(This goes to show that valid DKIM signatures mean absolutely nothing about the quality of the email itself. I'm sure we all knew this already, but I like to provide examples every so often.)
PS: It appears that we don't receive any valid, accepted email that has a single .jar in a .zip by itself. It's possible that this is a case like singleton nested zipfiles, where we should just block all of these out of hand. On the other hand, if we did this I wouldn't get lovely log reports like this (we reject bad attachment types before we run them past Sophos PureMessage).
If one phish spam doesn't succeed, maybe another will
Here is another entry in the annals of spammers trying extra hard, to go with an earlier one. Recently, our mail system logged the following two interesting cases. Let's start with the first one:
<ID> attachment application/octet-stream; MIME file ext: .pdf <ID> attachment application/octet-stream; MIME file ext: .htm rejected <ID> from firstname.lastname@example.org to <redacted>: identified virus: Mal/Phish-A, Troj/PDFUri-FUP detail <ID> Subject: [PMX:SPAM] [PMX:VIRUS] Re: Document for Last Shipments
It seems our spammer is trying to get people two ways, trying both some malware and also a phish spam as a HTML attachment. At one level this feels like a sensible approach; if the recipient's system blocks the malware attack, maybe the recipient will still respond manually to the phish spam. But that seems to assume a world where people can have malware not work and be blocked without having big red alerts show up all over the entire email. Perhaps that is the case; if so, that's depressing.
Then there's the second and perhaps more interesting case:
<ID> attachment application/octet-stream; MIME file ext: .html <ID> attachment application/octet-stream; MIME file ext: .html rejected <ID> from 18.104.22.168/TAX@GOV.COM to <redacted>: identified virus: Troj/Phish-DFM, Troj/Phish-DGF detail <ID> Subject: [PMX:SPAM] [PMX:VIRUS] Tax Clearance Certificate
(This 'TAX@GOV.COM' spammer turns out to have been trying for a while from several sources.)
This prompted me to go looking through our logs in search of messages that were identified by Sophos as having multiple bad things in them in the hopes that I'd find a double-phish case. Sadly I wasn't able to find any, and most of them were less interesting than these ones. There was one exception, but that's going to be another entry.
NFS directory reading and directory file type information
NFS has always had an operation to read directories (unsurprisingly
READDIR). In NFS v2, this operation simply returned a list
of names (and 'fileids', ie inode numbers). One of the things that
NFS v3 introduced was an extended version of this, called
that returns some additional information along with the directory
listing. This new operation was motivated by the observation that
NFS clients often immediately followed a
READDIR operation by a
bunch of additional NFS calls to get additional information on many
or all of the names in the directory. In light of the fact that
file type information is available in Unix directories at least some of the time (on many Unixes),
I found myself wondering if this file type information was sufficient
for an NFS server to implement
READDIRPLUS, so that such a Unix
READDIRPLUS requests purely from reading the directory
As far as I can see, unfortunately the answer is that it isn't.
Directory file type information only gives you the file type of
each name, while NFS v3's
READDIRPLUS operation is specified in
RFC 1813 as returning full
information on each name, what the standard calls a
on page 22). This is basically the same as what you get from
and this implies that the NFS server has to read each inode to pull
up this information. That's kind of a pity, at least for NFS v3,
and one of the consequences is that you can't get the same type of
high-efficiency file type scanning over NFS v3 as you can locally.
We have historically used NFS v3 only and so I default to mostly
looking at it. However, there's also NFS v4 (specified in RFC
7530), and once I looked at
it, it turns out to be different in an important way. NFS v4 has
READDIR operation, but it has been defined to allow the
NFS client to specify what attributes of each name it wants to get
back. A NFS v4 client can thus opt to ask for only fileids and file
type information, which permits an NFS v4 server to satisfy the
READDIR request purely from reading the directory, without having
stat() each file in the directory. Even in the possible case
where file type information isn't known for all files, the NFS v4 server would only have
stat() some files, not all of them.
With that said, I don't know if NFS v4 clients actually make such
READDIR requests or if they actually ask NFS v4 servers
to give them enough extra information that the server has to
everything. Sadly, one thing clients could sensibly want to know
to save time is the NFS filehandle of each name, and the filehandle
generally requires information that needs a
(Learning this about NFS v4 may make us more interested in trying
to use it, assuming that we can make everything work in NFS v4 with
sec=sys' Unix style 'trust the client's claims about
UIDs and GIDs' security.)