Third party ClamAV signatures seem to include a lot of phish and other spam

October 24, 2019

For reasons well beyond the scope of this entry, we are currently looking at how ClamAV would do at checking our incoming email for malware, or at least the portion of our incoming email that remains after we immediately reject messages with certain sorts of bad attachments. Our initial experimentation just used whatever collection of signatures the Ubuntu 18.04 LTS package of ClamAV downloads and uses by default, which rapidly demonstrated that this was nowhere near good enough.

(We determined this by cross comparing ClamAV's results against the malware spotted in incoming email by our current system.)

To try to deal with this, we added the unofficial signatures that are fetched by the apparently well regarded third party script that's commonly used for this, which come from an assortment of places and are obviously maintained and updated by an assortment of people. This collection of signatures seems to be much better at recognizing the malware that our users get in email, but in watching our logs I noticed that what I consider 'malware' wasn't all that these signatures were recognizing. Instead, many of the signature names recognized in our email had names that included things like '.Scam.', '.Spam.', '.Junk.', '.Jurlbl.' (apparently used for 'junk URLs', cf), '.Phish.', and so on.

At one level, this is nothing new; our current system also behaves this way. At another level, ClamAV is detecting a lot more of this than our current system does. I currently believe that one of the reasons for this is that the two systems work somewhat differently, as we have them currently set up. As we found out once we started logging the information, our current system seems to only detect phish attempts and other text-based bad things in actual attachments, not in things that are theoretically in-line parts of the message (either as MIME parts or as the whole message). The significant volume we see from our current system is because a lot of senders of this spam appear to actually send HTML files as attachments and so on, which triggers our current system running them past its phish signatures.

As we have Exim configured, we pass the whole message to ClamAV and, as far as I know, ClamAV scans the whole thing. ClamAV very clearly doesn't restrict its signature matching to MIME attachments; it seems perfectly happy to try to match signatures against message text too, which significantly increases the opportunity for signatures to match. That ClamAV will match signatures this way probably also leads to people coming up with a lot more signatures for this sort of thing; in our current system, there's obviously not as much point in preparing a signature for something you're not seeing in actual attachments, so I have to assume that the vendor mostly doesn't bother to do so.

(The pragmatic issues that this raises are out of scope for this entry. But I'm now wondering if we should extend our testing to see what ClamAV thinks about our outgoing email, on the grounds that any detected signature in it would be a bad sign, whether or not it's a false positive.)

Written on 24 October 2019.
« The DBus daemon and out of memory conditions (and systemd)
An incorrect superstition about running commands in the Bourne shell »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Oct 24 22:47:56 2019
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.