Wandering Thoughts archives

A serious potential danger with Exim host lists in ACLs

Suppose that you have an Exim installation and you want to support some sort of source host based blocking (selective or otherwise) of incoming connections. The obvious way is to create an ACL section that looks something like this:

deny
    domains = +local_domains
    hosts   = ${if exists {UBLOCKDIR/hosts} {UBLOCKDIR/hosts}}
    message = mail from host $sender_host_address not accepted by <$local_part@$domain>.
    log_message = blocked by personal hosts blacklist.

(This one is a selective, per destination address host block list, hence the fun and games with UBLOCKDIR.)

This looks great and generally works but you've just armed a ticking time bomb, one that can blow your incoming email up with permanent temporary deferrals. The first problem is that Exim has no way in a host list to say 'this domain and any of its subdomains', in the way that the TCP wrappers '.host.com' will match both 'host.com' and 'fred.host.com'. If you want to match this case, the obvious way is to write two entries:

*.host.com
host.com

The first matches any subdomains of the domain; the second matches the domain itself. But you've just put the fuse in the bomb, because of just how plain host and domain names work in host lists. From the Exim specification with the emphasis being mine:

• If the pattern is a plain domain name [...] Exim calls the operating system function to find the associated IP address(es). [...]

  If there is a temporary problem (such as a DNS timeout) with the host name lookup, a temporary error occurs. For example, if the list is being used in an ACL condition, the ACL gives a "defer" response, usually leading to a temporary SMTP error code.

So here's what happens. You list '*.spammer.com' and 'spammer.com' in your blocklist. Spammer.com turns off their DNS (or their DNS server turns it off because hey, they're a spammer) but doesn't de-register their domain, so DNS queries to their nominal authoritative DNS servers either don't get answers or get non-authoritative 'look elsewhere' results. Although this is a permanent condition, it's considered a temporary failure in DNS resolution. Exim now defers all SMTP connections that consult this host blocklist, regardless of where they are from. For ever, or at least until you notice.

Now that I've read the Exim documentation in detail, it spells out that you can turn this behavior off with the special option +ignore_defer. You probably want to do this. Certainly we do.

My feeling is that you want to do this for every host list anywhere except ones used for real, strong access control (which probably don't want to be using DNS names anyways). Consider, for example, a host list used for exceptions to greylisting; you probably don't want that ACL to defer the connection if you can't resolve a domain in it.

Sidebar: the other surprise in Exim host lists

Suppose that you have a host list like this:

*.spammer.com

192.168.0.0/16

Surprise: any connection from a host in 192.168/16 that does not have valid reverse DNS will not match the list. The moment you list a hostname wildcard in a host list, any IP address without a hostname automatically fails to match that entry or anything later in the list (or file if the list is in a file). It will match IP address patterns that are earlier in the list, though, so you get to remember to list all IP address patterns first. This behavior is documented if you read the documentation carefully.

Per the fine documentation this behavior can be turned off with +ignore_unknown. Now that I've found this, I need to make some configuration changes.

This is generally less dangerous than the host list defer time bomb, but it depends on what you're using the host list for. If you have a locked down configuration where you're using the host list for strong access control, well, you have potential issues here.

A little habit of our documentation: how we write logins

Ove the years, we've developed a number of local conventions for our local documentation. One of them is that we always write Unix logins with < and > around them, as if they were local email addresses, so that we'll talk about how <cks>'s processes had to be terminated or whatever. When I started here this struck me as vaguely goofy; over time it has rather grown on me and I now think it's a quite clever idea.

Writing logins this way does two things. The first is that they become completely unambiguous. This is not much of an issue with a login like 'cks', but we have any number of logins that are (or could be) people's first or last names, and vice versa. Consistently writing the login with <> around it removes that ambiguity and uncertainty. The second thing it does is that it makes it much easier to search for a particular login in old messages and documentation. Searching for 'chris' may get all sorts of hits that are not actually talking about the login chris; searching for '<chris>' narrows that down a lot.

(Well, sort of. The reality is that we sometimes wind up quoting various sorts of system messages and system logs in our messages and of course these messages generally don't use the '<login>' form. However, often excluding these messages from a later search is good enough because we're mostly interested in the record of active things we did to an account.)

There's a corollary to the convenience of <login>: right now we have no similar notation convention for Unix groups. We write less about Unix groups than about Unix logins (and groups generally have more distinct names), but it would still be nice to have some convention so we could do unambiguous searches and so on.

Disk IO is what shatters the VM illusion for me right now

I use VMs on my office workstation as a far more convenient substitute for real hardware. In theory I could assemble a physical test machine or a group of them, hook them all up, install things on them, and so on; in practice I virtualize all of that. This means that what I want is the illusion of separate machines and for the most part that's what I get.

However, there's one area where the illusion breaks down and exposes that all of these machines are really just programs on my workstation, and that's disk IO. Because everything is on spinning rust right now (and worse, most of it is on a common set of spinning rust), disk IO in a VM has a clear and visible impact on me trying to do things on my workstation (and vice versa but I generally don't care as much about that). Unfortunately doing things like (re)installing operating systems and performing package updates do a lot of disk IO, often random disk IO.

(In practice neither RAM nor CPU usage break the illusion, partly because I have a lot of both in practice and VMs don't claim all that much of either. It also helps that the RAM is essentially precommitted the moment I start a VM.)

The practical effect is that I generally have to restrict myself to one disk IO intensive thing at once, regardless of where it's happening. This is not exactly a fatal problem, but it is both irritating and a definite crack in the otherwise pretty good illusion that those VMs are separate machines.

(The illusion is increased because I don't interact with them with their nominal 'hardware' console, I do basically everything by ssh'ing in to them. This always seems a little bit Ouroboros-recursive, especially since they have an independent network presence.)

Virtual disks should be treated as 4k 'Advanced Format' drives

Here's something that's potentially very important, as it was for me today: if you're using an ordinary basic virtualization system (where you have guest OSes on top of a regular host OS), your virtual disks almost certainly have the performance characteristics of 4K physical sector size disks.

(In some situations they may have even bigger effective physical sector sizes.)

This happens because in a standard basic virtualization system, the guest OS disks are just files in a host OS filesystem and that host OS filesystem almost certainly has at least a 4 Kbyte basic block size. Sure the files are byte-addressable, but writing less than 4 Kb or writing things not aligned on 4Kb boundaries means that the host filesystem will generally have to do a read modify write cycle to actually write the guest's data out to its disk file (and then later do unaligned reads to get it back). Depending on how the virtualization system is implemented this can also require a whole bunch more memory copies as the VM hypervisor re-blocks and de-blocks data flowing between it and the host filesystem instead of just handing nice aligned 4 Kb pages off to the filesystem, where they will flow straight through to the hardware disk driver.

Under a lot of circumstances this won't actually matter. Many (guest) filesystems have a 4 Kb or bigger basic block size themselves and issue aligned IO in general, regardless of what they think the disk's (physical) block size is; if this is the case, the IO to the disk files in the host filesystem will generally wind up being in aligned 4 Kb blocks anyways. But if you have a guest OS filesystem that does not necessarily issue aligned writes, well, then this can make a real difference. ZFS is such a filesystem; if it thinks it's dealing with 512 byte sector disks it will issue all sorts of unaligned writes and reads.

The punchline to this is that today I doubled the streaming write and read IO speeds for my ZFS on Linux test VM with a simple configuration change (basically telling it that it was dealing with 4K disks). The IO speeds went from kind of uninspiring to almost equal to ext4 on the same (virtual) disk.

(My Illumos/OmniOS test VM also had its ZFS IO speed jump, although not as much; it was faster to start with for some reason.)