Sometimes it's useful to have brute force handy: an amusing IPMI bug
Once upon a time we had gotten in some new servers. These servers
had an IPMI and the IPMI could be configured to send out email
alerts if something happened, like a fan stopping or a power supply
losing power. Getting such alerts (where possible) seemed like a
good idea, so I dutifully configured this in the IPMI's web interface.
Sensibly, the IPMI needed me to set the origin address for the
email, so I set it to sm-ipmi@<us>
(and then made sure there was
an sm-ipmi
alias, so our mail submission machine would accept the
email).
Of course, configurations are never quite done until they're tested.
So I poked the IPMI to send me some test email. No email arrived.
When I went off to our mail submission machine to look at its logs,
I got rather a surprise; the logs said the machine had dutifully
rejected a message that claimed a MAIL FROM
address of
=sm-ipmi@<us>
.
While the insides of an IPMI's embedded software are inscrutable
(at least to lazy sysadmins who are not security researchers), this
smells like some form of classic data storage mismatch bug. The web
interface thinks the email address should be stored with an '=' in
front, maybe as an 'X=Y
' thing, whereas whatever is actually using
the address either has an off by one character parsing bug or doesn't
want the extra leading =
that the web interface is adding when
it stores it.
There are probably a bunch of ways we could have dealt with this.
As it happens our mail system is flexible enough to let us do the
brute force approach: we just defined an alias called '=sm-ipmi
'.
Our mail system is willing to accept an '=' in local parts, even
at the start, so that's all it took to make everything happy. It
looks a little bit peculiar in the actual email messages, but that's
just a detail.
A more picky email system would have given us more heartburn here.
In a way we got quite lucky that none of the many levels of checks
and guards we have choked on this. Our alias generation system was
willing to see '=' as a valid character, even at the start; the
basic syntax checks we do on MAIL FROM
didn't block a = at the
start; Exim itself accepts such a MAIL FROM
local part and can
successfully match it against things. I've used mail systems in the
past that were much more strict about this sort of stuff and they'd
almost certainly have rejected such an address out of hand or at
least given us a lot of trouble over it.
(I don't even know if such an address is RFC compliant.)
The whole situation amuses me. The IPMI has a crazy, silly bug that should never have slipped through development and testing, and we're dealing with it by basically ignoring it. We can do that because our mail system is itself willing to accept a rather crazy local part as actually existing and being valid, which is kind of impressive considering how many different moving parts are involved.
PS: I call this the brute force solution because 'make an alias with a funny character in it' is more brute force than, say, 'figure out how to use sender address rewriting to strip the leading = that the IPMI is erroneously putting in there'.
PPS: Of course, some day maybe we'll update the IPMI firmware and
suddenly find the notification mail not going through because the
IPMI developers noticed the bug and fixed it. I suppose I should
add the 'sm-ipmi
' alias back in, just in case.
|
|