Sometimes it's useful to have brute force handy: an amusing IPMI bug

May 8, 2015

Once upon a time we had gotten in some new servers. These servers had an IPMI and the IPMI could be configured to send out email alerts if something happened, like a fan stopping or a power supply losing power. Getting such alerts (where possible) seemed like a good idea, so I dutifully configured this in the IPMI's web interface. Sensibly, the IPMI needed me to set the origin address for the email, so I set it to sm-ipmi@<us> (and then made sure there was an sm-ipmi alias, so our mail submission machine would accept the email).

Of course, configurations are never quite done until they're tested. So I poked the IPMI to send me some test email. No email arrived. When I went off to our mail submission machine to look at its logs, I got rather a surprise; the logs said the machine had dutifully rejected a message that claimed a MAIL FROM address of =sm-ipmi@<us>.

While the insides of an IPMI's embedded software are inscrutable (at least to lazy sysadmins who are not security researchers), this smells like some form of classic data storage mismatch bug. The web interface thinks the email address should be stored with an '=' in front, maybe as an 'X=Y' thing, whereas whatever is actually using the address either has an off by one character parsing bug or doesn't want the extra leading = that the web interface is adding when it stores it.

There are probably a bunch of ways we could have dealt with this. As it happens our mail system is flexible enough to let us do the brute force approach: we just defined an alias called '=sm-ipmi'. Our mail system is willing to accept an '=' in local parts, even at the start, so that's all it took to make everything happy. It looks a little bit peculiar in the actual email messages, but that's just a detail.

A more picky email system would have given us more heartburn here. In a way we got quite lucky that none of the many levels of checks and guards we have choked on this. Our alias generation system was willing to see '=' as a valid character, even at the start; the basic syntax checks we do on MAIL FROM didn't block a = at the start; Exim itself accepts such a MAIL FROM local part and can successfully match it against things. I've used mail systems in the past that were much more strict about this sort of stuff and they'd almost certainly have rejected such an address out of hand or at least given us a lot of trouble over it.

(I don't even know if such an address is RFC compliant.)

The whole situation amuses me. The IPMI has a crazy, silly bug that should never have slipped through development and testing, and we're dealing with it by basically ignoring it. We can do that because our mail system is itself willing to accept a rather crazy local part as actually existing and being valid, which is kind of impressive considering how many different moving parts are involved.

PS: I call this the brute force solution because 'make an alias with a funny character in it' is more brute force than, say, 'figure out how to use sender address rewriting to strip the leading = that the IPMI is erroneously putting in there'.

PPS: Of course, some day maybe we'll update the IPMI firmware and suddenly find the notification mail not going through because the IPMI developers noticed the bug and fixed it. I suppose I should add the 'sm-ipmi' alias back in, just in case.

Written on 08 May 2015.
« Why keeping output to 80 columns (or less) is still sensible
What addresses we accept and reject during mail submission »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri May 8 02:18:46 2015
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.