Wandering Thoughts archives


What tool you use to read IPMI sensor information can matter

Most modern servers have some sort of onboard IPMI (sometimes called a service processor), and there are good reasons to want to read the IPMI's sensors instead of relying purely on whatever sensor readings you can get from the Linux kernel. There are a number of open source projects for reading IPMI information on Linux (and sometimes other systems); I have direct experience with ipmitool and FreeIPMI, and the Arch wiki lists a few more.

Until very recently I would have told you that these tools were more or less equivalent for reading IPMI sensors and you could use whichever one you wanted to (or whichever had the more convenient output format). Locally we've used ipmitool for this, and generated Prometheus metrics by parsing its output. Except that we haven't been able to this on all of our machines, because on some of our models of Dell servers, reading the IPMI sensors is achingly slow. I was all set to write a grumpy entry about the need for reading IPMI sensors to be fast in order to be really useful when I decided to test FreeIPMI to see if it would give me better diagnostics about why reading IPMI sensors on these Dell servers was so slow.

To my surprise, FreeIPMI's ipmi-sensors was able to read IPMI sensors with blazing speed on these servers. This blazing speed is partly the result of building a cache of sensor data records (SDRs), but even when ipmi-sensors has to build its cache it runs several times faster than ipmitool (on the Dell server models that are a problem for ipmitool). Even on Dell server models that ipmitool runs decently fast, ipmi-sensors seems faster.

I have no idea what the two projects are doing differently when they read IPMI sensors, but there clearly is a real difference in FreeIPMI's favor, contrary to what I expected. My minimum learning experience from this is that if I run into IPMI problems when I'm using one project, such as ipmitool, I should try other projects just to see. I'm also probably somewhat too cynical about the quality of implementation of IPMIs, at least from major server companies; Dell probably wouldn't ship an IPMI that really took five minutes to read out all of its sensors.

I've got a lot of ingrained habits around using ipmitool, along with some scripting. I'm probably going to switch at least the scripting to use FreeIPMI (which means installing and testing it on more servers here). I may not switch my habits, especially right away, since they're pretty ingrained and ipmitool is convenient in how everything is right there in one program and one manpage.

PS: I was lucky by already having some familiarity with FreeIPMI, since I'd used its ipmi-oem program to set some OEM-specific options in the IPMIs for some servers here.

linux/IPMISensorsProgramsMatter written at 23:36:07; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.