Sorting out IPMI and BMC terminology and technology

April 2, 2022

In the past I have tended to use the term 'IPMI' for several different things, in a somewhat confusing and imprecise way (much like how I've been imprecise about terms like 'SSD' and 'NVMe'). For various reasons I want to straighten that out in my head, so I'm going to write up my understanding of the whole area.

A BMC is a "Baseboard management controller", an additional server embedded in your real server (or server class motherboard). The BMC is alive and booted up any time the power is being provided to the physical server, regardless of whether the host system is powered on or not, and can manage and control various aspects of the host system's operation, such as whether the host system gets to be powered on. IPMI is the "Intelligent Platform Management Interface", a set of standards for some things that BMCs do and how to talk to BMCs. BMCs are not limited to only doing things covered in the IPMI standards, or being talked to only over IPMI interfaces. It's not uncommon for me and other people to say that a server has an 'IPMI' or is 'IPMI capable' or the like when we really mean that it has a BMC (and the BMC supports IPMI). An example of this is my entry on "the clock in your server's IPMI drifts over time".

BMCs generally have a network connection, for remote management (via IPMI and otherwise). This network connection may be on a dedicated port or on a port shared with the host system; I consider shared IPMI network interfaces to be a problem (although you can help with a port isolated network). BMCs also commonly have a variety of system health sensors for temperature, fan RPM, power usage, and so on; these sensors are often not available to the host system to read directly. More advanced BMCs can present virtual USB devices to the system and can snoop on the system's video output, which allows them to offer "KVM over IP" and virtual media. where you can interact with the host system remotely as if you were sitting in front of it with a physical monitor and keyboard.

(KVM over IP support in the BMC is not part of the IPMI standards and is sometimes an extra-cost item from server vendors.)

There are broadly standardized host system interfaces to the host's BMC for IPMI, such as SMIC or IPMB over I2C. The Linux kernel has some documentation on this, on general IPMI and IPMB, and also see OpenBSD's ipmi(4), which has nice short descriptions of several of them. If your operating system supports these interfaces, you can use general user level tools to talk to most BMCs from the host itself, which is handy because the BMC doesn't need to be connected to the network and configured. This will let you configure (or de-configure) the BMC's networking, and also read those BMC sensors so you can monitor them.

(Common open source packages for this are ipmitool and FreeIPMI. Apparently OpenIPMI is no longer being maintained, but at one time it was another option.)

Part of the IPMI standards is a set of network (remote) access protocols that can let you read sensors, control the host system power, check the IPMI "system event log", connect to virtualized host serial ports ("serial over LAN"), and often do other things. BMCs are not at all limited to only talking to people with these IPMI network protocols; instead, they commonly also have web servers, respond to SNMP, often have a SSH interface, and more. None of these additional connection methods are at all standardized, and as we've seen the BMC may expose additional information through them that isn't available via IPMI. Since KVM over IP isn't part of IPMI, all BMCs that do KVM over IP provide it through some of these additional connection methods (often through their web server). BMCs that provide SSH access generally don't give you a Unix shell but instead give you some sort of captive environment. There is no particular cross vendor standardization in how you use these captive SSH environments; expect to have to learn a new set of commands for each different vendor's BMC.

(Dell BMC SSH commands look nothing like Supermicro BMC SSH commands, for example. This can be irritating when you want to do the same thing, like connect to a virtualized "serial over LAN" serial port by SSH'ing to the BMC.)

Part of general, standardized IPMI communication is the ability to tunnel "raw" messages back and forth (both locally and I believe over the network). These raw messages can be used to implement vendor specific additional BMC functionality, both to expose additional information and to configure additional things (including semi-standard things that aren't covered by IPMI, like whether or not the BMC should run a web server or a SSH server). For some of what can be done, see the manual page for FreeIPMI's ipmi-oem. Vendors naturally don't generally document their commands, but some of them have been reverse engineered or otherwise derived. Vendor tools don't have to use IPMI raw commands to talk to their BMCs, especially over the network, but it's often the easiest way to implement vendor specific local commands.

The IPMI standards have the concept of users with passwords and access levels, although what access level is allowed to do what can be vendor specific. Where BMCs have additional network access methods, such as web servers or SSH servers, it's common to reuse the IPMI users and their passwords for these accesses. However, I don't believe it's required; in theory a BMC could have completely separate IPMI authentication and web server authentication. Even if the user names are shared, BMCs may allow you to set which users are allowed to access what, so you could create IPMI-only users and web only users.

Many BMCs can be configured through the host system's BIOS. This is convenient to assign the BMC its IP address during initial system setup and perhaps to configure your standard BMC user name and access password. Once upon a time BMCs came with pre-set and widely known default usernames and initial passwords (although they were vendor specific). Recent legislation in various places has forced system vendors to move to per-server initial passwords.

BMCs are most often little computers running some ancient version of (embedded) Linux with equally ancient and limited versions of SSH daemons, web servers (and TLS support), and so on. This means that they have security vulnerabilities, and also means that if you leave them up for too long they may have issues. The completely reliable way to force a BMC reboot is to physically unplug the power cables from the system and then let it sit for a minute or two. BMCs sometimes offer a way to reboot the BMC through a software command. Historically this has often also rebooted the host server, so we tend not to use it or trust it. If we want the BMC to reboot, we pull the power.

(It follows that you never, ever want to expose your BMC to the Internet, or in general to any general use network. Gaining full access to a BMC often allows total compromise of the host system.)


Comments on this page:

From 193.219.181.219 at 2022-04-03 03:32:29:

It's not uncommon for me and other people to say that a server has an 'IPMI' or is 'IPMI capable' or the like when we really mean that it has a BMC (and the BMC supports IPMI)

This is probably a SuperMicro thing; they emphasize in various documentation that their systems have an "IPMI 2.0 BMC" a whole lot.

BMCs sometimes offer a way to reboot the BMC through a software command. Historically this has often also rebooted the host server,

That's something I used to worry about too, but I don't think I've seen that happen in practice, neither with HP nor Dell – the host has always survived controller resets, firmware upgrades, and so on. (Admittedly the oldest system I've tried this on was DRAC4, so I don't know how far the 'historically' goes.)

By Albert at 2022-04-03 08:37:35:

I'd also mention redfish, which is now reasonably popular and available from most vendors.

By Zev at 2022-04-05 05:18:22:

I'd also mention redfish, which is now reasonably popular and available from most vendors.

...and provides yet another opportunity for a separate user database! (Yes, some vendors have entirely different sets of accounts for Redfish HTTP and web-UI HTTP. Don't ask me why.)

the host has always survived controller resets, firmware upgrades, and so on. (Admittedly the oldest system I've tried this on was DRAC4, so I don't know how far the 'historically' goes.)

While the host being unaffected by BMC resets is certainly what's supposed to happen, in practice (as noted in the article) BMC firmware is frequently buggy and prone to misbehavior. While I haven't recently encountered that exact pathology, just last week I discovered a fairly new system whose BMC's boot process would unconditionally trigger a host power-on event, so it was impossible for the host to remain off across a BMC reset.

The OpenBMC project (disclaimer: I work on it) is one effort to address the inconsistent and fairly sorry state of BMC firmware, though unfortunately at present the odds of being able to run it on an arbitrary piece of off-the-shelf hardware are fairly low.

Written on 02 April 2022.
« Some notes on finding and reading information over SNMP
Some notes on using snmpwalk to poke at devices with SNMP »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sat Apr 2 22:55:17 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.