The problem of 'shared' IPMI network interfaces

July 29, 2020

These days, most of our servers have some form of IPMI support, including a network connection for their IPMI and any associated services (like KVM over IP). However, there is a significant variation in how this network connection is provided in hardware, and that causes some problems for actually using IPMI with less expensive servers in environments where you care about security.

The best servers have a separate physical network port that's only accessible to and used by the IPMI. The physical machine may have three network ports on the back, but the host machine (the server) is only connected to two; the third is connected only to the IPMI. This is typical for our Supermicro servers as used in, for example, our Linux ZFS fileservers.

On other servers of ours, there is no network port that's connected only to the IPMI; instead, at some level, the available network ports are shared between the IPMI and the host system. Often, the BIOS and the IPMI can operate these ports in two modes, 'dedicated' and 'shared'. In dedicated mode, the host server is entirely locked out of one port and NIC, and only has one network interface available. In shared mode, the host and the IPMI magically share a network port; traffic for the IPMI goes to it (and is theoretically invisible to the host), while traffic for the host goes to it so that the port looks like a normal network port.

(There are variations on this depending on the server.)

I don't like this 'shared' mode. The problem with it is simple; any IPMI system needs to be on a secure network and with a shared network port, the host has potential access to your theoretically secure IPMI network, with all of the other crunchy and probably dangerously insecure IPMIs on it. Sure, traffic to the server's own IPMI will be fenced away from the host, but other traffic probably won't be (partly because there are some people who put their IPMIs on the same network as their hosts).

Even the existence of a shared mode is probably dangerous to the security of your IPMI network, because on some systems this setting can be changed through IPMI itself from the host (for example, see the various 'set-nic-mode' commands in FreeIPMI's ipmi-oem). An attacker who's compromised one such server can probably switch its IPMI mode from dedicated to shared, then put the host on the IPMI network and get to work.

Only a completely dedicated IPMI network port is safe from this, because only then does the host have no access to it no matter what the BIOS and IPMI are set to or can be changed to. So I wish everything had dedicated IPMI network ports, not ones that are potentially accessible by the host. Sadly, extra network ports and NICs cost more, so they don't appear on most of the inexpensive servers we tend to buy.

(Since the host can talk to the IPMI and IPMIs can have security bugs, a fully dedicated IPMI network port doesn't make you completely safe. In theory the host can compromise the IPMI and get the IPMI to proxy network traffic for it. In practice this is not something that's likely to be a risk for most people; it rises to the level of 'the intelligence agency is going to intelligence agency' [PDF, but it's James Mickens, you should read it].)

Written on 29 July 2020.
« Our ZFS spares handling system for ZFS on Linux
Putting IPMIs on a port isolated network to deal with shared network interfaces »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Jul 29 22:48:09 2020
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.