Putting IPMIs on a port isolated network to deal with shared network interfaces
Yesterday I wrote about the problem of 'shared' IPMI network interfaces, which is that when the host and the IPMI both have access to the same physical network port, you're exposed to a compromised host putting itself on your secure IPMI network and compromising other IPMIs (and then hosts) over it. There was a discussion about this on lobste.rs, where lobste.rs user sn made an excellent suggestion:
The L2 feature you are looking for is called a protected port. This should be available on any managed switch, but I’ll link to the cisco documentation: [link]
('Protected ports' are what I know as port isolation, where hosts connected to these isolated ports can only talk to designated 'uplink' ports, not with each other.)
This is a great suggestion and a great idea. Generally your IPMIs don't need to communicate with each other, they only need to communicate with upstream machines that monitor them, collect syslog messages, connect to them to manage servers, and so on. If you put all of the ports used for IPMIs on a port isolated network (or on a port isolated subset of switches), a compromised server can't bring up the host side of a shared IPMI network interface to talk to the other IPMIs; it can only talk to the upstream servers, which are hopefully a lot more secure than the IPMIs (which often aren't).
If we were to design a new IPMI network from scratch, I would at least suggest this and see if my co-workers could spot a reason it's a bad idea in our setup. Our current IPMI network drifted into that role (which is a story all in its own right), so it's an ordinary 'sandbox' private network without port isolation; we probably don't want to go back to revise it to be port isolated, especially these days.