Remote power control for your machines comes in two flavours

September 27, 2020

In yesterday's entry on our trouble free remote reboot of our machines, I mentioned that remote power control for all of our machines would be nice. When I said that, I was insufficiently specific, because there are actually two forms of remote power control and we mostly have one of them, but it's not always good enough. I will call these two sorts of remote power control external and internal remote power control.

In external remote power control, your servers and machines are plugged in to a smart power bar or smart rack PDU that you can remotely control to turn outlets on or off. In internal remote power control, the machine itself has some form of lights out management that will control the machine's power; for instance, power control over (networked) IPMI, or a serial connection to the management process. External remote power control is much easier to set up and is what we have, but it's not quite as good as internal remote power control overall.

The largest limitation of external remote power control is that it leaves you vulnerable to BIOSes with undesirable power control settings. If you have a machine with its BIOS set to 'when power returns after a power loss, stay turned off', then you can't force reboot the machine with external remote power control; once you turn the power to it off, it won't come back up again until you go in to push its physical power button. The machine has power, but the BIOS has been told to keep it off until you push that button.

(Plain KVM over IP will usually not get you out of this, because the BIOS is not powering up the keyboard and video. Wake on LAN probably would, but if your BIOS is set to keep the power off, you probably didn't set WoL up either.)

Plain external remote power control also doesn't do anything if you have the machine plugged into a UPS (either completely, for all of its power supplies, or partially, with one redundant PSU plugged into a UPS and the other into your smart PDU). If you can remote control outlets on the UPS, you can deal with this, but I'm not sure how many UPSes (even rack ones) support this. I don't believe ours do.

(Our fileservers have dual PSUs with one on a UPS and one on line power this way.)

Internal remote power control deals with both issues because it can directly control the machine's PSUs and will simulate you pushing that front panel power button to overcome BIOS settings. The drawback of having only internal remote power control is that you can't completely cut power to the machine, including to its lights out management processor (you can generally tell the LOM to reboot, but not to turn itself off). So the best of both worlds is to have both smart PDUs and lights out management on your machines.

Comments on this page:

By Andrew at 2020-09-27 19:18:03:

Most "smart" UPSes in my experience do support turning the load on and off, although it may be one big switch, not port-by-port control. Of course, for the "on" option to be useful you need to have the UPS's USB hooked up to a device that doesn't rely on the UPS for power, like a Pi with an independent battery backup.

By dozzie at 2020-09-27 20:28:55:

The drawback of having only internal remote power control is that you can't completely cut power to the machine [...] So the best of both worlds is to have both smart PDUs and lights out management on your machines.

Indeed. I'll add also BMC/IPMI losing its settings, hanging up, or otherwise acting faulty (I had over 3k machines from HP, Dell, and Lenovo under my care and uncooperative IPMI was a major headache for mass OS reinstallations). You definitely want to have control also over external power supply.

Written on 27 September 2020.
« We rebooted all of our servers remotely (more or less) and it all worked
Looking at DKIM information for our 'good' email (September 2020 edition) »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Sep 27 16:18:57 2020
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.