Unfortunately BMCs are black boxes that are supposed to just work, so there's no way for ordinary system administrators like me to peer inside.

True. Fortunately, we now have 'OpenBMC' [1], which may (or may not) alleviate this issue somewhat. [2] [3]

[1] https://github.com/openbmc/openbmc

[2] https://www.phoronix.com/news/NVIDIA-OpenBMC-Contributions

[3] https://www.phoronix.com/review/ampereone-a192-32x

We have a Supermicro server that had a similar problem. Whenever the server was rebooted the BMC also needed a reboot or the server would be stuck on some POST state. This continued for maybe six months, but the server has been behaving properly for a long time for some reason. Don't know why, I don't think it has received any BMC or BIOS updates since before the problems.

During the first couple times I ended up airing the power cords which handled the BMC reboot before I noticed there was easier option.

This is actually quite common in embedded devices. I've done quite a bit of troubleshooting getting to the bottom of these type of issues with on-premise devices we had.

In almost all cases I've seen in the past 10 years, almost every weird glitchiness behavior (non-deterministic) that involved embedded devices in the workplace was the result of capacity issues, with the next runner-up being EOL/hardware component failure issues.

For example, the most common failure involved DSL/Cable modem/unmanaged (some low-end managed) legacy networking appliance that was set by default to log RX/TX errors along with a substantial amount of metadata.

This would trigger and log whenever dirty electricity caused an issue at a site cyclically.

The embedded devices would in most cases continue logging (internally) until capacity was used up. In some rare cases with specific vendors, these errors can propagate across a site network.

For most devices this is only visible by physically connecting to a functioning UART/JTAG port on the problem device to get a console before lockup. It will lock up once capacity is hit.

Almost all embedded devices (with few exceptions), run a *nix variant under the hood, with tmpfs mapped to volatile memory.

Once it rolls, the glitching starts. Once you reset, the tmpfs is cleared, and the counter/logging starts over. The rate of time that this happens is often environment (site specific)-vendor implementation paired.

You see effectively the same characteristics with Redhat servers where some greenhorn didn't set up rotating logs, or didn't properly manage capacity.

The lack of electrical isolation from the PSU is problematic for most computing devices. I've come into some sites with real janky setups, like where someone tried to DIY a BMC with an RPI with a standard wall wart (+no interfaced RT clock). SMH.

I should clarify that I too tend to use terms IPMI and BMC somewhat interchangeably (imprecisely), my previous response referenced mostly embedded devices in general (including IPMI/KVMoIP embedded devices), but I include BMC (we also used Dell iDrac's which had low level issues as well; often seemingly triggered by power issues).