Due to world and local events, we have all abruptly been working from home and will be for some time. The process has made me wish for a number of differences in our server environment to make it what I'll call more remote resilient, more able to cope with life when you don't have sysadmins down the hall during the hours of official support.

One big ticket thing I now wish for is some kind of virtual machine host for test machines, one that can be used remotely over some combination of SSH and a remote graphics or desktop protocol as opposed to basically being a local desktop setup. We're accustomed to installing and testing things on spare physical hardware or (for me) VMWare Workstation on my office desktop, but that's no longer an option now that we're not in the office (and we can't do it on our home desktops; our install setup and various other aspects of our environment assume we're on the local network, with high bandwidth and low latency). I'm sure we could set this up on a spare server, but of course that requires some work in the office.

(Since this is only for testing builds and software and so on, it doesn't have to be the kind of full scale VM environment that we'd need for production use.)

In great timing, we had our first ever Dell 1U server PSU failure on Monday, taking down our CUPS print server until someone could come in and swap the hardware around. These days I like redundant power supplies but we tend to only consider them for very important machines, like our ZFS fileservers. That's a sensible choice for normal times when sysadmins are in the office and we can swap hardware relatively rapidly, but these are not normal times; having dual power supplies in anything that's a singleton server would clearly create more remote resilience. I don't know if there are 1U or 2U servers with dual power supplies that are only moderately more expensive than basic 1U servers, but if there are perhaps we should consider getting some in our next round of hardware purchases.

(We probably can't afford to make almost all of our servers have dual PSUs, but it certainly would be nice if we could.)

The final thing I'm rather missing is pervasive support for remote management that goes all the way up to KVM over IP and using remote CD (or DVD) images. We have a serial console server, but that only gets some things and you can't remotely install (or reinstall) a machine through it (plus, our 'serial consoles' are not the machine's real console). Our old SunFire 1U servers had full scale KVM over IP and it was very great, but since then only higher end machines like our ZFS fileservers have it; our basic Dell 1U servers don't. KVM over IP is generally an extra cost feature in one way or another (either in the form of more expensive servers or as an explicit license) and we've traditionally not paid that cost, but it does cost us remote resilience in various ways.