== Wishing for a remote resilient server environment (now that it's too late)

Due to [[world and local events
https://en.wikipedia.org/wiki/2019%E2%80%9320_coronavirus_pandemic]], [[we
https://support.cs.toronto.edu/]] have all abruptly been working
from home and will be for some time. The process has made me wish
for a number of differences in our server environment to make it
what I'll call more *remote resilient*, more able to cope with life
when you don't have sysadmins down the hall during the hours of
official support.

One big ticket thing I now wish for is some kind of virtual machine
host for test machines, one that can be used remotely over some
combination of SSH and a remote graphics or desktop protocol as
opposed to basically being a local desktop setup.  We're accustomed
to installing and testing things on spare physical hardware or (for
me) VMWare Workstation on my office desktop, but that's no longer
an option now that we're not in the office (and we can't do it on
our home desktops; [[our install setup ../linux/UbuntuOurInstallSystem]]
and various other aspects of our environment assume we're on the
local network, with high bandwidth and low latency). I'm sure we
could set this up on a spare server, but of course that requires
some work in the office.

(Since this is only for testing builds and software and so on, it
doesn't have to be the kind of full scale VM environment that we'd
need for production use.)

In great timing, we had our first ever Dell [[1U
https://en.wikipedia.org/wiki/Rack_unit]] server {{AB:PSU:power
supply unit}} failure on Monday, taking down our CUPS print server
until someone could come in and swap the hardware around. [[These
days I like redundant power supplies ../tech/RedundantPowerSuppliesII]]
but we tend to only consider them for very important machines, like
[[our ZFS fileservers ../linux/ZFSFileserverSetupIII]]. That's a
sensible choice for normal times when sysadmins are in the office
and we can swap hardware relatively rapidly, but these are not
normal times; having dual power supplies in anything that's a
singleton server would clearly create more remote resilience. I
don't know if there are 1U or 2U servers with dual power supplies
that are only moderately more expensive than basic 1U servers, but
if there are perhaps we should consider getting some in our next
round of hardware purchases.

(We probably can't afford to make almost all of our servers have
dual PSUs, but it certainly would be nice if we could.)

The final thing I'm rather missing is pervasive support for remote
management that goes all the way up to KVM over IP and using remote CD
(or DVD) images. We have [[a serial console server ConsoleServerSetup]],
but that only gets some things and you can't remotely install (or
reinstall) a machine through it (plus, [[our 'serial consoles' are not
the machine's real console DroppingSerialConsoles]]). Our old SunFire 1U
servers had full scale KVM over IP and it was very great, but since then
only higher end machines like [[our ZFS fileservers]] have it; our basic
Dell 1U servers don't. KVM over IP is generally an extra cost feature in
one way or another (either in the form of more expensive servers or as
an explicit license) and we've traditionally not paid that cost, but it
does cost us remote resilience in various ways.