Chris's Wiki :: blog/sysadmin/VirtualizationFundingProblem

Seriously using virtualization clashes with our funding model

November 9, 2020

In a comment on my entry on us needing to get experience with Ubuntu 20.04, Miksa pointed out the advantages of investing heavily in virtualization. These definitely exist, but among other practical issues, heavy investment in virtualization clashes with our funding model, which you could describe as 'erratic' or 'nonexistent'. Our equipment funding is quite erratic and as a result we have to buy stuff mostly in bursts.

A significant commitment to and investment in virtualization really calls for big servers, ones that have a lot of memory and CPUs to support a lot of virtual machines at once (and possibly a lot of storage, or a very fast storage network). Big servers are expensive and you have to spend that money all at once, both the initial purchase and their replacement when they reach their (finite) sensible operating lifetime. Periodic big investments on big VM host servers may well cost less overall than a series of smaller but more spread out investments on physical servers, but they require you to have large sums of money available at predictable times. Unfortunately, we don't.

(There's an alternate model of lots of smaller virtualization hosts with fewer VMs on each host, but it's not what people talk about or recommend.)

This is the same effect I've seen before with network routers and switches, where buying a highly capable but expensive router doesn't make sense for us. With routers and switches we're better off with a bunch of smaller switches than trying to do everything in a few big units, because we can buy and replace them piece by piece. The same is true for us with virtualization servers, at least in the conventional model of them. Our funding model is mostly incompatible with periodic big purchases.

(We've been surprisingly good at turning over our fileserver infrastructure more or less on schedule, but that has traditionally had a natural forcing function because hard drives wear out. Even then we pushed one fileserver generation alarmingly far, to the point where we saw significant drive failure rates.)

PS: This funding issue isn't solved by using someone else's host infrastructure so that we pay this as an ongoing smaller operating cost instead of a periodic hardware purchase cost. We don't have reliable money for cloud operating costs either, and that can be a worse problem.

Comments on this page:

By Mark C at 2020-11-10 07:27:02:

Am I the only one that constantly reads different stories about virtualise everything/none/something and for infrastructure - hybrid/only-cloud/on-prem-only. Finally all these are applicable for different circumstances (as you say). Sometimes all these are needed concurrently. Say the economics department is OK with a huge Samba share (local/cloud - do not care), whereas the Physics department wants everything local to analyze million images.

Not sure about Canada's IT/Uni payscales, my view is that this is a recruitment problem - funding of these skilled IT people with long-term contracts in EU (whether it is HPC stuff or ZFS or virtualisation or hybrid) - at a (low) payscale of Universities (not easy). Otherwise one gets a mismash of whatever a short-term employee implements. If there are long-term employees then one can reasonably optimise costs but...

(Love your posts)

By Ben Cotton at 2020-11-10 08:15:00:

The counterproductivity of the funding model is one of the things I miss the least about working in academia. Our problem was that it was easy for faculty to buy hardware on grants and very difficult to buy services. For faculty without grant funding and for department operations in general, we relied a lot on hand-me-downs from other departments. The result was a lot of old and wildly-different hardware, which increased the support burden.

By Blissex at 2020-11-13 07:24:16:

«funding model, which you could describe as 'erratic' or 'nonexistent'. Our equipment funding is quite erratic and as a result we have to buy stuff mostly in bursts.»

I have seen really famous (much more so than the already excellent UofT) top-tier, "rich", academic institutions where the budget for "business as usual" IT operations was by deliberate choice zero (except for a bit of "caretaker" manpower), and everything was funded by development projects tied to specific needs, and funding terminated at the end of the project.

This was driven by eminent professors in top level resource planning committees whose experience was that every academic only gets funding by bidding for specific research projects, never for merely existing. So they applied that model to persistent IT infrastructure that required continuity as it was used by multiple time-overlapping academic projects.

Written on 09 November 2020.

«	Getting the git tags that are before and after a commit (in simple cases)

Logging fatal exceptions in my Python programs is not enough