Having non-x86 machines makes your life harder, especially in a mixed setup

June 20, 2023

Here's a confession: while I like the idea of good non-x86 CPUs and machines in theory, I don't really want to operate any of them in practice. I especially don't want to operate them in a heterogeneous environment along side x86 machines.

The basic reason to avoid non-x86 if you have a choice is that x86 is the dominant default, and today everything else is a relative minority environment. My impression is that the 64-bit ARM Linux world is reasonably common, but even there it's not common enough for Mozilla to provide a pre-compiled Firefox binary (and let's not ask about more commercial software, such as for video conferencing). Using a non-x86 system is pretty much guaranteed to be a parade of paper cuts, where binaries don't exist, people haven't tried their software, software has architecture specific bugs, and so on. Meanwhile, you can safely assume that everyone supports 64-bit x86.

(I'm not up on the state of 64-bit ARM support in other free Unixes. Here's OpenBSD and FreeBSD (also, also?).)

The reason to avoid a multi-architecture environment is that in most situations they're significantly more annoying and troublesome than single-architecture ones. You need multiple sets of binaries (often of entire packages) and generally a way to pick the right binary for the right architecture. There are ways to make the picking problem easier, but they generally involve sharing less between machines. If you try to share things across architectures, you'll get to find all of the ways that modern systems aren't really set up for that, partly because it's been decades since most environments really tried to do this.

This leads to the second reason to avoid non-x86, which is that you'll almost certainly wind up with a mixed setup where you still have some x86 machines (and perhaps a lot of them). Generally you won't be able to cordon these off into single-architecture environments that you can handle completely separately, so going non-x86 also means going mixed.

Sometimes what you gain from using non-x86 is worth the pain. But there's pretty much always going to be pain.

(This is related to how in the old days we didn't use multiple Unixes by choice, since many of the same issues apply.)

PS: This only applies when you need to actively operate the machines. If the machines are sealed black boxes that you only push buttons on, you don't really care what's inside.

Comments on this page:

Conversely my previous employer, all-I on AWS, has converted almost all its workloads to Graviton2 and 3 for the substantial cost savings. One that was delayed was PostgreSQL, because it only got arm64 SMP optimizations in version 14 and we would have to upgrade first.


That said, server ARM options are thin if you want to own your own, specially since Qualcomm threw in the towel.

Besides not being able to natively run some tricky/proprietary software without a Qemu userspace shim or resorting to a (slow, simu-/emulated) VM, writing and deploying code for services running on x86_64 and aarch64 Linux from an ARM laptop has rarely caused me any issues. Same goes for Windows. Mixed architecture infrastructure in the data center with proper boundaries is also essentially transparent at this point. Configuration management at a higher level abstracts most distro and architecture issues to irrelevancy, or at least a one-time pain, and a decent CI setup can help immensely with ferreting out build problems between architectures. It's not 1995 anymore.

When Apple announced the transition to their own silicon, my team decided to drop them. Having mixed-arch laptops would mean someone had tier-1, and someone got tier-2 support. We weren't about to throw away young x86 machines.

We do use ARM (Graviton2) for RDS, where none of it is our responsibility; and for an SFTP server, which requires no third-party binaries and has no need for a VM clone. The real money is still made on x86, though.

Written on 20 June 2023.
« Failing to draw lines between 'script' and 'program'
Domination has a lead time »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue Jun 20 23:09:02 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.