Wandering Thoughts archives

What sort of server it takes to build Firefox in four and a bit minutes

I tweeted:

Our new compute server builds Firefox Nightly from source in 4 minutes 22 seconds, as well it should given how beefy it is. The build process does manage to get the machine to 100% usage of all CPUs, which is impressive (and nice) since there are 112 of them.

Then in response to a question about the link phase's duration and parallelism:

The usage of all CPUs lasts about two minutes, with a ramp up at the start and down at the end, then some wiggly usage of a lot fewer CPUs, with a couple of periods where only three are being used (and up to 10-13 in use). I can't see a clear link phase in the build report.

(This is for a full from scratch build.)

This is a (very) big server but in some respects not an unusual one; subject to component availability (and money), anyone could get their own version relatively trivially. It has two AMD Epyc 7453 28-core processors (hence the odd number of CPUs), 512 GB of RAM, and two NVMe drives in a software RAID mirror.

(Or instead of buying one, you could probably rent time on an equivalent server from a cloud provider on a minute by minute basis.)

As officially reported in Ubuntu 18.04, the server has only two NUMA zones:

node distances:
node   0   1 
  0:  10  32 
  1:  32  10 

I'm not sure this is completely accurate for Zen 3 Epycs in practice, but it's the official view.

One interesting effect of the NVMe drives (and perhaps other things) is that a build with a cold disk cache for the Firefox Nightly source tree is only a few seconds slower than a build with a warm cache. However, the NVMe drives do slow down enough under load that despite Linux software RAID preferring to read from the first disk, it sometimes reads from the second drive, although nowhere as much. This has the interesting effect that the second NVMe drive reports a lower average IO time than the first one.

(All of this is captured with a certain degree of granularity through our Prometheus based metrics system.)

There was a day when a system like this would have been essentially impossible for us to have; something this big and powerful would have been far too expensive for us to afford (and also physically very large and probably required special power). I rather enjoy how the world has changed to bring more and more computational power into our hands.