Understanding how fast Ethernet really is (and in what units)
I've been around computer networking for a long time, but in all of that time I've never fully understood what the speed numbers of the various Ethernet standards actually meant. I knew what sort of bandwidth performance I could expect from '1 Gigabit' Ethernet as measured on TCP streams, but I didn't know exactly what '1G' meant, beyond that it was measured in bits (for example, was it powers of ten G or powers of two G). My quiet confusion was likely increased by the numbers that the Prometheus host agent reports, where a Gigabit Ethernet interface is reported as '125,000,000'.
('1G' in powers of ten decimal is 1,000,000,000. I'm writing it out with commas because otherwise comparing the two numbers by eye is painful.)
The Wikipedia page on Gigabit Ethernet starts out a bit confusing (for people like me) by talking about how the physical link generally uses '8b/10b encoding', which inflates the line rate by 25% over the data rate. However, what is '1G' about 1G Ethernet is the data rate, not the raw on the wire rate. That data rate is '1000 Mbit/s' (per the Wikipedia page), which tells us that we're dealing with powers of ten bit rates. When you divide that 1000 Mbit/s of data by 8 (to convert to bytes), you get 125 (decimal) Mbytes/s, and the network interface rate reported by the Prometheus host agent now makes sense.
(As a practical matter this means that the 8b/10b encoding is something we ignore. It's a coincidence that the line rate of '1250 Mbit/s' looks similar to the '125 Mbytes/s' data rate.)
If we then convert from powers of ten to powers of two, we would expect a theoretical bandwidth of around 119 MiBytes/sec (sort of using the official binary prefix notation). We're not going to get that bandwidth at the TCP data level, because this doesn't account for the overhead of either Ethernet framing or TCP itself.
I'm not as clear about how 10G Ethernet is encoded on the wire, but it doesn't matter for this because what we care about is still the data rate. 10G Ethernet has a data rate of 10,000 Mbit/s ('10 Gbit/s' in Wikipedia's list of interface bit rates), which translates to 1,250 Mbytes/sec (power of ten). This theoretically maps to 1192 MiBytes/s (ten times 1G Ethernet), but you're never going to get that bandwidth for TCP data flows because of the assorted overheads.
(This of course explains the Prometheus host agent's report that the 10G network speed is '1,250,000,000', and similarly that the 100 Mbit speed is '12,500,000'. All of which are much clearer when written out with commas for separation, or whatever character is used for this in your locale.)

