On right and wrong ways to harvest system-level performance stats
One of the great things about blogging, especially over a long time, is that it gives you opportunities to be spectacularly wrong in public. (I'm actually sort of serious about this being a good thing, but that's another entry.)
Back in 2012, I wrote an entry called "The wrong way to harvest system-level performance stats", in which I maintained that single-shot stats gathering of absolute counters was the wrong way and that the host agent should compute and publish ongoing deltas between measurements. This issue matters partly because it fundamentally affects how you can gather stats; if you have single-shot stats, you can collect them with simple stateless scripts that run and exit, but if you need differences, you also need continually running scripts to gather those differences.
There is no nice way to put it: I'm wrong here. System level stats should be gathered as absolute (total) values whenever possible, not as differences (deltas) over some time interval. Perhaps I was somewhat less wrong back in 2012 than I am today, but today you definitely want to gather only absolute measurements and then leave everything else up to your metrics system backend. And even in your metrics backend, you want to store the absolute values in addition to any deltas that you pre-compute for convenience and lower CPU usage.
In 2012 it was at least somewhat more likely that you would be dealing with rapidly rolling 32-bit counters and an environment where you could only collect samples relatively infrequently. Today, 64-bit counters are everywhere and more powerful systems mean that we can afford to collect stats quite frequently. Also, I think metrics system backends are clearly more powerful today than they were six years ago; today's systems on today's hardware can reasonably do quite sophisticated computations over fairly large amounts of data.
(In other words, you're no longer setting your CPUs on fire if you want to compute even fine-grained deltas with counter reset checks every time you (re)display a graph.)
The first fundamental reason for using absolute values in preference to anything else is that it's much easier to go from absolute values to differences over the time range of your choice than it is to go from differences over a specific time range to either absolute values or differences over another time range. You can do the second, but it is a bunch more work and you run into various potential precision issues. Going from absolute counter values to deltas over various times is relatively straightforward; you just need frequent enough samples and then to scan for counter resets.
The second fundamental reason is that gathering the actual stats is almost always easier. Operating systems mostly export absolute counters because absolute counters are the easiest thing to do, and then your stats gathering agent can be stateless and simply dump out the raw values from the system. You can even do this with scripts, as people were advocating (and implementing) back in 2012.
As a bonus, you wind up relatively indifferent to whether any particular stat is a counter that is supposed to always go up or a gauge that goes up and down. Interpretation of this is up to the metrics system or even to the user writing queries against your data, and they're the best placed to know what they want and what things really mean. Your stats gathering process can simply blindly read and export everything it can get its hands on.
So why have I realized that I was wrong? Simple; I got practical experience by building and using a metrics system that used a lot of absolute stats as its raw values and computed deltas from them. Prometheus is perfectly happy to do this (although there are issues), and having easy access to absolute stats has also made it easy to use them, which is sometimes the right way to present information. Now that I've used a system based around absolute stats, I've been smacked in the nose by how clearly off my previous entry is and it's time to admit it and set the record straight.