Chris's Wiki :: blog/sysadmin/StatsGatheringNeedsDeltas Commentshttps://utcc.utoronto.ca/~cks/space/blog/sysadmin/StatsGatheringNeedsDeltas?atomcommentsDWiki2012-09-28T12:55:44ZRecent comments in Chris's Wiki :: blog/sysadmin/StatsGatheringNeedsDeltas.By Chris Siebenmann on /blog/sysadmin/StatsGatheringNeedsDeltastag:CSpace:blog/sysadmin/StatsGatheringNeedsDeltas:7c24c9ce12d7b21d69022883ed9626cbdd60fcc8Chris Siebenmann<div class="wikitext"><p>The raw counters come from the operating system, limitations included. I
entirely agree with you about 32-bit counters not being a good idea, but
I've still seen OSes and drivers that did that (if I remember right, it
was driver specific).</p>
</div>2012-09-28T12:55:44ZFrom 124.169.157.68 on /blog/sysadmin/StatsGatheringNeedsDeltastag:CSpace:blog/sysadmin/StatsGatheringNeedsDeltas:eae5ea3b61172c60c7179ebfa18f8960e3da3b6eFrom 124.169.157.68<div class="wikitext"><p>Rollovers are easy enough to deal with, it's the double rollover that gets you.</p>
<p>Still, if you are counting the bits transferred on a heavily loaded gigabit interface, you'd realise pretty soon that you can't realistically use a 32bit counter.</p>
<p>I agree with the other commentators that deltas are best dealt with outside of the monitoring system.</p>
<p>Monitoring and collection must be as lightweight as possible so it adds as little load as possible with performance of the underlying system. This is why <a href="http://dtrace.org/blogs/about/">DTrace</a> in Solaris 10 is so awesome. Kernel level instrumentation for everything at next to 0.0% load.</p>
<p>Francis</p>
</div>2012-09-28T05:43:13ZBy Chris Siebenmann on /blog/sysadmin/StatsGatheringNeedsDeltastag:CSpace:blog/sysadmin/StatsGatheringNeedsDeltas:592ccee61a16620ee46d3774aa26da9b8f6d0396Chris Siebenmann<div class="wikitext"><p>You make a fair point about the timestamps; I don't actually know how
common tools like Graphite and so on deal with timestamps. It may well
be that everyone already receives and stores timestamp plus measure
pairs and this is a non-issue.</p>
<p>I think that host agents are the right place to deal with counter
rollover for two reasons. First, the host agent is the natural place to
put measure-specific information about what counter rollovers and other
glitches actually look like. Second, if the backend can assume that the
measure data is rollover-free, any delta computations are much simpler.
It can simply take the start and end points, subtract one from the other
(for both the delta time and the change in the measure), and be done. If
the backend handles rollovers, all delta computations must scan all of
the intermediate data points in order to spot and fix up any rollovers.</p>
</div>2012-09-27T16:37:51ZFrom 96.228.233.77 on /blog/sysadmin/StatsGatheringNeedsDeltastag:CSpace:blog/sysadmin/StatsGatheringNeedsDeltas:7c2b616b1b67e64b12bd3a3d8935e3afc4c06643From 96.228.233.77<div class="wikitext"><p>I certainly agree that anything timestamping the metric data as it reaches the destination is completely broken. What tools do that?</p>
<p>Your other point about counter rollover is real. I guess I don't see the need to solve the problem at every host agent as opposed to at a policy-configurable endpoint. I'm sure there's a good analogous debate about application logic vs. database stored procedure logic to gleen from here.</p>
</div>2012-09-26T15:27:46ZBy Chris Siebenmann on /blog/sysadmin/StatsGatheringNeedsDeltastag:CSpace:blog/sysadmin/StatsGatheringNeedsDeltas:02e58bcc53d08863e8abd1da8d460a606f0b836dChris Siebenmann<div class="wikitext"><p>I don't think that the OS should be doing the delta transformation;
there are far too many problems with that. I think it belongs in the
basic programs that harvest stats from the OS and either present them
to the user or ship them off to higher-level systems like graphing or
monitoring software.</p>
<p>So why not doing all of the delta computation in the graphing and
monitoring software? My answer is that there are two issues that are
hard to deal with in the backend software: timestamps and counter
rollover and other errors. With fine-grained stats, it isn't good enough
to timestamp the raw measurements when the backend receives them from
the collector; the collector really needs to ship a pair of timestamp
and measure off to the backend, and then the backend needs to use the
timestamps. You simply can't assume that the delay between reading the
measure from the system in the collector and the backend receiving the
measure is constant or that any changes in it are unimportant.</p>
<p>Raw measure counters do overflow and experience other glitches;
sometimes this happens quite frequently (a 32-bit byte count on a
gigabit network card will roll over in less than a minute of full-rate
traffic, for example). Something has to recognize and handle this, and
I believe that doing it in the backend significantly complicates the
backend. It also slows down many derivative-based computations, since
the backend must now scan all data points between the start and end
points to find counter rollovers.</p>
<p>(A special case of counter rollovers is system reboots, which restart
all of these running counters from zero. This means that <em>all</em> such
counters 'roll over' eventually and you need to handle this.)</p>
</div>2012-09-24T15:27:16ZFrom 96.228.233.77 on /blog/sysadmin/StatsGatheringNeedsDeltastag:CSpace:blog/sysadmin/StatsGatheringNeedsDeltas:9bea97ada660b681de311c1713768f8f4ad2cd6fFrom 96.228.233.77<div class="wikitext"><p>Performance data presentation transformation does not belong at the OS level. There is a rightful place for that unknown transformation to take place, and it sits between the OS and your eyes. Anything lower-level than that is making likely-wrong assumptions about my needs.</p>
<p>I can't really get myself to believe that you think otherwise.</p>
</div>2012-09-24T13:58:39ZFrom 128.101.204.74 on /blog/sysadmin/StatsGatheringNeedsDeltastag:CSpace:blog/sysadmin/StatsGatheringNeedsDeltas:be39b481b9b90ba30a2bf9105caaa2af3aa36743From 128.101.204.74<div class="wikitext"><p>Personally I use collectd to shoot the raw counters over to Graphite and then on display use Graphite's derivative function to show me change over time. </p>
<p>I am of the belief that you should store the raw data and manipulate it later to fit your purposes.</p>
</div>2012-09-24T13:29:44ZFrom 78.86.151.9 on /blog/sysadmin/StatsGatheringNeedsDeltastag:CSpace:blog/sysadmin/StatsGatheringNeedsDeltas:082e62e2df40a95abe5c022043591687f61fd199From 78.86.151.9<div class="wikitext"><p>Maybe my morning coffee hasn't kicked in yet, but I don't get why "This fundamentally requires an ongoing process". If you run a script at t1 and it outputs v1, and you run a script at t2 and it outputs v2, you can compute a rate, no?</p>
<p>What I think you mean is that if a stats monitoring system can only use values produced by the scripts directly, and can't compute rates for you, then you need to make your script compute the rate. And it can't do that if it simply returns a kernel counter. One way to resolve this would be to switch to a model of ongoing stats collection processes; such a process can compute a rate simply because it can hold v[n-1] and t[n-1] in memory. But I can imagine another approach which does not require an ongoing process: hold that previous pair in a file somewhere (e.g. under /var/lib/blah) and read and overwrite that file each time the script runs.</p>
<p>But there's another approach, which feels to me like the correct one: Treat computing rates as a presentation issue. The stats monitoring system should receive and store the underlying counter values. By knowing (or allowing the user to configure) which series represent delta stats, it can computer the rate series from the underlying data when it draws graphs orconsiders whether to fire alerts, etc.</p>
</div>2012-09-24T07:14:30Z