What you're looking for with a Grafana dashboard affects its settings
Recently I wrote about how we chose our time intervals in dashboards, where the answer is that we mostly
because for our purposes this is the best option. But this raises
the question of what is our purpose with our dashboards. Put another
way, why do we not care about seeing brief spikes in our dashboards?
Broadly speaking, I think that dashboards can be there to look for signs of obvious issues, to look for signs of subtle issues, or to diagnose problems in detail (when you already know there's an issue and you're trying to understand what's going on). Pretty much all of our dashboards are for some combination of the first or the last, and we don't normally go looking for subtle issues.
(The flipside of looking for signs of obvious issues is reassuring you that there are no obvious issues right now. From a cynical perspective, this may be the purpose of a lot of overview dashboards.)
When you're looking for obvious issues, broad overviews are generally
fine. If you have periodic very short usage spikes but nothing else
notices on a larger scale, you almost certainly don't have an
obvious issue. Similarly, showing very short usage spikes on a
broad overview graph isn't necessarily useful unless you believe
that these spikes are the sign of a larger issue. As a result, you
might as well use
$__interval even though it makes short term
spikes disappear when you're looking at longer time periods.
When you're trying to diagnose problems in detail you already know
something is going on and you're probably looking at fine time
scales around specific times of interest. At fine time scales, a
properly set up Grafana dashboard will show you all of the information
available, including fine grained spikes, because it's using a very
$__interval since it covers only a small time range. This
is certainly my experience with our dashboards, where I often wind
up looking at only five or ten minute time windows in order to try
to really understand what was going on at some point.
Looking for subtle issues is an interesting challenge in dashboard design. I suspect it's hard to do without knowing a fair bit about how your environment is supposed to behave (or at least believing that you do). At this point it's not something that I'm doing very much of in our dashboard design (although I've sort of done some of it).