Grafana and the case of the infinite serial number

November 17, 2020

One of our more peculiar Grafana dashboards exists to show more or less point in time SMART data for the disk drives on a given server (or some of them). We capture this information in Prometheus for various reasons, and since we have it in Prometheus we want to look at it in a more convenient way than direct PromQL queries. Also, this lets us easily look at the differences in SMART metrics between two periods (not all of which are meaningful, of course). One of the bits of 'SMART' data that we capture is drive serial numbers and model names (and PCI paths). Recently I added a Grafana table to display this to the dashboard, and when I did some of the drive serial numbers displayed in the table as ∞, the Unicode infinity symbol.

At first I thought that the Prometheus query we were using had some problem. Then I looked more closely at the actual serial numbers in question, which were things like '1801E1098900', and the penny dropped. Grafana was interpreting these as numbers written in scientific notation (so '1801 E +1098900' here) and converting them to extremely large floating point values, which it then displayed as infinity. This didn't happen with all serial numbers for drives of this sort, because most of the serial numbers had hex digits in them other than the one 'E' that triggered parsing these as scientific notation numbers.

(It was probably happening with some serial numbers that had the 'E' later, such as '17461934E396', but in those cases the result was less obviously off than an infinity symbol.)

One of the reasons this was surprising to me was that these serial numbers don't come from the Prometheus metric values, but instead from labels on the metric. In Prometheus these labels are always strings, so I tacitly expected Grafana to also always treat them as strings. Instead Grafana guessed at their type, and apparently was willing to accept a variation in type from row to row in the table (since some didn't parse as floating point numbers but were displayed without error). This is kind of what you want, but it also kind of surprised me and I'll definitely have to remember it for the future. In Grafana tables, it's safest to always set a type for all rows.

In current Grafana (we're using 7.3.2 as I write this), the way to deal with this is to set an explicit type for the table field, or even for all fields if they're all really strings (as they are for this table). In the table's editing interface, you want the 'Overrides' tab over on the right and then a 'fields with name matching regex' of '.*' and a 'Unit' of 'String', which is found in the 'misc' section of units.

Written on 17 November 2020.
« POSIX write() is not atomic in the way that you might like
Apple Silicon Macs versus ARM PCs »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue Nov 17 17:39:57 2020
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.