The fun of 32-bit bugs

As computers (and disk space, and memory, and etc etc) get larger, the quantities that we deal with routinely get bigger too. And when they get bigger, fun things start happening.

Today's fun thing was that I doing some measurements of disk IO speed on a machine with 2 gigabytes of memory. My usual rule of thumb is to work on at least twice the amount of main memory to crush cache effects, which meant I was telling my benchmarking program to read and write 4 GB.

Which turned out to be kind of a problem, because I had declared a variable as int instead of long (or better yet, off_t). At 4GB it rolled over and various interesting things happened. I count myself fortunately that it was instantly obvious that something was bad; it could have just resulted in quietly wrong numbers that I might not have noticed.

As our systems and what we do with them get bigger, I imagine I can look forward to more and more incidents like this. Already, 2GB files are becoming pretty common and it is more and more irritating when tools don't deal with them, or don't deal well with them (some versions of less will eat large files but mangle the percentages and jumping to percentages, as I found out recently).

One benefit of 64-bit computing is that many of these problems can be papered over on 64-bit platforms by a recompile, since that makes long big enough again. (Some people will decry that as a quick fix.)

(Yes, technically this wasn't a 32-bit bug, it was a 31-bit bug. Close enough, says I.)

Sidebar: why not just use bonnie++ or the like?

Two reasons. First, bonnie++ benchmarks too much; at the moment I'm only interested in streaming read and write speeds. Second, bonnie++ was giving me odd results and I wanted to crosscheck them with something else.