It's early Sunday morning, which is the time when Fedora kicks off its weekly checks of MD arrays. I know this time extremely well, not because it causes visible IO load but because I actually watch my kernel messages and software RAID floods them during this check. And by that I mean that roughly once every five seconds I can look forward to a burst of messages like this:

md: delaying data-check of md51 until md53 has finished (they share one or more physical units)
md: delaying data-check of md50 until md53 has finished (they share one or more physical units)

I was initially going to write yet again about kernel messages not being sensibly ratelimited (cf), but once I started looking it appears that this is more of some sort of bug in the code. Only my home machine generates this flood of messages; my work machine, which also has software RAID arrays, generates these messages only at the start or end of a data check. The differences between the machines is that the work machine has fewer mirrored arrays (4 versus 8) and all of its arrays are on one pair of disks (at home, half are on one pair of disks and half on the other pair, because one pair is for system stuff and one pair is for my data).

Mind you, the work machine appears to have some message anomalies as well, but it doesn't emit floods of them. Particularly, take a look at this set of warnings emitted at the start of a data check:

[392229.258284] md: delaying data-check of md14 until md13 has finished (they share one or more physical units)
[392229.374681] md: delaying data-check of md13 until md15 has finished (they share one or more physical units)
[392229.374705] md: delaying data-check of md14 until md13 has finished (they share one or more physical units)
[392229.376020] md: delaying data-check of md14 until md13 has finished (they share one or more physical units)
[392229.376022] md: delaying data-check of md13 until md15 has finished (they share one or more physical units)

That's some interesting stuttering going on there, and also some recursion (md15 is the array that just started being checked).

As I was writing this entry, it took long enough that I've now discovered that this is wrong; my work machine sees this sort of thing too. When the md15 data check finished and the md13 one then started, my work machine also started repeating messages about md14 being delayed every few seconds. Fortunately md13 is small so this didn't last long. Something is clearly screwing up here.

(All of this means I should figure out how to send a bug report or two to the software RAID people.)

On the good news front, now that I've looked into this I've discovered that the number of concurrent data checks is controlled by the MAXCONCURRENT setting in /etc/sysconfig/raid-check. Setting this to '1' is not quite correct for my home machine (which could check one array on each pair of disks at the same time), but it does make the messages go away. I'll live with a somewhat more extended data check time in order to not have a message flood.

(I don't know if anyone besides Fedora and RHEL/CentOS do this periodic software RAID check, but if your system does and you see a message flood, you might want to look for a similar setting.)