The problem with making bug reports about CentOS bugs

October 2, 2014

I mentioned yesterday that I had not made any sort of bug report about our NetworkManager race bug that we found on CentOS 7. The reason why is pretty simple: where can I report it that will do any good?

I can't report it to Red Hat as a bug against Red Hat Enterprise 7. Red Hat does take public bug reports against RHEL the last time I looked, but I'm not running real RHEL, I'm running CentOS. Even if I could, reinstalling a machine with real RHEL 7 simply to be able to make an 'official' bug report for this issue is not an effective or efficient use of my time. By now we're never going to use NetworkManager even if it's fixed; we don't actually need it and the the risks of another bug existing are too high.

(I actually did go through this exercise for one bug, but that was that the RHEL/CentOS 7 version of systemd winds up putting kernel messages in syslog under the wrong facility. This is both potentially likely to get fixed and something that rather matters to us. And that was a straightforward and easily demonstrated bug, which makes bug reports trivial to file (and it still wound up taking up a bunch of time and talking with Red Hat because it initially looked like rsyslog was at fault).)

There seems to be no point in reporting this to CentOS because it's not a CentOS bug as such. CentOS simply rebuilds the upstream RHEL RPMs, so there is exactly nothing they can or will do to fix this bug (assuming it's not specific to the CentOS rebuild of NM, and I have no reason to believe that it is). The corollary to this is that the only bugs I suspect are worth reporting to or against CentOS are basically packaging bugs.

(With that said, people do seem to report a lot of bugs in the CentOS bug tracker.)

Even if I felt like wading into the upstream swamps of NetworkManager, we're not using anything like the current upstream version of NM (RHEL 7 and thus CentOS 7 is absolutely guaranteed to be behind the NM times RHEL and thus CentOS almost certainly patches NM a bunch). As a result upstream would quite rightly basically laugh at me (perhaps politely, via a 'try to reproduce this with current NM').

(Having written that, I've just discovered that RHEL/CentOS 7 has a more recent version of NM than Fedora 20 does. CentOS 7 ships, apparently pulled from git on 2014-03-26, while Fedora 20 has apparently from 2013-10-03. No, don't ask me. Possibly Red Hat felt that it was really important to use as fixed up a version of NM as possible before a RHEL release.)

Once upon a time this was just how CentOS had to be. But now that Red Hat has kind of taken over CentOS, it strikes me as a rather inefficient way to operate; RHEL is basically passing up the bug finding work that CentOS users are doing. With that said, it seems that Red Hat may unofficially accept 'RHEL' bug reports that actually happen with CentOS, but if so this is not documented anywhere that I can casually dig up (outside of some CentOS bugs with replies that ask people to re-file the bug in the RHEL bugzilla).

(And this lack of documentation is likely causing people other than me to not even bother filing bugs.)

PS: also, I hope it's obvious to people that a setup that routinely causes bug reporters to have to refile their reports in another bug reporting system is a hostile one that implicitly discourages bug reports. It's hoop-jumping in splendid form. If this is the real CentOS procedure, it should be changed (especially now that Red Hat is so involved in CentOS).

Comments on this page:

From at 2014-10-02 01:11:51:

I have reported centos bugs to the redhat bugzilla. I have just left out in the report I was running centos ;-)

They all got eventually fixed.

By opk at 2014-10-02 06:15:33:

If it is essentially an upstream bug and not packaging I tend to think it's far better to wade into the upstream swamps as you call it. I once packaged something for Debian and mainly gave it up because of the volume of bug reports that were basically for upstream but I had to verify, reproduce, and forward them.

By Twirrim at 2014-10-02 12:39:32:

I've submitted CentOS bugs upstream to RHEL as well. Funnily one of them just got fixed some 3-4 years since I first submitted it...

Now that CentOS is actually a RedHat project it might not be quite as painful, I'm hoping that communication lines have been opened better.

By Pete at 2014-10-03 17:11:45:

My interaction with Dan on topic of NM problems was a bit of a mixed bag like you expect. He is excessively pleasant and attentive to bugs (quite unlike that other celebrity). Fixes to RHEL 7 are going to be delivered. But of course you have to commit to running versions with debugging and then of course there's "the latest" even for the 7. Due to the critical nature of my NM use, I had difficulties experimenting with it.

Red Hat has something that is called "Z-stream": patched up versions of older releases. But one normally cannot get onto that without paying.

By Johnny Hughes at 2014-11-07 13:51:49:

You could not be more wrong about your assumptions:

1. You can enter RHEL-7 bugs by just registering an account.

2. People enter CentOS bugs all the time there. There are currently 716 bugs open that mention CentOS. If it is a bug, then report it. That is how things get fixed.

3. Red Hat and CentOS are affiliated, but CentOS is still a rebuild of Red Hat source code. CentOS is, by design, bug for bug compatible with other things built from the same source code.

4. is community help .. meaning that our users and volunteer QA team answer questions there. Not only should you report bugs there .. you should also find and fix, the report the fix there. That is how open source works. You should fix the problem, report the fix to and if you can .. I mean, you are getting the software for free, right?

By cks at 2014-11-07 23:15:18:

In reply to Johnny Hughes:

You could not be more wrong about your assumptions:

Then CentOS has documentation issues, because I, as a sysadmin using CentOS and reading the web pages (including the lead in to the bug tracker) was left with these beliefs. If there is documentation that says otherwise, I did not find it, and I looked.

  • You can enter RHEL-7 bugs by just registering an account.

As mentioned in the entry, I know this ('Red Hat does take public bug reports against RHEL [...]') and I've even done it (as mentioned).

  • People enter CentOS bugs all the time there. [...]

The issue is not whether people enter bug reports, it's whether Red Hat wants them to and will do anything with those bug reports. If they do, again, there is a documentation issue because it is not mentioned. In addition, as far as I can tell filing such a bug report requires lying; since there is no 'CentOS X' component, you must lie and report it against RHEL X. As a personal policy matter I do not lie in bug reports. If I cannot file a bug report without lying, I do not file a bug report.

(I don't consider it lying if some documentation says 'RH accepts bug reports from CentOS, file them as bugs against RHEL'. And yes, I consider filing things with improper components to be a lie even if I say straight up in the description that this is against CentOS instead of RHEL, because I've caused someone filtering on real RHEL bugs to spend time looking at and possibly discarding my bug report.)

  • [...] CentOS is, by design, bug for bug compatible with other things built from the same source code.

There are a number of things that could be said here, including about the difference between theory and practice and that CentOS does change some packages because they're obliged to by Red Hat policies, but fundamentally the compatibility is simply a potential technical enabler. The real issue is a policy one; even if CentOS is bug for bug compatible, what matters is Red Hat's policy decision about whether or not they pay attention to or want bug reports against CentOS. Right now the answer to that question is invisible and the presumption I arrive at is that the answer is thus a default 'no'. I maintain that this presumption is rational.

As I explicitly noted in the entry, it may be that Red Hat's policy is to accept 'RHEL' bug reports against CentOS but if so this is not documented and perhaps not official. In the absence of documentation, one gets uncertainty and the resulting unfiled bug reports (including mine).

As for your item four in general, I have a strong reaction to the message it (hopefully) inadvertently sends. See that entry for much more.

Written on 02 October 2014.
« NetworkManager and network device races
When using Illumos's lockstat, check the cumulative numbers too »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Oct 2 00:08:41 2014
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.