CentOS's switch to CentOS Stream has created a lot of confusion

December 11, 2020

After the news broke of CentOS's major change in what it is, a number of sysadmins here at the university have been discussing the whole issue. One of the things that has become completely clear to me during these discussions is that the limited ways that this shift has been communicated has created a great deal of confusion, leaving sysadmins with a bunch of reasonable questions around the switch and no clear answers (cf).

(It doesn't help that the current CentOS Stream FAQ is clearly out of date in light of this announcement and contains some contradictory information.)

This confusion matters, because it's affecting people's decisions and is hampering any efforts to leave people feeling good (or at least 'not too unhappy') about this change. If Red Hat and CentOS care about this, they need to fix this, and soon. Their current information is not up to the job and is leaving people lost, unhappy, and increasingly likely to move to something else, even if they might be fine with CentOS Stream if they fully understood it. The longer the confusion goes unaddressed, the more bridges are being burned.

(The limited communication and information also creates a certain sort of impression about how much Red Hat, at least, cares about CentOS users and all of this.)

The points of confusion that I've seen (and had) include what the relationship between updates to CentOS Stream and updates to RHEL will be, how well tested updates in Stream will be, how security issues will be handled (with more clarity and detail than the current FAQ), what happens when a new RHEL release comes out, and whether old versions of packages will be available in Stream so you can revert updates or synchronize systems to old packages. It's possible that some of these are obvious to people in the CentOS project who work with Stream, but they're not obvious to all of the sysadmins who are suddenly being exposed to this. There are probably others; you could probably build up quite a collection by quietly listening to various discussions of this and absorbing the points of confusion and incorrect ideas that people have been left with.

Comments on this page:

(Disclosures: I'm the CentOS Stream (and Fedora) Program Manager at Red Hat. I was not involved in the CentOS Linux decision)

I feel like the "you" here is directly squarely me-ward, so I'll address what I can here and work with the team on getting the FAQ updated. :-)

what the relationship between updates to CentOS Stream and updates to RHEL will be

CentOS Stream will get the updates intended for the coming RHEL minor release. What's going into CentOS Stream now is for RHEL 8.4, due in ~five months (since with RHEL 8, we have promised six months for minor releases).

how well tested updates in Stream will be

It's not Fedora Rawhide. We are moving a lot of our CI from internal resources to Stream, and at least some of the QA effort. I don't have the exact details, but I would expect Stream to be roughly of the same quality as RHEL is today. RHEL will get better by virtue of bugs caught in CentOS Stream being fixed before release. Of course, if you're only using CentOS Stream, improvements to RHEL don't particularly matter to you.

how security issues will be handled (with more clarity and detail than the current FAQ)

Due to contractual obligations and embaragoes, many security updates will be shipped in RHEL first and then brought into Stream. Some low-priority issues may get silently fixed in rebases and the like. We won't engineer vulnerabilities back in when they're fixed upstream in order to follow the policy. From a security standpoint, it's no worse than CentOS Linux, and may result in updates coming faster.

what happens when a new RHEL release comes out

I don't know if the messaging is totally settled here, but for now I'll say that there is a CentOS Stream that corresponds to a RHEL major version. You will not wake up one morning to find you've gone from 8 to 9. There will be overlap for a period of time. I'm comfortable saying I expect Stream to correspond to the "Full Support" phase of the RHEL lifecycle, which gives you about a five year life cycle and two years of overlap after a new major release.

whether old versions of packages will be available in Stream so you can revert updates or synchronize systems to old packages

I have no idea, but I'll find out and get it included in the FAQ.

I honestly believe that CentOS Stream will be suitable for the majority of CentOS Linux users, and a huge improvement for some. I think this was the right decision for Red Hat to make. For those who absolutely need a RHEL rebuild, everyone I've talked to wishes those efforts luck and does not want to obstruct them. We're just not doing it ourselves anymore.

That said, I really don't like the timing of this change and had someone asked me, I would have said as much. As a former sysadmin, I have nothing but sympathy for the folks who find themselves suddenly in the position of having a year to figure out what to do with their fleet.

Despite thinking this is the right decision, I'm not happy about how we've communicated it. I'm not able to undo it, but I am working to fix the gaps in our messaging. I understand, as does everyone else I've talked to, that we've lost a lot of credibility with the user community because of this. That makes it all the more critical that we deliver on what we've said we'll do.

I don't have details on what we're announcing next year for the improvements to the RHEL Developer Subscription, but I'm told they'll address the concerns for a lot of CentOS Linux users (I wish we could have made that announcement first, but cats are hard to herd, apparently).

By cks at 2020-12-11 13:30:15:

On the updates: traditionally for RHEL and thus CentOS there has been an ongoing series of security and bugfix package updates that you could get with 'yum update', which were periodically rolled up into RHEL point releases. It's possible that some package updates only appeared once point releases came out; I never paid attention to point updates, just ran 'yum update' every so often. Looking at the RHEL package updates, not the point releases, it's not clear how Stream fits in.

(One assumes that non-embargoed package updates will appear first in Stream, then propagate to RHEL's package updates after some delay, but what delay and why, is not clear. If there are package updates that only show up when RHEL point releases are made, not in the ongoing package updates, they probably also show up in Stream as a 'preview' of what is coming in RHEL once the point release is made. And perhaps Stream will also have bigger changes that would not have previously been made in RHEL at all.)

Okay, so here's a little bit of the sausage making... RHEL has what we call the Y-stream and Z-stream (as in X.Y.Z). CentOS Stream is focused on the Y-stream, which is typically what you'd get in a batched update, as opposed to the Z-stream which comes out as it goes. So the next question is "when do Z-stream updates appear in CentOS Stream?" and that's another question I don't have an answer to today. But it's on my list now :-)

Written on 11 December 2020.
« A probable benefit to enabling screen blanking on LCD displays
Sometimes a problem really is just a coincidence »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Dec 11 00:57:17 2020
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.