Why RPM's .rpmnew files don't work in practice

May 19, 2010

In theory, RPM has a solution to yesterday's bad package upgrade problem by way of what are generally called .rpmnew files. When you've modified a configuration file, update the RPM it comes from, and the new version of the RPM has also changed the configuration file, RPM does not overwrite the configuration file but instead creates <config-file>.rpmnew with the new version of it.

This does not work in practice. The problem is that there are entirely too many false positives; it is entirely routine to install packages, change nothing about them, upgrade the packages, and have .rpmnew files sprayed across your system. In this situation, requiring sysadmins to carefully diff each .rpmnew file with its original version and decipher what needs to be done is a great way to cause sysadmins to never install updates, especially because doing this reconciliation often requires us to actually understand the configuration files.

(Note that there's no universal answer for what to do with the .rpmnew files; sometimes you use them and throw away the old version, and sometimes you throw them away because your system has auto-edited important things into the old version. If you're sufficiently unlucky, the new RPM has important changes and the file has been auto-edited and you get to merge them yourself while cursing everyone concerned.)

In practice, I expect that almost everyone ignores .rpmnew files; in fact, if you use the graphical interfaces to package updating I don't think you even get told about them. The immediate consequence of this is that no package update should depend on having its .rpmnew files reconciled by hand, because it's not going to happen.

(If you build such a package update anyways, congratulations, you are an asshole. And I mean that in the technical sense.)

Just to add the icing on this particular cake, I believe that you can get .rpmnew files when upgrading multi-arch packages, for reasons that are similar to how you can get multi-arch file conflicts. This situation is at least easily recognizable, because the .rpmnew file is identical to the normal config file.

(Perhaps modern versions of RPM recognize this situation by now; I am somewhat behind on Fedora releases.)

Sidebar: sysadmins and understanding configuration files

In case it is not obvious: it is extremely common for sysadmins to install packages that have 'configuration files' that we have not read up on and have no understanding of. This is not a bad thing, it is a thing to be aspired to, because it means that your package just works without needing to be customized.

For a direct example, how many of you understand the OpenSSL configuration file, openssl.cnf? I suspect very few people do, yet pretty much everyone has OpenSSL installed and so has an OpenSSL configuration file. If understanding openssl.cnf was a requirement to installing and using OpenSSL, well, I suspect you can guess how well that would work out.

(And yes, on some of my machines I have openssl.cnf.rpmnew files.)


Comments on this page:

From 98.122.161.79 at 2010-05-19 14:40:12:

Got a suggestion on behavior you'd be happier with?

Yum can track when a .rpmnew file gets created and you could write a plugin to do whatever you want, really.

So - what do you think the right behavior is?

From 143.48.3.57 at 2010-05-19 17:09:53:

I have a hard time buying this whole argument. What it comes down to is that there are very few exceptions to two main cases:

  • You haven't changed the file. You accept the vendor's defaults, and you should continue to accept the vendor's defaults unless you have a compelling reason not to. You have no reason to diff the file, since the defaults are good enough for you. Overwrite the config with the .rpmnew file.
  • You have changed the file. Since you've already edited this config, then clearly you know what it does. You should diff and remediate changes manually, since this is a production configuration and you had a compelling enough case to go out of your way to customize it in the first place.

Only the second case is interesting, and the logistics of it should be determined by the vendor's update strategy. In the case of RHEL/CentOS, updates to trivial version/release bumps typically don't do anything interesting, and you can almost universally ignore them until the next minor update of the OS if you like.

From my perspective, you're arguing two different things. You're arguing that system administrators don't always understand the files in /etc. That's true. But you're also arguing that a sysadmin has to manually look at every single file with an .rpmnew equivalent, or their system will fall apart and do horrible things. But the subtext of your argument is one of two things: that RHEL/CentOS are incredibly fragile systems that will make your system fall apart in subtle ways and that the problem is with the magic of the entire system playing with your config files without your knowledge or consent, or you're overtly saying that most sysadmins are full of shit and don't know enough to do their jobs (but this is somehow the package manager's fault). Either way, I'm not convinced, and if these configs are so important that it's too dangerous to replace one default with another, I don't think that automatically remediating changes via something with as little intelligence as diff/patch is the way to go either.

Remediating is as simple as following this process:

  • find /etc -name \*.rpmnew
  • If I haven't made changes to these config files, replace the old versions with the new ones. If OS magic made changes to them before, it will make changes to them again. If not, I'm confident that this is a bug I've already caught in QA, and I've reported it to the vendor.
  • IFS=`echo -ne "\n\b"`; ( for f in `find /etc -name \*.rpmnew`; do diff -u ${f%%.rpmnew} $f; done ) | /usr/share/vim/vim##/macros/less.sh
  • Remediate as necessary.

I've never had this process take me more than two minutes on any production system.

--Jeff

By cks at 2010-05-19 18:01:59:

@Jeff: My overall argument is that if you have not yourself edited a config file, either you should be able to ignore the .rpmnew file or the package update has a bug. Your argument to my first entry is effectively that this is not so and that it is not a bug for a package update to force you to inspect and mangle .rpmnew files even if you have not yourself edited the config file.

If OS magic made changes to them before, it will make changes to them again. If not, I'm confident that this is a bug I've already caught in QA, and I've reported it to the vendor.

My past experience with Red Hat distributions is that this at least was not the case; there were RPMs that directly edited files supplied by other RPMs only when they were installed. Do something that recreated the original file from scratch (such as replace it with its .rpmnew version) and you lost all of those changes.

(The classical case is /usr/share/info/dir, the master index of all GNU Texinfo documentation files installed by packages. I'm confidant that there are others.)

By cks at 2010-05-19 21:13:59:

Got a suggestion on behavior you'd be happier with?

Not for RPM behavior, really, apart from fixing the multi-arch bug. The problem isn't that RPM creates .rpmnew files; the problem is that you can't ignore them because packages do stupid things.

(That's why my original entry blamed the smartmontools package for the problem, not RPM.)

From 174.112.198.41 at 2010-05-19 22:31:13:

@Jeff:

You have changed the file. Since you've already edited this config, then clearly you know what it does.

Only if you remember both that you changed the file, what the purpose of that change was, and what the change actually was.

If there was an easy way to see how the file in production differed from the file that was originally placed there, you could get automatic notification that the .rpmnew file might indicate you have to repeat your change, and you could then get automatic notification as to what your change actually was. From there one would hope you would remember what the point of the change originally was.

-- David Mackintosh

From 69.113.211.148 at 2010-05-20 08:54:40:

@David:

That's a problem that was solved long ago by having proper change management procedures. If your organization can't track down the date/time and purpose of a particular change to a file, your processes are flat-out broken -- how can you tell the difference between an intentional change, an accidental change, or a malicious change? Some people do this the super-bureaucratic way with ticketing systems and signoff, which works for big enterprises with huge SLAs, but the simplest way to handle this is with a revision control system (given, of course, that you use real commit messages). On any given server, for any given file, you should be able to do a git-log php.ini or whatever and get a list of all the changes made and why they happened. (Redmine or some other repo browser is also a really nice tool for aggregating all these changes into a nice web UI.)

From 124.150.39.160 at 2010-05-20 13:20:16:

Everyone defending RPM's current practice: see how Debian does it, particularly ucf which supports a three-way merge.

James

From 76.113.53.58 at 2010-05-20 22:44:05:

One funny thing is, packages used to produce lots of fake .rpmnew's: even if configuration was byte-to-byte identical, the .rpmnew were created anyway. I used to run this after every update:

find /etc -name '*.rpmnew' | while read f; do d=$(dirname $f); b=$(basename $f .rpmnew); if cmp $d/$b $f; then mv $f $d/$b; fi; done

These days I run a batch-diff and then patch.

From 69.113.211.148 at 2010-05-20 22:53:10:

@James:

I absolutely agree, and I love the way that Debian handles things, especially the way that the criticality of the file update is tagged so you can review the really important files and ignore the ones that don't matter. The bigger issue ties into yesterday's post, and that is: "is the .rpmnew behavior of RPM sufficiently problematic where it should cause problems on production servers because the file differences are so numerous and difficult to remediate that it's too frustrating to even try?"

I contend that it absolutely isn't, though I would love to see RPM handle merging the way that dpkg does.

Written on 19 May 2010.
« An illustrated example of how not to do package updates
The limitations of C's varargs support »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed May 19 01:35:11 2010
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.