Chris's Wiki :: blog/linux/NoEnableOnUpgrade

Enabling services on package updates is a terrible mistake

August 13, 2015

Let's start with my tweet:

The #CentOS 6 iptables package update unconditionally (re-)enables the service and thus turns on firewalls. BRB, setting things on fire now.

Really, it does. It's right there in the RPM postinstall script:

; rpm -q --scripts iptables
[...]
postinstall scriptlet (using /bin/sh):
/sbin/ldconfig
/sbin/chkconfig --add iptables
[...]

Of course iptables by itself does nothing or rather just applies whatever rules you already have, but your CentOS 6 machine almost certainly has a restrictive set of rules in /etc/sysconfig/iptables that were written there by system-config-firewall during the system install. Turning on the iptables service will cause them to be applied at the next reboot, and in our case this took out incoming external email for more than twelve hours because of course those rules blocked incoming connections to the necessary TCP ports.

(Yes, there were two problems there. We know.)

At one level this is a straightforward total failure of good packaging; a package update should never enable a currently disabled service. Automatically enabling a service on the initial package install may or may not be a good idea, but changing the system state on a mere package upgrade is clearly utterly wrong. A deliberately disabled service suddenly turning on is generally going to do something bad to the overall state of the system; the only question is just how bad. Iptables is well placed to make this really bad.

(In turn this means that there was a major process failure here. This issue is almost certainly present in the original RHEL 6 update that CentOS 6 built their package from, and Red Hat Enterprise of all distributions should have better update validation than that. This should not have gotten through code review.)

At another level this is also a joint failure between RPM and chkconfig because both make it extra hard to do the right thing. RPM has only a single 'postinstall' script which is run after both installs and upgrades, which means that you have to remember to have your shell code explicitly check for which case you're in (and it's not at all easy to test the upgrade case). Chkconfig and in general the whole Red Hat init.d symlink system don't draw any distinction between 'what the package wants to do by default' and 'what the local sysadmin has specifically set up', which leaves packages easily able to make mistakes that override sysadmin decisions like this. Put the two together and you have an explosive mixture where any failure can blow your foot off. This is not a resilient system.

(Systemd does much better than the init.d stuff here precisely because it has a clear distinction between these two things.)

Comments on this page:

By dozzie at 2015-08-13 04:10:14:

@cks:

At one level this is a straightforward total failure of good packaging; a package update should never enable a currently disabled service.

You confuse removing service from booting with disabling it. chkconfig --add iptables should not re-enable a service disabled with chkconfig iptables off.

But of course I agree that enabling and starting a service by default, like Debian does, is a bad idea.

By Ewen McNeill at 2015-08-13 05:50:07:

@dozzie: I think there's a reasonable argument that if you go from "service not installed" to "service installed" you likely want to run the service, and thus would probably like it turned on/started (this is I believe the Debian/Ubuntu rationale for turning services on at first install: apart from some which are useless/dangerous without site-specific config, and have some custom flag to default to off, eg Shorewall -- a firewall tool). It's maybe not perfect, but it is reasonably defendable.

But I do agree with cks that during the transition from "service was installed" to "service installed (and is now a newer package)" the state of the service (starts automatically/only started manually) shouldn't change. And that there should be an obvious way to turn an installed service off such that it won't magically turn back on again at reboot/package upgrade/whatever. A bunch of unix-like systems have gotten this wrong over the years.

Separating "package default action" and "local admin preferred action" seems like a good start towards sane behaviour. As would having a standardised approach to indicating "package installed but do not run service on first boot" rather than per-package ad hoc solutions. Ideally one which allowed starting the service manually sometimes, using the same boot scripts run by hand (eg, "service FOO start" by hand) -- which a bunch of the ad hoc solutions do not allow.

Ewen

PS: There's a similar argument that installing a service should automatically add firewall rules to permit it to be reached, as that is likely what you want when you install the service. Which would have made the "firewall comes on by itself" scenario less tragic. But I guess it's harder for mail services, as a bunch of servers end up with a local MTA that shouldn't accept connections from the outside world.

By Dan Astoorian (Dan.Astoorian) at 2015-08-13 15:46:11:

For what it's worth, this behaviour is actually documented.

The only way I can think of that /sbin/chkconfig --add iptables should enable the service is if someone had previously done a /sbin/chkconfig --del iptables. The chkconfig man page reads:
  --del name
         The  service  is removed from chkconfig management, and any sym-
         bolic links in /etc/rc[0-6].d which pertain to it are removed.

         Note that future package installs for this service may run  chk-
         config  --add,  which  will re-add such links. To disable a ser-
         vice, run chkconfig name off.
I don't disagree that upgrading a package shouldn't cause a service to be enabled, but in this case it looks like you changed something that the package reasonably expected to be left alone: instead of using the supported mechanism for disabling the service, you uncoupled it from the mechanism and were surprised when the upgrade put it back.

Written on 13 August 2015.

«	What common older versions of `free` are telling you

My current views on using DomainKeys (DKIM) here