Wandering Thoughts archives

A realization about the recent Red Hat Enterprise security issue

For those people who haven't heard, Red Hat recently suffered a security breach that allowed an attacker to get some bogus OpenSSH RPM packages signed as valid Red Hat Enterprise Linux packages. Red Hat says that the packages were never added to RHN, the update system for Red Hat Enterprise, but presumably the attacker has copies.

(That the package signing is separate from RHN makes sense to me, since I expect that Red Hat needs to sign various sorts of RPMs for testing purposes well before they may get put into general release in RHN.)

This leaves one with the question of what the attacker can do with these packages. It recently struck me that these packages enable an unpleasant perfect conjunction of three or four security issues, like so:

• as the attacker, you use Dan Kaminsky's DNS bug to persuade your victim system that your server is the RHN update server.

• you exploit an apparent vulnerability in yum's SSL certificate verification code to have your server successfully masquerade as the RHN update server.

  (The apparent vulnerability is mentioned, without any details, in the Attacks on Package Managers FAQ. It's possible that whatever it is, it's not exploitable by an attacker in this way.)

• you serve up package metadata that lists these compromised OpenSSH packages as available updates (and the most current version of OpenSSH).

• because these OpenSSH packages have valid signatures, they will pass all of yum's checks and be installed. Game over.

I suspect that this is one reason that Red Hat has issued a new OpenSSH update; once you install Red Hat's update, these bogus packages will no longer be seen as more recent versions of OpenSSH. Even if the attacker manages to feed them to yum under false pretenses (eg, by claiming in the metadata that they have a different version number), I believe that yum will reject them as older than the installed versions.

(Disclaimer: that's a belief, not a certainty; I have not tested it.)

I suppose this makes a third approach to exploiting unsigned repository metadata.

Why noting security fixes in Linux kernel changelogs doesn't really help

People periodically ask for the Linux kernel commit messages and changelogs to mark changes that are security fixes, generally so that they can easily know that they need to cherry-pick just those fixes into their own kernel versions. Ignoring the higher level level issues with this for the moment, there are at least three practical difficulties:

• not all security fixes are recognized as such at the time; sometimes even the people making a change think that they are just fixing an ordinary bug, and the security implications are only recognized later.

• just because you can find a specific change that fixes a security bug doesn't mean that you can use it as-is in another kernel. This goes beyond merely the need to backport the specific change; it means that you need a deep understanding of the code so you can tell if just the change alone is sufficient, or if the fix actually implicitly relies on an earlier change that modified some behavior that is now important for this bug and fix.

  (Even having exploit code that you can test against isn't complete assurance, because it just tells you that this exploit doesn't work any more; it doesn't tell you that you have completely patched the hole.)

• it is fundamentally the wrong approach if you are maintaining a kernel code base, because it means that you are always scrambling to close vulnerabilities after they have been very publicly disclosed. What you should be doing is being proactive instead of reactive; you should be finding out about the problem and developing your own patch for your kernel version concurrently with the patch development for the main kernel.

There is one relatively sensible use for this sort of information: deciding if you need to upgrade the kernel on your production machines to the latest available version, and how urgently. But this doesn't require specific details and marked changes, just 'this kernel is known to fix security issues in the following areas' summary information.

How RPM verification deals with prelinking

Given the problems that prelinking adds to verifying that two systems are identical, you might sensibly ask how RPM verification deals with this problem. Unfortunately, the answer is not calculated to make security people happy.

RPM has no native knowledge of how to analyze prelinked things; instead it delegates this work to an external program. When RPM detects that it is trying to verify a prelinked file, it starts the program and expects it to produce the original, un-prelinked version of the file on standard output. RPM then verifies the checksum of this against what it should be.

(RPM detects prelinked files by looking for a couple of flags in the ELF headers. Things without valid ELF headers or without the flags are just verified as-is, which is the good news in all of this.)

What external program is used for this is not hard-coded into RPM but is instead specified by the __prelink_undo_cmd RPM macro (found in /etc/rpm/macros.prelink on Red Hat and Fedora machines). On a normal system this runs prelink with the -y flag, which verifies that the current version of the file is properly prelinked and then dumps out the un-prelinked version (you can read the details in the manpage).

(prelink itself needs to verify that the prelinking is correct, because nothing else will.)

I am not sure how all of this interacts with rpm's -R flag, which runs at least some scripts and the like chroot()'d. Ideally it would run your version of prelink instead of the version in the hierarchy you are pointing -R at, but I suspect that this won't work very well right now.

(To make this work, prelink would need to know that it should find all of the libraries for things that it is verifying in the 'alternate root', instead of in your regular system.)

How to exploit unsigned repository metadata

Courtesy of Don Kaminsky's slides for his Black Hat DNS talk, I wound up reading Attacks on Package Managers, which identifies a number of attacks that can be mounted if you can set yourself up as a package repository mirror for your favorite Linux distribution. A number of the attacks are straightforward, but judging from the Slashdot discussion the dangers of unsigned repository metadata are not as clear. So here is an attempt to explain them simply.

Suppose that you are an attacker with a repository mirror (the paper shows that this is not a challenge), and further suppose that there is a package, call it 'frobnitz', that has a security hole that you can exploit (this too is rarely a challenge). Unfortunately (for you) it is an uncommon package that's only rarely installed (perhaps it is an obscure browser plugin for some weird format such as VRML). Your goal is to force frobnitz to be installed on as many machines as possible so that you can exploit it.

With unsigned repository metadata and an uncautious package manager, you have two approaches:

• for every newly updated package that gets added to the repository, edit its dependency information in the repository metadata to claim that the new version now depends on frobnitz. Many package managers will then download and install frobnitz as part of updating the other packages without noticing that the repository dependency metadata disagrees with the actual (signed) package metadata.

• generate a version of the repository metadata that claims that frobnitz provides a huge number of things (especially virtual packages and dependencies), and then take steps to insure that frobnitz will be the preferred thing to satisfy any of these dependencies (possible for many package managers). This will cause many people who install a new package to pick up frobnitz in the process, although the overall install may be broken (since the actual thing they need isn't there, as you fooled the package manager into installing frobnitz instead).

Both of these attacks work fine with signed packages (as do all of the attacks in the paper), and they will also work if only some version of frobnitz is vulnerable, since you can always make that the current version in your repository metadata.

(If you have to use an old version of frobnitz you have only a limited time window to exploit the vulnerability; sooner or later the user's system will contact an honest mirror and fetch a fixed version of frobnitz. Of course, this is where Kaminsky's DNS exploit can come into the picture.)

The good news is that even without signed repository metadata, both attacks can be prevented by a cautious package manager that re-checks dependency and provides information against the actual downloaded packages. (I suspect that a number of package managers will be getting updates soon, since making package managers cautious is a lot easier than changing your repository metadata format.)