Wandering Thoughts archives

I don't understand how net.ipv4.conf.*.rp_filter can work

First, the background. net.ipv4.conf.*.rp_filter controls some IP address source validation filtering done on incoming IPv4 packets. It has three values:

0	No filtering is done.
1	Packets are discarded if they come in on any interface except the one that a reply to the source IP would go out on.
2	Packets are discarded if a reply to the source IP could not be sent out any interface.

(A more formal description is in ip-sysctl.txt in the kernel documentation. Like all interface sysctls, it can be set separately for each interface, as a default, and for all interfaces.)

I don't understand how this can possibly work. Well, I understand how it works, I just don't understand how it can possibly do any good in most configurations. And I don't understand how a setting of '1' can possibly work at all in multihomed configurations where the multihomed machine is not the sole router for every network it's connected to that is not where its default route points.

First, as far as I can tell a setting of '2' is equivalent to '0' if you have a default route set (the usual case). With a default route set, all source IPs are reachable and so '2' will never discard packets, which is exactly the same as '0'.

For a machine with a single network interface and a default route, all settings are equivalent (for the same reason as above; all source IPs are reachable through your single interface). If you do not have a default route, either '1' or '2' will discard packets that come from networks you do not have routes for.

It is the multihomed case where things explode. Suppose that you have a multihomed host with two network interfaces, net-1 and net-2, with IP-1 on net-1 and IP-2 on net-2. With an rp_filter value of 1, a machine on net-2 cannot talk to this machine's IP-1 address unless the packets pass through the multihomed machine on the way to net-1, ie the multihomed machine is the router for the net-2 machine. If the packets go through another router, they will arrive on the multihomed machine's net-1 interface but the replies would go out the net-2 interface, so they fail the check.

Effectively this creates a bad version of an isolated interface, with the packet reachability restrictions but without the multiple split routing tables that make multihomed hosts actually work. As a bonus it hides the restriction deep in the networking sysctls, where you have to be an expert to find it.

(I suppose that there are some advantages to this half-hearted approach, in that it avoids some limits in the policy based routing version of it.)

By the way, I stumbled over this courtesy of Ubuntu 10.04 setting rp_filter to 1 by default. We have multihomed non-routing machines, and when we set up an Ubuntu 10.04 test version things promptly exploded. If I was not already suspicious of network sysctls, we could have spent quite a lot of time trying to find out just why the machine was ignoring certain sorts of network traffic.

(As it was I did 'sysctl -a | fgrep net. | sort' on both a 10.04 and an 8.04 machine and then looked for settings that were different. Ubuntu 10.04 may not be the first version that sets this, but 8.04 definitely didn't.)

PS: a much more useful version of this sysctl would be a 'private' flag on interfaces. If an interface had the private flag set, packets with a source IP address that was routed through that interface would only be accepted on that interface; all other interfaces would discard such packets.

My impressions of Google Chrome so far

My first impression of Chrome is that it's fast. My torture test is Flickr's new interface, which has some JavaScript that manages to bog down my machines with every version of (Linux) Firefox that I've tried, including the 4.0 beta builds (which theoretically have significantly faster JavaScript). On Chrome it flies, to the point where it feels faster with JavaScript on than Firefox does with the JavaScript turned off in NoScript.

(This is in fact what motivated me to try Chrome to start with. I am relatively active on Flickr, so I really noticed when it suddenly became achingly slow. After working through a whole series of attempts to get a faster Firefox, I gave in and tried out the reputedly fastest browser. To my surprise it was indeed much faster.)

Using a bare Chrome install was a great way of finding out which bits of my Firefox interface I really need in order to feel effective. So far my necessary Chrome extensions are:

• MiddleButtonScroll, which does just what it says it does; it gives you what Firefox calls autoscroll, where holding down the middle mouse button lets you scroll the page around.

  (No doubt everyone just uses their mouse wheel to scroll and doesn't care, but I'm still a holdout.)

• Smooth Gestures plus the Smooth Gestures plugin to make it work on Linux (it's covered on the main Smooth Gestures page). There are several gestures extensions for Chrome, but this was the first one I found that said it more or less works on Linux, which it does; there are issues, but nothing I can't live with for now.

(It turns out that for casual browsing I really want the ability to conveniently navigate purely with the mouse, including scrolling the page.)

I haven't explored my options for something like NoScript and CookieSafe. There's a NotScripts extension, but it currently has some annoying requirements that made me decide I didn't want to deal with it right now, and apparently there is also some degree of native support for JavaScript control in Chrome, which I haven't looked into at all.

(If I was trying to use Chrome full time in place of Firefox, I would also need to find some replacement for Stylish and ideally It's All Text as well. Or, as it turns out, I would just need to install Stylish, as it's already been done for Chrome.)

I'm using the official Google Chrome build, not Chromium. I installed the 32-bit RPM version despite having a 64-bit machine, and I see no reason to do otherwise unless you have unusual needs. For a start, Flash just works (it's bundled into the RPM). I turned off the package's automatic updates by the simple expedient of 'chmod a-x /etc/cron.daily/google-chrome'. To their credit, Google explicitly tells you in advance that their package installs an auto-update process and adds their repository to your yum configuration, so I knew to go hunting.

(I'm aware that Google documents another procedure for turning the automatic update off. Per previous commentary, I'm a sysadmin so I feel much happier about code that isn't running at all.)

Sidebar: the Chrome yum repo configuration

The Chrome RPM doesn't contain an explicit /etc/yum.repos.d file for their repository, apparently because Google is trying the heroic task of building one RPM that can work on several different RPM-based distributions with different ways of doing all of this. Instead it is created by the cron script the first time it runs, which is a slight problem if you turn off the cron script entirely.

To save any interested parties the job of reading the script's source, here is the current google-chrome.repo that the script creates (for 32-bit machines):

[google-chrome]
name=google-chrome
baseurl=http://dl.google.com/linux/rpm/stable/i386
enabled=1
gpgcheck=1

In fact, I see that Google documents all of this here, including adding the necessary package signing key and the 64-bit repo configuration.

(Note that they use a somewhat different 'name=' value between their website and what the cron script sets up.)

A Windows moment on my laptop

I had a Windows moment with my laptop today. I normally leave it suspended instead of powered off, and when I unsuspended it I discovered that its networking had stopped working. The network-controlling widget in my taskbar just said 'Networking unavailable', and the various system management interfaces were opaque about what might be wrong and why; the network devices were there, they weren't reporting any obvious problems, and so on. The system just wasn't having anything to do with networking, whether wired or wireless. So I used the usual Windows solution: I rebooted the machine and the problem went away.

The only thing wrong with this story is that my laptop runs Fedora 13.

The problem here is not the existence of Network Manager and its collection of GUIs for managing networking on a modern Gnome desktop (although some old Unix people may feel otherwise). The problem is that none of them were willing to tell me why networking wasn't available. They knew there was a problem; the Network Manager applet even told me so. It just offered no diagnostic and no suggested remedy, not even a 'make networking available again' option to go along with its helpful message.

What makes this a Windows moment is not the GUIs; it is the black boxes. I don't object to making Linux management more user friendly, but the systems for this need to do one of two things: either they need to have black box problem resolution (where there is a 'make networking work again' button), or they need obvious ways to open up the black box to tell you what is wrong and what you need to do to fix it. If you normally manage with GUIs, those GUIs need to have that detailed 'what is wrong and what you can do about it' interface.

(Looking back at the incident, I suspect that some crucial Network Manager daemon process died and that if I had looked at logfiles somewhere I might have found this mentioned. But I didn't, because Network Manager has successfully made itself into such a black box that I have no idea what all of its pieces are and no interest in learning, because it seems pointless.)

Sidebar: why I like Network Manager

The short summary is that it gives me a simple interface to a complex world and makes that world more or less work. I can plug my laptop into any network that gives out DHCP and it will automatically come up on it. I can associate myself to one of several different wireless networks (with different keys). I can bring up a VPN over all of this. And I do all of this at various times.

With sufficient research I could write scripts and run commands to do all of this, and keep opening terminals and running sudo as I move around between networks and suspend my laptop and so on. Network Manager means that I don't have to. I'm lazy, so I appreciate that.

(Could it be better? Of course. But most of the ways I can think of to make it better are to let me express relatively complex policies, like 'if wired networking is available, use it; otherwise, if this wireless network is available, associate with it and then bring up this VPN'.)

My (probably wrong) assumption about Flash on Fedora

In hindsight, it is somewhat peculiar that I spent a considerable time using Fedora 11 with a broken Flash setup instead of seriously looking into the situation. What happened is that I made a simple assumption: I assumed that Flash and the associated infrastructure needed to run it on a 64-bit machine were second class citizens, looked down on and basically unsupported.

If I'd assumed that Fedora wouldn't leave Flash broken on 64-bit machines, then it being broken on my machine was clearly either something wrong with my machine that I could fix (which is what it turned out to be) or something worth a bug report. In either case I would have dug into it and probably solved the problem much earlier than I did. Instead I assumed otherwise, and also that any bug report involving Flash would would be promptly closed as either 'not our problem' or 'nothing we can do since Adobe Flash is a black box' (or both).

Although it sounds crazy, I don't think that this was an irrational assumption to make. I've at least perceived that the 'free software' view of Flash has generally been that it was an annoying and alarming proprietary technology that should be avoided, and certainly free software people weren't necessarily going to go out of their way to make it work. In this view, the odd thing wasn't that Flash wasn't working on my 64-bit machine; the odd thing was that it had ever worked.

(In fact at one point it didn't work by default; back in the time of Fedora Core 5 and 6, I had to compile my own versions of nspluginwrapper.)

One of the lessons I draw from both this incident and my earlier experiences of things starting to work on 64-bit Linux is that I need to periodically re-check my assumptions about what is and isn't supported on Linux. I may well be pleasantly surprised, as I have been in the past.

Getting Flash to work on my upgraded 64-bit Fedora 13 machine

(This could be subtitled 'the problem with upgrades instead of reinstalls'.)

I recently upgraded my office workstation from Fedora 11 straight to Fedora 13 (with a yum upgrade). Just like the last time I upgraded my Fedora install, 64-bit Flash promptly broke. I have since fixed that, but as it turns out I can't tell you exactly what the solution was.

(Like most people using 64-bit Fedora, I run Flash via nspluginwrapper's 64-bit to 32-bit bridge instead of as a native plugin. One of the interesting effects of using Flash this way is that it runs in a second process, outside the browser, which means that I've always had the kind of crash isolation that is only now coming into general use with browsers like Google Chrome and Firefox 4.)

This time around I was smarter than last time and immediately tried a number of things. Of course, first I tried the fix from last time, but it didn't work; this time I had another issue instead. As a short term workaround I installed a 32-bit Firefox 3.6.8 from mozilla.org into my own $HOME and switched to using it, which worked acceptably well (and Flash worked fine).

Using a spare virtual machine, I also discovered that Flash worked fine on a stock 64-bit Fedora 13 machine. A comparison of the package list between my normal machine and the virtual image showed that the stock install had a number of extra packages installed, so as an experiment in brute force I installed all of them on my workstation. To my surprise, 64-bit Flash promptly started working.

(Since 32-bit Flash was working, this is almost certainly some sort of hidden dependency that nspluginwrapper has. RPM tries to catch this sort of stuff, but its heuristics can only go so far, and it would be quite hard to catch this sort of issue in ordinary testing. Since I have finite time and low energy, I have not attempted to do all sorts of isolation tests to work out just what the dependency is. Frankly I am just as happy that I didn't have to work out how nspluginwrapper works.)

In case anyone else is in this situation, here are two resources:

• for people who may not have virtual machines and a bunch of bandwidth handy, f13-stock-rpms.txt is a list of the all of the RPMs (currently) installed on my 64-bit Fedora 13 virtual machine image. Note that this includes the Adobe Flash RPM et al, which I installed following these directions.

  For convenience and future-proofing, I have listed only the name and architecture (ie, rpm --qf '%{N}.%{ARCH}\n' -qa). Because of the sort locale braindamage, you may need to re-sort this before using it yourself.

• for the sufficiently curious, f13-rpms-added.txt is the exact set of RPMs that I fed to 'yum install' on my upgraded F13 machine and that made Flash work. Keenly energetic people are invited to remove these RPMs one by one from their Fedora 13 system until nspluginwrapper breaks.

(Some of the RPMs are pretty clearly not relevant, for example all of the firmware ones.)

One of the lessons I take away from this is that I may want to do this sort of installed RPM resynchronization after every future Fedora upgrade that I do, just as a matter of course. While I always knew that an upgrade (and especially a yum upgrade) is not quite equivalent to reinstalling from scratch, I had previously been assuming that the differences were small and unimportant. Clearly this is sometimes false.

(This means that I should spin up a 32-bit Fedora 13 virtual machine, since my 32-bit laptop has also been upgraded from Fedora version to Fedora version instead of reinstalled from scratch. It's probably missing packages too, although I haven't noticed any problems yet.)

Upgrading to Fedora 13 with my first yum upgrade

I've just gotten through with upgrading my office workstation from Fedora 11 to Fedora 13 with my first yum upgrade, and I have to say that the experience was reasonably pleasant. Reasonably pleasant here means that nothing exploded and I could use my machine throughout the process, which is what I wanted out of the whole exercise.

I did have to do some cleanup work in order to get the upgrade to start by removing various packages that caused yum to choke, all ultimately due to the RPM multi-architecture file problem. Most of the problems were orphaned 32-bit Fedora 11 packages, where there was only a 64-bit package in Fedora 13 and so the old 32-bit package clashed with the new 64-bit one; I fixed those by removing the 32-bit packages. The one odd problem was yum deciding it needed the 32-bit KOffice libraries as well as the 64-bit ones for some inexplicable reason (I didn't have any 32-bit KOffice components installed in Fedora 11). I solved that problem by removing KOffice entirely.

The upgrade process itself took three hours or so to run (much like my last upgrade), but I'll call that fair since it was doing just under 6,000 package operations between upgrades and installs of new packages and cleanups of old ones. I didn't have any problems running it inside an X session, although I took the precaution of running it inside screen and as advised I excluded bitmap-fonts-compat from the main upgrade, doing it only at the end after I had quit out of X and was about to reboot the machine.

(I am technically not entirely finished the upgrade, because I now need to weed out a number of orphaned packages, sweep the system for .rpmnew files and work out what to do with them, and so on. But the important work is done and my system works again; what's left is basically cleanup.)

In fact I was impressed by how much ran well during the whole process. I actually started using the new Firefox part way through the upgrade and nothing exploded, somewhat to my surprise.

My experience in the upgraded Fedora 13 environment is mostly positive, in that most of the important things on my system have worked without problems. I did have a spot of alarm when I couldn't get any sound, but it turned out that PulseAudio had muted my output device. (I have to say that pavucontrol does not make this state very obvious if you are not already familiar with it.)

(Overall I am not entirely happy with Fedora 13 right now, but that is because USB serial ports seem to be utterly broken on my machine and the perversity of the universe chose tonight of all nights to somehow totally kill my main phone line, DSL included. I could really use that backup PPP connection right now, but instead it is causing my kernel to log messages about processes hanging in the depths of the TTY and USB systems.)

A personal view of the Fedora versus Ubuntu issue

A commentator on the previous entry more or less suggested that I switch from Fedora to Ubuntu. As it happens, this isn't something that I'm likely to do.

In one sense there ought to be no real difference for me between the two. My personal Linux environment is so customized that I'm essentially indifferent to what the distribution does with its desktop environment (the usual point of contention for ordinary users); things like Gnome and KDE choices and which of the latest peculiar system for managing bits of the system are in use don't matter if you don't use them. And pretty much any distribution is going to include the basic elements that I use and care about. While I'm used to how to manage and navigate around Fedora, I could perfectly well learn Ubuntu/Debian packaging and management too (and I already know some of it, since we use Ubuntu on servers at work).

In another sense, no. I have too many bad experiences with Ubuntu (courtesy of work, again). Through my experiences with both Ubuntu and Fedora, I've wound up trusting Fedora a lot more in terms of packaging, bug reports, and the decisions that each distribution makes; I'm sure that Ubuntu is trying, but I am equally sure that they do slipshod work and make dubious decisions because I've experienced it myself. Ubuntu is quite good for some things (the out of the box desktop experience, primarily), but I am not really interested in those things and I have the strong impression that Ubuntu is not very interested in areas outside those things.

(Fedora makes mistakes too, but I feel that they do more solid and more thorough work in general. I can't imagine Fedora changing to a volatile /var/run but not fixing all their packages, for example.)

Additionally and more inflammatorily I feel strongly that Debian and thus Ubuntu has made some fundamental missteps in things like multi-architecture support and package installation, and Fedora has not. These issues are probably arcane to a lot of people, but they matter to me and thus strongly tilt me towards Fedora.

(Not RHEL/CentOS, for reasons covered here.)

Why buying a Linux machine is not such a simple thing

I'm somewhat considering buying a new home machine. If I used Windows, this would be an easy process; I could just go over to the local cluster of computer stores and pick up an inexpensive generic white box machine, or just buy something from Dell et al if I felt the need for a name brand. I wouldn't have to think about what components were inside the box because the result would invariably run Windows, although perhaps not without minor glitches; everyone builds things that run Windows.

(We're a university, so of course there's a cluster of local computer stores nearby. As for inexpensive, well, over the past few years it's become quite impressive how much computer you can get for not all that much money.)

However, I want a Linux machine. This means that I get to care a lot about what components are inside the box, because there is no way I can just assume that Linux will run on whatever random hardware I get, much less run well. Ignoring graphics cards entirely, there's all sorts of hardware where Linux either doesn't have drivers or doesn't have good drivers.

(And to be fair, a certain amount of this hardware is simply bad and would be a problem under Windows too.)

Thus I need the right components, and that means that I need to find out what the right components are, which means that I get to walk back into the ever-changing swamp that is PC hardware just so I can find out what hardware I should even start looking at. In theory I could settle for just 'has a Linux driver', but if I have to pick the components myself I might was well pick good components, ones that are known not to have odd failure modes.

(This process is not helped by the utter inability of the modern Internet search to provide useful links to good summary and feature comparison sites in between the 5671689 different 'buy here! boost our pageviews!' sites. I'll probably have to turn to Wikipedia to get a decent summary of the current state of CPUs and memory technology, too.)

All of this is, frankly, annoying and a hassle. The annoyance involved is one reason that I am still only somewhat considering replacing my home machine; after all, it works (more or less) and leaving it alone keeps my annoyance level down.

(Honesty compels me to admit that part of the problem is my perfectionist streak. Any time I spec out hardware I tend to get sucked into over-thinking and over-engineering the result; especially if I'm paying for it, I can't settle for a merely okay machine. I don't necessarily want the fastest and most powerful components, but I agonize over picking good ones and arranging everything just right. This is why I am unlikely to go into one of the local computer shops, get the current component list for one of their generic machines, and see if all of the bits work well enough under Linux.)