Wandering Thoughts archives

How I set up my isolated testing Firefox environment

Yesterday I covered how I actually run two web browsers, one for my real browsing and one for various sorts of testing and isolated browsing. That raises a question: how do you set up something like that?

In theory, you could just use an alternate Firefox profile (with a script to start your test Firefox with the right magic arguments). In practice I have never trusted Firefox's profile system that much, partly because in the past I spent some time testing out potentially buggy development builds. So I have always used the bigger hammer of changing my $HOME.

On Unix, Firefox (and Mozilla before it) has always respected $HOME (if set), instead of doing something unsocial like looking up the user's home directory in the passwd file. Thus the simplest way of getting a completely independent testing environment is changing $HOME to point to somewhere convenient. For my general testing Firefox environment, I use a persistent directory in ~/tmp (so that I can set Firefox preferences and have them stick); for scratch testing, you can just set $HOME to, say, /tmp/ffox-test, make the directory, run Firefox, and delete the directory afterwards.

You will also want to start the testing Firefox with the -no-remote command line argument (see here); otherwise your testing instance may just send things to your main browsing session. I deal with this in a somewhat eccentric way, so you may need to experiment with the best way of making everything work given Firefox's insistence on doing various bits of remote control.

(I put all of this stuff in a script that I call 'foxnative'. Remember to do 'export HOME' after setting $HOME, just to be sure.)

At one point I needed some extra steps to make sound work with PulseAudio, because it required some magic files in your home directory in order to connect to the PulseAudio server and it too respects $HOME. Back when I ran into this, I wound up using rsync to copy ~/.pulse to the new $HOME; however, that appears to be unnecessary now in current versions of Fedora.

(This shows one quiet little hazard of scripts, which is that they can automate something that turns into folklore over time while you're not paying attention.)

Fixing Upstart's coupling of startup script presence and activation

While Upstart has the problem of joining together the idea of a startup script being present and it being active (per my earlier entry), there is a relatively simple way of distributions to fix this while preserving all of the benefits of Upstart.

Although Upstart sort of apes traditional runlevels, it really operates based on events; services declare what events they depend on, and they start when those events happen. Upstart events are completely abstract (although it has some built in ones), and they can be generated by hand; 'initctl emit fooevent' will tell Upstart that fooevent has happened.

Given that Upstart startup scripts can be configured to only start when multiple events have happened, the fix is straightforward; you make all services depend not only on their usual events but also on a synthetic <service>-enabled event. The system generates all of the necessary *-enabled events during boot, based on some scheme to keep track of which services are enabled (and possibly which runlevels they're enabled in, although runlevels don't really make much sense any more).

(How exactly the system keeps track of what enabled events to emit is a small matter of design. I suspect that the best one is a directory somewhere, because creating and deleting files in a directory has proven to work well for other similar problems. A script to emit suitable events is then basically 'cd /etc/somewhere && for i in *; do initctl emit $i-enabled; done'.)

As a side effect this preserves all of the virtues of the current Upstart setup, such as being able to start services by hand with initctl (something that is not preserved by schemes that require symlinks or renaming files). It also needs no grand re-architecting of Upstart or the rest of the system, and could be rolled out easily and incrementally by any distribution that wants to (especially since distributions are already customizing Upstart service config files or writing them from scratch).

Sadly I suspect that the odds of Ubuntu or any other Upstart-using distribution adopting such a thing are relatively low. (Perhaps Fedora will give me a pleasant surprise.)

Why I am unhappy with Upstart right now

Upstart has a little problem: it erases the difference between something having a startup script and whether that script is activated or not. There is no equivalent in Upstart to the distinction between /etc/init.d and /etc/rc?.d; everyone puts startup configurations directly in /etc/init, where Upstart will process them. This matters quite a lot to me, because there are plenty of packages that we want installed without having their daemons be active.

Editing the /etc/init file is not a solution (even if it's done by a script), because the file is owned by the package, not the sysadmin. An Upstart .conf file contains a great deal more than whether or not a daemon is enabled (and under what circumstances); it also has how to start the daemon and what associated processing you need to do. This means that its contents are logically tied to the rest of the package and may need to be updated when the rest of the package is.

If you edit the file and the package is updated, either your changes are lost (and the daemon either starts running when you didn't want it to or stops running when you did) or you now get to merge your changes with the package's changes by hand. Neither is a good outcome when all you wanted to do was stop the MySQL daemon from running by default.

There's also a consequence for how people package daemons. It used to be that a lot of daemons were shipped in a default-disabled state, where after installation they were just available but you had to use chkconfig or the equivalent to turn them on. This was perfectly viable when we had a chkconfig (and GUI versions of it), but Upstart removes all that and makes it much more difficult and annoying to change the enabled/disabled state of a daemon. I suspect that the result will be a lot more daemons being installed default-enabled (if the packager has to pick an option, it's the more useful one), and in turn this means I'll be installing a lot less daemons.

What is going on with Samba's POSIX locking on NFS on Linux

Once we looked at the evidence from our Samba problem, it was relatively easy to come up with a broad theory of what was going wrong. All of the signs pointed to some sort of NFS-related locking clash where Samba was applying multiple locks that it (and the local kernel) did not think clashed with each other, but where the NFS server felt that they did clash and rejected one.

As far as I can see, this is indeed what is going on.

Current versions of Samba both flock() files and (if POSIX locking is enabled) apply fcntl() locks to them. This does not conflict for local files as the two sorts of locks are independent (and this is even documented in the flock(2) manpage).

In old versions of the Linux kernel, flock() didn't work at all over NFS; flock() locks were purely local (where they continued to not conflict with POSIX locks). In Linux 2.6.12, the Linux NFS client was changed to make flock() locks work over NFS by quietly converting flock() locks to server side POSIX locks; this conversion happens in the depth of the kernel NFS client, and the local kernel's general locking layer is unaware of it. Since two POSIX locks can obviously conflict with each other, this conversion means that from 2.6.12 onwards flock() locks now conflict with fcntl() locks on NFS filesystems.

Hence the common symptom of the problem: if you upgraded your system such that you crossed over the 2.6.12 version boundary, Samba's dual locking went from non-conflicting to conflicting (on NFS filesystems only) if the Windows client program made just the right sort of locking requests. Evidently Office 2003's 'save in Office 2003 format' code does so, and other programs do not.

(I believe that Samba takes a read flock() when clients open files, so from the symptoms it looks like Office 2003 was trying to acquire a write lock of some sort when you opened or saved the file. I can see why this was changed in subsequent versions of Office.)

Linux, Samba, NFS, and POSIX locking

We recently upgraded our Samba servers from an old version of Ubuntu and Samba to a new version of both. When we did this, we found that Windows XP machines running Office 2003 stopped being able to save documents in Office 2003 formats. Attempted saves would fail with a more or less generic error, and opening documents would stall for a while and then open them read only.

(Yes, it really is this specific. For example, Office 2003 can save the same files in other formats without problems.)

If you do appropriate searches, the Internet is (relatively) full of people reporting this same problem. Invariably they were upgrading a machine to a relatively modern Linux distribution, and they were using NFS-mounted filesystems (people with local filesystems on their Samba servers don't see this issue). The solution everyone winds up with is turning off POSIX locking in Samba:

posix locking = no

Unsurprisingly, this is what was required for us as well.

(We performed a series of experiments that established that turning off any other single locking-related option was not good enough, although we did not try all possible combinations of other Samba locking options.)

The stuff you find on the Internet stops there, but there is more to the story.

On the surface this sounds like an alarming option to have to turn off, especially in an environment (such as ours) where people can do stuff to their files without going through the Samba server. In practice, we happened to notice that it does not cause Samba to entirely stop doing locking, even locking over NFS. With POSIX locking turned off, locks were still visible both locally and on the NFS fileserver, and we verified things such as that a Unix program locking a target file (on another machine) would cause Office 2003 to stall and then open the file read-only.

(There are probably some things that don't work right, but we were happy enough to verify that general locking still worked.)

As it happens this was a big clue that led us to what is going on, but that's another entry.

PS: if you do this searching, you will also turn up a bunch of Samba people repeatedly saying that you should not use Samba to re-export NFS mounted filesystems. However well intentioned this advice is, it is completely infeasible in any production environment of significant size so ignore those people.