Wandering Thoughts: Recent Entries For 2012/06/22

How the Linux kernel command line is processed (more or less)

Because I just had to research this (right down to reading kernel source), here is how the Linux kernel handles its command line arguments.

(The kernel command line is set in the bootloader in various ways. Grub lets you edit it on the fly, assuming that you can interrupt grub in time before it autoboots your kernel.)

First, the kernel goes through all kernel options and handles them. In theory all of the kernel options are documented in Documentation/kernel-parameters.txt in the kernel source, but beware: on a modern system that boots using an initial ramfs, a number of these options are really handled by the initial boot process instead of the kernel.

(For example, root=... and init=... are handled by the initial boot process, because the initramfs /init is what actually mounts the root filesystem and starts the real init.)

For anything that is not a kernel option and does not have a '.' in its name (these are assumed to be unhandled module parameters), one of two things happen. If it is of the form 'a=b' or (I believe) 'a=', it's placed in the environment that will be passed to the initial user-level process (generally either /init from the initial ramfs or /sbin/init). If it doesn't look like an environment variable setting it becomes one of the command line arguments for the initial user-level process. Everything from the kernel command line also appears in /proc/cmdline.

Generally the initial user-level process then immediately reads and pseudo-parses /proc/cmdline, pulling out various options that it cares about. Depending on your distribution, it may or may not pay any attention to its command line arguments (and for an initial ramfs /init, may or may not pass them to /sbin/init). Generally they are at most passed to /sbin/init; initial ramfs processing usually prefers to read everything from /proc/cmdline (partly because everything winds up there regardless of its specific form).

(I say 'pseudo-parses' because /init may or may not handle things like quoted arguments.)

Some but not all distributions make use of the initial environment variables; for example, Fedora sets $LANG right from the moment of boot this way.

Sidebar: what some distributions do with init command line arguments

All of these are for when you have an initial ramfs with its own /init.

• Ubuntu 10.04's and 12.04's /init pass command line arguments to Upstart's /sbin/init but otherwise ignores them. What options /init accepts is documented in the initramfs-tools manpages.

• Fedora 17's /init appears to completely ignore command line arguments and does not pass them to systemd's /sbin/init. However, it appears that you can specify arguments in an init= option along with the program to run (unlike Ubuntu, where it can only be an executable). What options /init accepts is documented in the dracut manpage.

  (I expect that this is true of Fedora 16 and earlier but I lack the energy to dig up a Fedora 16 initrd image, unpack it, and carefully read through its /init just to be sure.)

• Red Hat Enterprise 5 is sufficiently old that I think it uses a somewhat different scheme to transition from the initial ramdisk to your root filesystem. Its initrd /init certainly doesn't seem to do anything with command line arguments.

These days initial ramdisk images are gzip'd CPIO archives, so they can be extracted with:

mkdir /tmp/unpack
cd /tmp/unpack
zcat /boot/init<whatever>.img | cpio -id

Reading /tmp/unpack/init is the most interesting thing to do, although it may be relatively opaque.

The effects of DTrace's problems

A while back, Brendan Gregg left a comment on my entry about why we haven't taken to DTrace. In one part, he asked:

I was a little confused at first about the language and documentation issues [with DTrace]. Usually language discussions [...] are intended to pick one over another, but in this case there is no other option to choose from. So if these problems are raising the barrier to entry for some people, and they aren't entering, then what are they doing? Leaving system problems unsolved?

There are two answers to this.

The first answer is that DTrace is two things at once; it is both a way of diagnosing problems on Solaris and a potential way of attracting people to Solaris (and all its variants), to continue to use it, and to use it for more things. Let us focus on the latter thing for the moment. When and where DTrace is hard to use it becomes less attractive; at the limit, if you feel that you can't really use DTrace for anything it ceases to become an advantage for Solaris at all. If you want Solaris to succeed as an OS, this should matter.

(I think that the theoretical advantages of having DTrace may have been oversold in general. As a pragmatic matter I think that most people don't expect to have system problems (they expect the system to just work), so I suspect that they drastically discount the availability of good diagnosis tools because they expect to not need any. People who know that they are running at the ragged edge of performance will have a different opinion, but many people are not in this situation.)

The second answer, put simply, is yes; sysadmins are leaving system problems unsolved because DTrace is too hard to use. Not the big crippling problems, of course, because those are the problems you have no choice about solving. But smaller problems, the little glitches that happen sometimes or the relatively low impact performance degradations? Yes, some of them are going unsolved. Also going unsolved are the problems that people don't even know they have because they've never looked, the ones where people have no idea that something is actually wrong and their system could work better with some changes. DTrace being hard(er) to use is especially damaging to the latter because of course if you don't think you have a problem, the cost to benefit ratio of looking into your system appears infinite.

(I've argued that this is not actually the case, but I think it's at least a very hard thing to sell. Especially to overworked sysadmins with other issues to tackle when you are asking them to invest a significant chunk of time.)