Wandering Thoughts archives

One irritation in xargs's interface

Xargs is generally a nice command that more or less works right. Some people could criticize Unix for needing it so much (which is mostly a product of command line length limitations) and the need for -0 is a bit annoying, but on the whole it's good. But xargs has one little corner case that is really annoying; as a bonus, it's even non-portable in an irritating way.

Here it is, presented in illustrated form:

$ xargs echo does run </dev/null

Now the question: will this produce any output? In other words, does xargs run the command once even if there are no (extra) arguments to give to it? The answer is that it does in some but not all versions of xargs:

• Solaris 10 runs echo once and has no option to disable this.
• GNU findutils xargs (commonly used on Linux) normally runs echo once but can turn this off with -r aka --no-run-if-empty.
• FreeBSD doesn't run echo and has no option to change this. Recent versions accept -r for compatibility with GNU xargs; old versions don't.
• OpenBSD runs echo once but can turn this behavior off with -r.
• Mac OS X doesn't run echo and has no -r argument.

Based on the current manpage, NetBSD xargs behaves the same as FreeBSD xargs (including accepting a do-nothing -r argument).

The Single Unix Specification for xargs is rather ambiguous about what behavior is allowed or required; it certainly never definitely states things either way (and it has no -r argument). My close reading leads me to believe that SuS probably requires xargs to run echo once, but only by implication. This would match what I believe is historical behavior (as suggested by Solaris, which is very historical). I assume that at some point FreeBSD decided that this historical behavior was a bad idea and changed it.

My view is that (historical) xargs behavior is stupid and is a bear trap waiting to bite you in unusual situations. You almost never want to run the xargs command even if there is nothing for it to operate on. In many situations and usages you'll get odd results if there is nothing to operate on; in extreme cases you may get dangerous explosions. This is an easy issue to overlook because everyone almost always uses xargs in situations that do generate arguments list (especially when you're testing your command lines or scripts). In fact I suspect that many people using xargs on Linux, Solaris, and OpenBSD machines don't even know about this potential gotcha, which sort of proves my point.

(This entry is yet another illustration of how a simple entry idea can turn out much more interesting than I expected when I started writing it. Before I started actually checking systems I would have confidently told you that all versions of xargs would run echo once; I had no idea how tangled the actual situation was.)

The FreeBSD iSCSI initiator is not ready for serious use (as of 9.1)

Our ZFS fileservers use iSCSI to talk to the backend disk storage, so if FreeBSD is to be a viable Solaris replacement for us its iSCSI initiator implementation needs to be up to the level of Solaris's (or Linux's). I recently did some basic testing to see if it looked like it was, and I'm afraid that it's not; as of FreeBSD 9.1, the iSCSI initiator seems suitable only for experimentation. In my testing I ran into two significant issues and one major issue, after which I stopped looking for further problems because it seemed pointless.

The first issue is clear in the FreeBSD iscsi.conf(5) manpage, which is simply full of 'not implemented yet' notes for various iSCSI connection parameters. Unfortunately a number of these are (potentially) important for good performance. This is the least significant issue, since I wouldn't really care about it if everything else worked (instead it would just be a vague caution sign).

The second issue is how the iSCSI initiator is managed and connections are established. Basically, FreeBSD provides absolutely no support for this and in particular there is no boot time daemon that you run to connect to all of your configured iSCSI initiators. Instead you get to somehow run one instance of iscontrol(8) per target, restarting any that die because their error recovery is (as the manual says) not 'fully compliant' or due to other issues. iscontrol has at least some bad limitations; in my experimentation, it would not start against a target that had no LUNs advertised (which is valid). I did not test its behavior if all LUNs of a target got deleted while it was running, but I wouldn't be surprised if it also exited (which would be a bad problem for us).

Both of these pale against the major issue, which was performance or the lack thereof. I'm not going to quote numbers for reasons I'll discuss later, but it was bad at all levels: in ZFS, in UFS, and just banging on the raw iSCSI disk. Streaming read performance was roughly 3/5ths of what Linux got in the same environment. Untuned streaming write performance was one tenth of the streaming read performance, but with drastically increased iSCSI parameters (not needed for Linux) FreeBSD managed to pull that up to 2/3rds of its read performance and 2/5ths of what Linux could get. Web searches turned up other people reporting catastrophically bad iSCSI write speeds (I suspect that they did not tune iSCSI parameters, which reduces it to merely bad), so I don't think that this is just me or just my test environment.

This level of (non)performance is a complete non-starter for us. It's so bad that there is no point in me spending more time to go beyond my quick experiment and basic tests. Under other circumstances I might have looked at the code and dug into things further to see if I could find some fixable defect, but I don't feel that there's any point here. The other issues make it clear to me that no one has run the FreeBSD iSCSI initiator in production (at least no one sane), and I have no desire to be the first person on the block to find all of the other problems it may have.

(The situation with iscontrol alone makes it clear that no one has exposed this to real usage, because no sane sysadmin would tolerate running their entire iSCSI initiator connection handling that way. I don't object to separate iscontrol instances; I do object to no master daemon and no integration with the FreeBSD startup system.)

(Also, you don't want to know how FreeBSD handles or in this case doesn't handle the various iSCSI dynamic discovery methods.)

All of this leaves me disappointed. I wanted FreeBSD to be a viable competitor and alternative, something that we could really consider. Now our options are much narrower.

(Well, I can always hope that the FreeBSD iSCSI initiator improves drastically in the next, oh, year, since we're not about to replace our current Solaris fileserver infrastructure right away. We've only just started to think about a replacement project; it may be two or three years before we actually need to make a choice and deploy.)

Sidebar: my test environment and cautions

At this point I will say it out loud: I was not testing FreeBSD on physical hardware. I discovered all of this during very basic tests in a virtual machine. Normally this would make even me question my results, but I did a number of things to validate them. First, I tested (streaming) TCP bandwidth between the FreeBSD VM and the iSCSI backend (which is on real hardware) and got figures of close to the raw wire bandwidth; I can be reasonably sure that the FreeBSD VM was not having its network bandwidth choked by the virtualization system. Second, I also ran a Linux VM in the same virtualization environment and measured its performance (network and iSCSI). As noted above, it did significantly better than FreeBSD did (despite actually having less RAM allocated to it).

It's always possible that FreeBSD iSCSI is choking on something about the virtualization environment that doesn't affect its raw TCP speed or Linux. My current view is that the odds of this are sufficiently low (for various reasons) that it is not worth the hassle of spinning up a physical FreeBSD machine just to be sure.

(Partly this is because I found other people on the Internet also complaining about the FreeBSD iSCSI write speeds. If I was the sole person having problems, I would suspect myself instead of FreeBSD.)

I suppose the one quick test I should do is to feed the FreeBSD VM a whole lot more memory to see if that suddenly improves both read and write speeds a whole lot. But even if FreeBSD had Linux-level read and write performance, the other significant issues would probably sink it here.

A dive into the depths of yes `yes no`

The blog entry of the current time interval is m. tang's yes `yes no`, which winds up exploring just that; as the author says, doing it 'sort of slowed my computer below the threshold of usefulness, so I had to restart it'. Unfortunately the author's original explanation that the second, outer yes buffers the output of yes no endlessly (eating up all memory in the process) is totally wrong (as various people have noted and corrected since the entry started going around). As it happens I think that there are some interesting things hiding under the covers here, so I'm going to talk about them.

First off, let's understand why this command line probably explodes your system. To be clear I'll rewrite this in a more modern but less aesthetically looking shell syntax and call it 'yes $(yes no)'. This is more or less equivalent to:

shvar=$(yes no)
yes $shvar

Just the first line alone is enough to blow up your shell because it asks the shell to read an endless amount of input and try to hold it in memory (here in the form of a shell variable). The same thing happens in the original command line, just without the intermediate shell variable.

You might wonder why the shell doesn't have some limit on how much input it's willing to read this way. While this is a self-inflicted accident, it's not as if Unix machines really deal well with running out of memory; on a 64-bit machine you could easily blow up the entire system doing this (on a 32-bit machine you might run into per-process address space limits before then). Saving you from this would be at least somewhat nice. I suspect that the real answer basically boils down to 'tradition'; this is such a rare (and self-inflicted) situation that no shell has bothered to deal with it yet and since no shell has, not dealing with it has become the default.

(Unix has a great deal of this sort of 'someone else did it this way first' historical practice that has basically fossilized over the years. Even if it doesn't necessarily entirely make sense it's often easier for people who are reimplementing commands to just go with existing (lack of) practice. Part of this ties into the social problems involved with changing things in Unix.)

Beyond that, though, there are some issues with having a limit. First you have to decide on the semantics of what happens when the limit is hit. Do you discard the output entirely or truncate it? Do you count this as a failure for the purposes of set -e or do you pass on the exit status of the 'yes no', whatever that is (and it may not be a failure)? In the case of 'yes $(yes no)' do you even try to run the second yes (with a truncated or empty argument list) or do you fail the entire command on the spot? There are arguments either way for much of this (and the choices interact with each other); you'll have to figure out what's the most useful answers in practice, whether your proposed change breaks any existing script practices, and then how much POSIX lets you get away with (if you care about being a POSIX-compatible Bourne shell; things like zsh have it easier here).

Then there's the issue of what the limit should be. We don't want to just limit the shell to the kernel's exec() limit (which is on the combined size of the arguments and the environment); it's valid to simply read a lot of output into an unexported shell variable and then process it. In fact in some situations this is how you deal with an 'arguments too big' problem. So what do you set? People are probably going to complain about almost any value.

(I suppose the real answer is to have the limit be user-settable. You could even start out with the limit available but default to unlimited, then a year or two later introduce a default limit.)

Sidebar: $() and set -e today

Since I just experimented with this:

set -e
echo 1 $(false)
v=$(false)
echo 2

This will echo '1' but not '2' in Bash, dash, ksh, FreeBSD's sh, and Solaris 10's /usr/xpg4/bin/sh, which makes me assume that this is actually what POSIX requires. Just to be different, Solaris 10's /bin/sh doesn't echo anything (even if you flip the order around).

(I have not been masochistic enough to obtain and boot a PDP 11 V7 image just to see what the V7 sh would do, but I suspect it's the same as Solaris /bin/sh.)