Wandering Thoughts archives

Two uses of fmt

The venerable fmt program is not something that I normally think of as a command that I use in my scripts or on the fly pipelines; I usually think of it more as, say, something that I use to reflow paragraphs in vi. But it has quietly been making its way into an increasing number of them because it turns out that fmt is the easy and lazy way to do two symmetrical things: to word-split lines and to merge 'one word per line' output back to a single line.

Word splitting is straightforward:

somecmd | fmt -1

Every version of fmt that I've seen accepts this and does the right thing; you get one word per line after fmt.

Word joining is much more annoying and it's all because of GNU coreutils. Real versions of fmt accept very large format widths:

/bin/ls -1 | fmt -9999

Unfortunately the GNU coreutils version of fmt has a maximum width of 2500 characters. Worse, it has a 'goal' width that defaults to 93% of the actual width, so if you're worried about getting close to that limit you need to use:

/bin/ls -1 | fmt -2500 -g2500

In practice I usually use 'fmt -999' in command pipelines because my actual output line is going to be nowhere near 999 characters to start with.

(Usually when I'm doing word merging it's because I'm going to take the one line that results and paste it to something else, which imposes a relatively modest line length limit in practice.)

What this points out is that fmt is not really the ideal solution to this (and in fact the FreeBSD version of fmt also has oddities, such the man page's description of behavior around the -n switch). The traditional Unix solution to these problems is tr, using it to either turn spaces to newlines or newlines to spaces. The problem for me in practice is that to use tr I need to remember or re-derive the octal value of newline (it's \012, by the way) and that is just a bit too much hassle. So I use fmt, vague warts and all.

(The other drawback of tr is that 'tr " " "\012"' will have a trailing space and no final newline. Usually this is not a big deal.)

Actually in writing this I've discovered that I'm behind the times. All of the versions of tr that I use today will accept \n instead of the octal literal. Either there was a day when this wasn't true or I just never read far enough in the tr manpage (and had it stick) to notice that particular feature. (I'm probably still going to keep on using fmt, though.)

My growing entanglement into vi

It started with vi becoming my sysadmin's editor, the editor that I used for quick edits because it was everywhere, worked in minimal environments, and it started fast. But of course it didn't stop there. Any good tool has a virtuous circle where more use makes you more familiar with it and thus makes it easier to use so you use it more; vi goes well beyond that in terms of rewarding extended use. Vi's march into my editing life has not been fast but it's feeling more and more relentless as time goes by, especially when I do things like specifically configure git to use vi instead of my normal default. I'm not using vi pervasively quite yet, but increasingly my major holdout (writing email in my full email environment) feels a little bit awkward.

(My normal default $EDITOR is a script that tries to intelligently pick the editor to use based on my current environment based on things like whether or not I have X available.)

This has not fundamentally changed my view of vi as a whole (it remains not my favorite editor). I am simply being seduced by convenience and familiarity, and running into the limits and issues in my major other editor. Not that vi is bad (rather the contrary), but I still miss things from my other editors and often would sort of prefer to be using them.

(Possibly this attachment to my major other editor is just emotion speaking.)

While I've been additional learning vi (well, vim) features slowly over time, I still have not really attempted to become solidly familiar with Vim's advancements over the core vi editing commands (I'm going to wave my hands about the reasons why, but see above about vi still not being my favorite editor). If I get more seriously into vi, and it seems inevitable that I will, I should probably change that. My obvious weak areas are the areas where vi itself is weak: working fluidly with multiple files and also with split screens for editing two files simultaneously. Mastering doing this in Vim would remove one significant reason that I revert to other editors.

(I will probably always edit Python, C, and Go code in GNU Emacs when I have a choice. But there is a fair amount of other multi-file work that would at least be more convenient if I knew what I was really doing in Vim.)

I know that Vim has a universe of advanced text movement and text manipulation commands but I'm honestly not sure that I feel much interest in learning them. The mere fact that there is a universe of them is kind of daunting and I have no confidence that they'd speed up the sort of editing work that I do very much. Probably some of them would, so I suppose I should at least glance over the list to see if anything stands out.

(This has come out more rambling and thinking aloud than I thought it would. I do think that there's something interesting about how vi has wormed its way into my computing life as more and more the editor I reach for, but I don't have the words for it right now.)

Making large selections in xterm (and urxvt and Gnome Terminal)

Suppose that you have a large chunk of output in a terminal window, specifically more than a full screen's worth, and you want to copy it into an email message, text file, or however else you may be logging it for the record. As I knew vaguely but had never really read up on or used until very recently, it turns out that there is a convenient way to do this in xterm. Specifically, this is what the right mouse button is for; it extends the selection from where it is until the current point.

So in xterm what you do to select a huge selection is select a bit right at one end (the start or the end), scroll to the other end, and carefully hit the right mouse button at where you want the selection to end. The selection is instantly extended. You can do this several times if you want, extending the selection each time. Odder and less easily controlled things happen if you hit the right mouse button somewhere inside the selection.

This doesn't work in Gnome Terminal. Instead what you have to do is start the selection with the left mouse button and while making it, drag the mouse cursor to the edge of the window (or outside the window). G-T will scroll things for you, extending the selection in the process. G-T's scrolling is sufficiently rapid that this is a reasonably convenient and intuitive process, arguably better than xterm's.

Urxvt gives you both options; you can extend the selection explicitly with the right mouse button or let urxvt scroll things for you in the same way as Gnome Terminal. The one drawback is that urxvt by default scrolls inconveniently slowly (and there doesn't seem to be any way to control this from what I can see in the manual). You can scroll with a mouse scrollwheel and it works reasonably well although a bit jumpily in my quick test.

(Xterm doesn't scroll at all if you drag the mouse out of the window while you make a selection.)

In a brief test, KDE's Konsole works the same way as Gnome Terminal. I suspect that this is going to be the common behavior of more or less all modern 'smart' terminal emulators because it makes the most sense and it's relatively discoverable (unlike the right mouse button in xterm).

The three levels of read-only NFS mounts

It's sometimes useful to understand that there are three ways that an NFS mounted filesystem can be 'read-only'. Let's call them three levels:

• You can mount the NFS filesystem read-only on the client. The client kernel will then enforce this, disallowing write actions and so on. These days this is generally mostly handled in high level VFS code, since it's common behavior across filesystems.

  As with all remote filesystems, this read-only status is purely local to your client machine. Your machine doesn't get to order the NFS server not to make any changes on the filesystem (that would be laughable) so the NFS server is perfectly entitled to allow the filesystem to change underneath you and to have other clients mount it read-write (and write to it). If NFS is working right, you will see those changes at some point.

• The server can export the NFS filesystem read-only (either to you or just in general). The NFS server code will then disallow all write actions that clients send it, returning an appropriate 'read only filesystem' error to errant clients (if any). Even if the NFS mount is exported read-only to all clients, it's still valid for the exported filesystem to be changed locally on the NFS server.

  (As far as I know, whether or not the NFS export is read-only is invisible to the client. It's purely something internal to the server and can even change on the fly.)

• On the server you can mount the exported filesystem read-only (or otherwise set it that way). On competent NFS servers this disallows all writes to the filesystem, regardless of whether they're NFS or local and regardless of whether the filesystem was exported read-only by the NFS server.

  (On competent NFS servers, all NFS server operations on the exported filesystem go through the VFS et al and so have the standard handling of read-only mounts applied to them automatically.)

These can certainly be stacked on top of each other (a read-only server filesystem, NFS exported as read-only and mounted as read-only on clients) but they don't have to be. For instance you can NFS export filesystems as read-only but mount them read-write on clients (we do this here for complex reasons).

Now let's talk about atime and atime updates. In NFS, atime updates are the responsibility of the server, not the clients. More specifically they are generally the responsibility of the underlying server filesystem code or VFS, not specifically the NFS server code, and as such they can happen when you read data through a read-only NFS mount or even a read-only NFS export. The NFS clients asks to read data, the NFS server code makes a general VFS 'get me data' call, and as a side effect of this the VFS or the filesystem updates the atime (if atime updates are enabled at all).

(This implies that not all client reads necessarily update the server atime, because a client may satisfy a read from its own file cache instead of going to the server.)

If you think about it this is actually a feature. If you have atime enabled on a read-write filesystem mount, you have told the (server) kernel that you want to know when people read data from the filesystem and lo, this is exactly what you are getting. The read-only NFS export is just to tell the NFS server that it should not allow people to do 'write' VFS operations.

(Since you can export the same filesystem read-write to some clients and read-only to others, suppressing atime updates on read-only NFS exports could also produce odd effects. Read a file from client A and the atime updates, read the file from client B and it doesn't. And all because you didn't trust client B enough to let it actually make (filesystem level) changes to your valuable filesystem.)

Sidebar: NFS exporting of read-only filesystems

You might think that the NFS export process should notice when it's exporting a read-only filesystem as theoretically read-write and silently change it to read-only for you. One of the problems with this is that on many systems it's possible to switch filesystems back and forth between read-only and read-write status through various mechanisms (not just mount). In practice you might as well let the NFS server accept the write operations and have the VFS then reject them; the outcome is the same while the system is simpler and behaves better in the face of various things happening.

The three faces of sudo

For reasons beyond the scope of this entry I've recently been thinking about my attitudes towards sudo. Sudo is a complex program with a lot of options and several different ways of using it, and in the process of my thinking I've realized that for me it's effectively three things in one (and I feel differently about each facet). So here are my three faces of sudo:

1. sudo as a replacement for having specific setuid programs. You're using it to give (passwordless) access to something for some group of people (or everyone); instead of writing a setuid program you use sudo to run a non-setuid program or script with the necessary privileges. Often you may want to wrap the sudo invocation up in a cover script so you can tell people 'just run /some/script'.

2. sudo as a way of giving non-sysadmin staff limited and guarded access to especially privileged and dangerous operations. This is the traditional 'operators are allowed to run reboot' situation, which I'll summarize as 'restricted root powers'. Here the people using sudo are not full sysadmins and are not trusted to hold unrestricted root privileges.

3. sudo as the way you access unrestricted root privileges, where use of sudo replaces su. You're encouraged to use sudo to run specific commands (even a bunch of commands) instead of using it to just get a root shell and then doing stuff from there.

   (In practice, use of sudo this way temporarily turns your current shell session into a peculiar privileged hybrid environment where you can use root powers casually by prefixing a command with sudo.)

I think that there are lots of uses for sudo as a replacement for setuid programs. Setuid programs are hard to write securely and can only be written in a few languages. Using sudo lets you more or less safely write 'setuid' programs in, say, shell scripts or Perl or the like. Invocation of them is a bit funny (you have to say 'sudo <program>') but that can be hidden by a cover script. We use this here for a number of things (eg) and it works great.

I'm less sanguine about sudo as a way to give out restricted root powers, especially if you let people run ordinary programs instead of only custom-designed scripts. Unless you're very careful it's easy to accidentally give people a way out of your restricted jail, since programs are generally not designed to enforce a restricted environment and contain all sorts of odd holes. For instance, if you allow people to run 'vi /some/file' as root you've just given them full root access if they want it. The whole area is a massive minefield if you're faced with an attacker.

(This doesn't require your operators to be malicious. Unfortunately you've turned compromising an operator account into a great path towards root access.)

My feelings about sudo as a replacement for su are sufficiently complicated that they don't fit in this entry. The short version is that I think you're likely to be creating a different security model with different risks; how different they are depends on how you configure sudo. The more you make the risks of sudo match the risks of su, the more you turn sudo into su.