Wandering Thoughts archives

'Command line text editor' is not the same as 'terminal-based text editor'

A while back, I saw a mention about what was called a new command line text editor. My ears perked up, and then I was disappointed:

Today's irritation: people who say 'command line text editor' when they mean 'terminal/cursor-based text editor'.

I understand why the confusion comes up, I really do; an in-terminal full screen editor like vi generally has to be started from the command line instead of eg from a GUI menu or icon. But for people like me, the two are not the same and another full screen, text based editor along the lines of vi (or nano or GNU Emacs without X) is not anywhere near as interesting as a new real command line text editor is (or would be).

So, what do people like me mean by 'command line text editor'? Well, generally some form of editor that you use from the command line but that doesn't take over your terminal screen and have you cursor around it and all that. The archetype of interactive command line text editors is ed, but there are other editors which have such a mode (sam has one, for example, although it's not used very much in practice).

Now, a lot of the nominal advantages of ed and similar things are no longer applicable today. Once upon a time they were good for things like low bandwidth connections where you wanted to make quick edits, or slow and heavily loaded machines where you didn't want to wait for even vi to start up and operate. These days this is not something that most people worry about, and full screen text editors undeniably make life easier on you. Paradoxically, this is a good part of why I would be interested in a new real command line editor. Anyone who creates one in this day and age probably has something they think it does really well to make up for not being a full screen editor, and I want to take a look at it to see this.

I also think that there are plausible advantages of a nice command line text editor. The two that I can think of are truly command line based editing (where you have commands or can easily build shell scripts to do canned editing operations, and then you invoke the command to do the edit) and quick text editing in a way that doesn't lose the context of what's already on your screen. I imagine the latter as something akin to current shell 'readline' command line editing, which basically uses only a line or two on the screen. I don't know if either of these could be made to work well, but I'd love to see someone try. It would certainly be different from what we usually get.

(I don't consider terminal emulator alternate screens to be a solution to the 'loss of context' issue, because you still can't see the context at the same time as your editing. You just get it back after you quit your editor again.)

You can call bind() on outgoing sockets, but you don't want to

It started with some tweets and some data by Julia Evans. In the data she mentions:

and here are a few hundred lines of strace output. What's going on? it is running bind() all the time, but it's making outgoing HTTP connections. that makes no sense!!

It turns out that this is valid behavior according to the Unix API, but you probably don't want to do this for a number of reasons.

First off, let's note more specifically what Erlang is doing here. It is not just calling bind(), it is calling bind() with no specific port and address picked:

bind(18, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16 <unfinished ...>

Those arguments are INADDR_ANY and the magic port (port 0) that tells bind() that it can pick any ephemeral port. What this bind() does is assign the socket a local ephemeral port (from whatever the ephemeral port range is). Since we specified the local address as INADDR_ANY, the socket remains unbound to any specific local IP; the local IP will only be chosen when we connect() the socket to some address.

(This is visible in anything that exposes the Unix socket API and has a getsockname() operation. I like using Python, since I can do all of this from an interactive REPL.)

There really isn't very much point in doing this for sockets that you're going to use for outgoing connections; about all it achieves is letting you know your local port before you make the connection, instead of only afterwards. In exchange for this minor advantage you make one extra system call and also increase your chances of running out of ephemeral ports under load, because you're putting an extra constraint on the kernel's port allocation.

In general, IP requires each connected socket to have a unique tuple of (local IP, local port, remote IP, report port). When you leave an outgoing socket unbound until you connect(), the kernel has the maximum freedom to find a local port that makes the tuple unique, because all it needs is one of the four things to be unique, not necessarily the local port number. If you're connecting to different ports on a remote server, the same port on different remote servers, or whatever, it may be able to reuse a local port number that's already been used for something else. By contrast, if you bind() before you connect and use INADDR_ANY, the kernel pretty much has the minimum freedom; it must ensure that the local port alone is completely unique, so that no matter what you then do with listen() and connect() later you'll never collide with an existing tuple.

(See this article for a discussion of how the Linux kernel does this, and in fact this entire issue.)

Some Unixes may frontload all of the checks necessary into bind(), but at least some of them defer some checks to connect(), even for pre-bound sockets. This is probably a sensible decision, especially since a normal connect() can fail because of ephemeral port exhaustion.

I'm sure there's some advantage to this 'bind before connect' approach, but I'm honestly hard pressed to think of any.

(There are situations where you want to bind to a specific IP address, but that's not what's happening here.)

(I sort of always knew that it was valid to bind() before calling connect() but I didn't know the details, so writing this has been useful. For instance, before I started writing I thought maybe the bind() picked an IP address as well as the ephemeral port, which turns out not to be the case; it leaves the IP address unbound. Which is really not that surprising once I think about it, since that's what you often do with servers; they listen to a specific port on INADDR_ANY. All of which goes to show that sometimes it's easier for me to experiment and find out things rather than reason through them from first principles.)

The difference between 'Delete' and 'Destroy' in X window managers

If you use a certain sort of window manager (generally an old school one), you may have discovered that there are two operations the window manager supports to tell X applications to close themselves. Names for these operations vary, but I will go with 'Delete' and 'Destroy' for them because this is fvwm's terminology. Perhaps you've wondered why there are two ways to do this, or what the difference is. As you might guess, the answers are closely tied to each other.

The one way to describe the difference between the two operations is who takes action. When your window manager performs a 'Delete' operation, what it is really doing is sending a message to the application behind the selected window saying 'please close yourself' (specifically it is sending a WM_DELETE_WINDOW message). This message and the whole protocol around it is not built in to the X server and the wire protocol; instead it is an additional system added on top.

(Potentially confusingly, this 'please close your window' stuff is also called a protocol. People who want to see the details can see the XSetWMProtocols() manpage and read up on client messages. See eg this example, and also the ICCCM.)

On the other hand, the 'Destroy' operation talks directly to the X server to say either 'destroy this window' or more commonly 'terminate the connection of the client responsible for this window'. Unlike 'Delete', this requires no cooperation from the client and will work even if the client has hung or is deciding to ignore your request to close a window (perhaps the client believes it's just that special).

Generally there are big side effects of a 'Destroy'. Since most programs only create a single connection to the X server, having the server close this connection will close all of the program's windows and generally cause it to immediately exit. Even if only a single window is destroyed, this usually causes the program to get X errors and most programs exit the moment they get an X error, which of course closes all of the program's windows.

How programs react to being asked politely to close one of their windows varies, but usually if a program has multiple windows it won't exit entirely, just close the specific window you asked it to. Partly this is because the 'close window' button on the window frame is actually doing that 'Delete' operation, and very few people are happy if a program exits entirely just because you clicked the little X for one of its windows.

Because 'Delete' is a protocol that has to be handled by the client and some clients are just that old, there are or at least were a few core X clients that didn't support it. And if you write an X client from scratch using low-level libraries, you have to take explicit steps to support it and you may not have bothered.

(To be friendly, fvwm supports a 'Close' operation that internally checks to see whether a window supports 'Delete' and uses that if possible; for primitive clients, it falls back to 'Destroy'. I suspect that many window managers support this or just automatically do it, but I haven't looked into any to be sure.)

Sidebar: 'Delete' and unresponsive clients

It would be nice if your window manager could detect that it's trying to send a 'Delete' request to a client that theoretically supports it but isn't responding to it, and perhaps escalate to a Destroy operation. Unfortunately I don't know if the window manager gets enough information from the X server to establish that the client is unresponsive, as opposed to just not closing the window, and there are legitimate situations where the client may not close the window itself right away, or ever.

(Consider you trying to 'Delete' a window with unsaved work. The program involved probably wants to pop up a 'do you want to save this?' dialog, and if you then ignore the dialog everything will just sit there. And if you click on 'oops, cancel that' the whole situation will look much like the program is ignoring your 'Delete' request.)

I believe that some window managers do attempt to detect unresponsive clients, but at most they pop up a dialog offering you the option to force-close the window and/or client. Others, such as fvwm, just leave it entirely to you.

The state of supporting many groups over NFS v3 in various Unixes

One of the long standing limits with NFSv3 is that the protocol only uses 16 groups; although you can be in lots of groups on both the client and the server, the protocol itself only allows the client to tell the server about 16 of them. This is a real problem for places (like us) who have users who want or need to be in lots of groups for access restriction reasons.

For a long time the only thing you could was shrug and work around this by adding and removing users from groups as their needs changed. Fortunately this has been slowly changing, partly because people have long seen this as an issue. Because the NFS v3 protocol is fixed, everyone's workaround is fundamentally the same: rather than taking the list of groups from the NFS request itself, the NFS server looks up what groups the user is in on the server.

(In theory you could merge the local group list with the request's group list, but I don't think anyone does that; they just entirely overwrite the request.)

As far as I know, the current state of affairs for various Unixes that we care about runs like this:

• Linux has long supported an option to rpc.mountd to do this, the -g aka --manage-gids option. See eg Kyle Anderson's Solving the NFS 16-Group Limit Problem for more details. I have to optimistically assume that it's problem free by now, but I've never tried it out.

• Illumos and thus OmniOS gained support for this relatively recently. There is some minor system configuration required, which I've covered in Allowing people to be in more than 16 groups with an OmniOS NFS server. We have tested this but not yet run it in production.

• FreeBSD has apparently supported this only since 10.3. To enable it, you run nfsuserd with the new -manage-gids flag, per the 10.3 nfsuserd manpage. I suspect that you need to be using what FreeBSD calls the new NFS server with v4 support, not the old one; see the nfsd manpage.

• Oracle Solaris 11.1 apparently supports this, as reported by David Magda in a comment here. See Joerg Moellenkamp's blog entry on this.

I care about how widespread the support for this is because we've finally reached a point where our fileservers all support this and so we could start putting people in more than 16 groups, something that various parties are very much looking forward to. So I wanted to know whether officially adding support for this would still leave us with plenty of options for what OS to run on future fileservers, or whether this would instead be a situation more like ACLs over NFS. Clearly the answer is good news; basically anything we'd want to use as a fileserver OS supports this, even the unlikely candidate of Oracle Solaris.

(I haven't bothered checking out the state of support for this on the other *BSDs because we're not likely to use any of them for an NFS fileserver. Nor have I looked at the state of support for this on dedicated NFS fileserver appliances, because I don't think we'll ever have the kind of budget or need that would make any of them attractive. Sorry, NetApp, you were cool once upon a time.)