Wandering Thoughts

Trying out learning more Vim on demand

These days, I have a reasonably active interest in learning more Vim. Mostly I've done it by reading articles about Vim (eg, and I'm keeping an index). Sadly, a lot of the time this reading doesn't really stick. Recently I have been trying another approach, one that feels really obvious. When I want to do something, such as capitalize uppercase a number of words, I'll go do an Internet search for the answer (here [vim capitalize uppercase word], which immediately leads to a stackoverflow answer that's actually summarized right on the results page of the dominant search engine).

(I don't search for everything. Some things I want to do strike me as either too infrequent or too complex.)

The advantage of this is that it is what you could call highly motivated learning (I want to do this) and I reinforce it right away by using what I just read. If it doesn't entirely stick and this is something I do frequently enough, I'll soon re-do the search, re-read what I already read, and reinforce my memories. Perhaps sooner or later it will stick. If it's not something I do regularly it'll probably fade away, and I haven't wasted too much time on the search instead of doing whatever operation by hand (such as cw'ing the word and retyping it in CAPS).

There are two drawbacks to this approach that I can see (and probably some that I don't). The first is that this works much better on operators (such as gU to uppercase something) than on motions (in common vim jargon I believe these are called 'verbs' and 'nouns' respectively). It's natural to want to do an operation while not knowing how to; it feels less natural to want to move around or select in a new and unusual way. Second (and relatedly), this only works for things that I think are likely to be within Vim's capabilities. If I have no idea that Vim might even do something, I'm not going to try to look for it.

(An example of both at once is my stumbling over visual mode. I'm pretty sure it would never have struck me to do an Internet search for it.)

The second issue suggests that I should keep reading about general Vim features, so I have a broad idea of what Vim is capable of. Then I have a better chance of realizing that Vim probably has a way of doing the specific thing I care about right now, and looking it up.

My view of Wayland here in 2021

Somewhat recently there was some Wayland related commotion in the Unix tech circles that I vaguely follow, where Wayland people were unhappy about what anti-Wayland people are doing (for example, which may have been reacting to this). Somewhat in reaction to all of this, I had a Twitter reaction:

My Wayland hot take is that I have no idea how well Wayland works in general, but I do know that for me, switching to it will be very disruptive because I'll need a new window manager and none of the Wayland ones work like my fvwm setup does. So I am and will be clinging to X.

I can't say anything about what modern Wayland is like, because I have no personal experience with Wayland. On my work laptop I use Cinnamon, which doesn't support Wayland. On my home and work desktops, I use a highly customized X environment that would not port to Wayland.

At this point Wayland has been coming for more than ten years and has nominally been definitely the future of Unix graphics for four years. But it's fully supported on all Linux graphics hardware by only two environments (GNOME and KDE, see Debian's description of the hardware requirements). Only two of the five significant Linux desktop environments support Wayland on any hardware (GNOME and KDE, with Cinnamon, XFCE, and MATE not). Canonical is only just releasing a version of Ubuntu that uses Wayland for GNOME by default (21.04), and even then it may not do this on Nvidia GPUs (cf). There are some additional Wayland 'window managers' (compositors) such as Sway, but nothing like the diversity that exists on X (although there may be good reasons for this).

Today, it's not so much that people are refusing to use Wayland, it's that a lot of people cannot. If you're not using GNOME or KDE, you're pretty much out in the cold. If you're using an Nvidia GPU, you're probably out in the cold (even if you use GNOME, your Linux probably defaults to Xorg on your hardware). If you don't use Linux, you're definitely out in the cold.

It's true that X server development has more or less stopped (eg, also), although the X server 21.1 milestone seems small right now. It's also true that the current X server works and supports a broad range of desktop environments and Unixes, and plenty of people are using those.

PS: I don't like Nvidia GPUs either and I don't use them, but a lot of Unix people have them and some of those people don't have any choice, for example on some laptops (or people who need to do GPU computation). And they work in X.

Sidebar: Some Wayland references

See the Debian wiki page on supported and unsupported desktop environments, toolkits, and so on. Debian has been using Wayland for GNOME since Debian 10, released in 2019, at least on non-Nvidia hardware. Their Wayland and Nvidia wiki pages are unclear if their GNOME defaults to Wayland even on Nvidia hardware, but I suspect not. The Arch wiki has a list of Wayland compositors, but no information on how usable they are. The Gentoo "Wayland Desktop Landscape" may be helpful, especially since it rates some of the available compositors.

Why NFS servers generally have a 'reply cache'

In the beginning, NFS operated over UDP, with each NFS request and each NFS reply in a separate UDP packet (possibly fragmented). UDP has the charming property that it can randomly drop arbitrary packets (and also reorder them). If UDP drops a NFS client's request to the server, the NFS client will resent resend it (a 'retransmit' in the jargon of NFS). If UDP drops the server's reply to a client's request, the client will also resend the request, because it can't really tell why it didn't get a reply; it just knows that it didn't.

(Since clients couldn't tell the difference between a sufficiently slow server and packet loss, they also reacted to slow servers by retransmitting their requests.)

A lot of NFS operations are harmless to repeat when the server's response is lost. For instance, repeating any operation that reads or looks up things simply gives the client the current version of the state of things; if this state is different than it was before, it's pretty much a feature that the client gets a more up to date version. However, some operations are very dangerous to repeat if the server response is lost, because the result changes in a bad way. For example, consider a client performing a MKDIR operation that it's using for locking. The first time, the client succeeds but the server's reply is lost; the second time, the client's request fails because the directory now exists, and the server's reply reaches the client. Now you have a stuck lock; the client has succeeded in obtaining the lock but thinks it failed and so nothing is ever going to release the lock.

(This isn't the only way NFS file-based locking problems can happen.)

To try to work around this issue, NFS servers soon introduced the idea of a "reply cache", which caches the NFS server's reply to various operations that are considered dangerous for clients to repeat. The hope and the idea is that when a client resends such a request that the server has already handled, the server will find its reply in this cache and repeat it to the client. Of course this isn't a guaranteed cure, since the cache has a finite size (and I think it's usually not aware of other operations that might invalidate its answers).

In the days of NFS over UDP and frequent packet loss and retransmits, the reply cache was very important. These days, NFS over TCP uses TCP retransmits below the level that the NFS server and client see, so sent server replies are very hard to lose and actual NFS level retransmissions are relatively infrequent (and I think they're more often from the client deciding that the server is too slow than from actual lost replies).

In past entries (eg on how NFS in unreliable for file-based locking), I've said that this is done for operations that aren't idempotent. This is not really correct. There are very few NFS operations that are truly idempotent if re-issued after a delay; a READDIR might see a new entry, for example, or READ could see updated data in a file. But these differences are not considered dangerous in the way that a MKDIR going from success to failure is, and so they are generally not cached in the reply cache in order to leave room for the operations where it really matters.

(Thus, the list of non-cached NFS v3 operations in the Linux kernel NFS server mostly isn't surprising. I do raise my eyes a little bit at COMMIT, since it may return an error. Hopefully the Linux NFS server insures that a repeated COMMIT gets the same error again.)

Discovering Vim's Visual (selection) mode

Usually, when I want to operate on a block of text (for example to reflow it) I can select it with one of Vim's motion commands, such as '}' for a nominal paragraph. Vim's notion of a paragraph is sufficiently all-encompassing that this works for a lot of things that aren't strictly paragraphs but are, for example, ed style diffs. When I want to select something that doesn't match up with this, such as only part of a quoted email message, my traditional solution has been marks.

Vim's Visual mode is an alternative way of making these selections. To be a bit unkind to vim, it's vim's version of what would be mouse based selection in a GUI editor such as sam. Since it has visual feedback, it has many of the advantages of mouse based selection, but it also lets you take advantage of all of Vim's motion commands for moving around. In effect it's like a version of setting a mark that visually shows me what a "`a,." address would cover, and I can back up or move forward from the results of, say, } to jump the the end of the nominal paragraph.

(More likely I will start visual mode and then move line by line until I'm where I want to be. I often take a brute force approach to editing in Vim.)

As covered in the documentation, visual mode comes in three varieties; v for character by character selection, V for line by line selection, and Ctrl-v for a block selection (which I haven't used). Even with a character by character selection, some Vim commands (including many that I use) will operate on all lines with some selected characters in them. However, I probably want to get into the habit of properly using V instead when I'm planning on doing a line-wise operation, such as using ! to reflow quoted text with par.

I'm not sure how I stumbled into actually understanding visual mode. I'm sure I've accidentally hit v periodically in command mode and been irritated by the resulting mystery highlighting (fortunately hitting ESC a couple of times makes it go away), but recently I learned what it was actually for and motivated myself to start using it in some cases where I expect it to be more convenient than the alternatives.

(This entry is one of the ones I write partly to get a Vim feature to stick in my mind and partly to do some more reading on it so I know more of what I'm talking about.)

Remote X versus 'seamless windows' in remote desktop software

I recently said on Twitter that I expected x2go to have what I considered significant limitations for my ongoing use of remote X. This isn't because of any specific flaw in x2go; instead, it's because of a fundamental difference between remote X and what remote desktop software like x2go does even when it's implementing what the Wikipedia comparison of remote desktop software calls 'seamless windows'.

Normally, remote desktop software like x2go gives you an entire remote desktop in a large window. When the remote desktop is on a Unix machine, this is normally done by starting an X server (or perhaps a Wayland server) that's drawing on a chunk of RAM as a virtual framebuffer, instead of to a GPU and one or more physical displays. The X server then tracks changes to this virtual framebuffer and sends them to your local client, which displays them. Local X input events flow in reverse; your local X server sends them to the local client, which wraps them up in some protocol and sends them to the remote X server, which then passes them on its clients. Because what's running is a remote X server, it can run a full (X) desktop session (assuming that the remote X server supports all of the X extensions that the particular sort of desktop requires).

To implement a 'seamless window' in this model and get something like what you see when you run 'ssh -X xsomething', the straightforward approach is for the remote desktop software to still start a remote X server but only run a single X program that talks to the server, the remote program you want to start. The local RD client and the remote X server work together to only show you the area of the remote X server's virtual framebuffer that the single client's window occupies.

However, just passing the drawn pixels and the X input events back and forth is only part of having a real remote X program. A true remote X program can see and update X properties, it can participate in at least X's cut and paste systems, letting you copy things in and out of it, it will have proper window titles and desktop or taskbar icons (and it will be able to update its window title, as some programs do), it can appear and disappear (being 'mapped' or 'unmapped'), and it can ask to be placed at a specific location (and there's more). Full support for X properties requires not just access to X properties on the root window, but access to X properties on other client windows; this is how Firefox's X remote control works (cf).

If you're running a full remote desktop session, you don't want a lot of this. Perhaps you want some sort of cut and paste back and forth for convenience, but it's generally a feature that the remote desktop's root window X properties do not become your local root window X properties (and vice versa), for example. This means that a basic implementation of seamless windows with a remote X server won't support any of this. Adding some of this stuff is possible with code that's specific to the seamless window feature, but supporting everything becomes extremely difficult. In particular, a remote program seeing and updating X properties on your local windows pretty much requires an entire set of phantom windows in the remote X server.

(I was going to say that you'd also have problems if the remote program (or things started from it) wanted to open more top level windows, for example because you started GNU Emacs inside your remote xterm, but apparently at least some 'seamless window' implementations manage to handle this (or claim to).)

Pretty much all of the X programs that I run remotely require some or all of these advanced remote X features (including multiple independent top level windows). A 'remote X' system that performs better than 'ssh -X' but doesn't support most of these features isn't terribly useful to me.

(I talked about this a bit in my entry about my failure with Xpra, but I didn't get into detail about why this is hard and at odds with the 'remote desktop' approach.)

Remote X has been a life saver over this past year

It's now been a bit over a year since I was last regularly in the office (due to still ongoing events). One of the things that has turned out to be really important to working productively from home has been the X Window System's remote support (and SSH's support for X forwarding too, let's not forget that side). Especially, what has been important for me isn't just being able to run GUI programs in some way from a remote host, but being able to have them as regular windows on my desktop instead of corralled off into their own seperate 'remote desktop' world.

(This is on my mind recently because I've been reading another round of people saying that we all have to move to Wayland real soon now. Although apparently Wayland now has some sort of general forwarding in the form of waypipe, although there doesn't seem to be much written on the Internet about experience with it.)

The largest thing remote X has let me do is keep on using my normal email environment, which relies on exmh, xlbiff, and sam. To be productive, all of these need to run in separate X windows on my desktop that I can shift around in various ways (and so that xlbiff can appear on any virtual screen I'm on and then quietly disappear), not in a single giant remote session window. General remote X also lets me have the remote exmh send URLs in email to my local Firefox through Firefox's X based remote control. As a sysadmin a lot of my work involves interacting with email and while things aren't quite as fluid as they would be at work, they're a lot better than they would be without exmh.

(If I didn't have remote X, I probably would have tried to switch to MH-E in GNU Emacs, since that could be used through a text mode SSH session. This would have at least been a bunch of yak shaving.)

The other big thing has been that my home Internet connection is good enough to run VMWare from my work desktop over remote X. Being able to spin up virtual machines as usual has been essential for testing and developing all sorts of things under conditions where doing this with real hardware is difficult and infrequent. I could not have done this on my home machine, and doing it over some sort of RDP or VNC session would have been at least awkward and possibly too bandwidth intensive for my DSL connection.

(VMWare machines can have their consoles accessed over VNC, but most of what I deal with is the regular VMWare GUI for managing VMs. I have my VMs set up so that most of the time I only need to interact with them over SSH, although developing and testing our Ubuntu 20.04 install image did require console access. Since my remote VMWare doesn't scale up the VM consoles when displaying on my local HiDPI display, the less I have to interact with the consoles the better. The usual environment variables affect the VMWare GUI but not the actual consoles.)

I've periodically used an assortment of other X programs remotely. The biggest one is GNU Emacs, which I use as my superintelligent code editor if I'm seriously digging into some program. Modern LSP-based smart code editing in Emacs really wants to be run on a graphical display; it will try to work in a pure text environment, but it just doesn't go very well. If I'm doing any serious Go editing, I need either a remote X session for GNU Emacs or to drag the Go code over to my local machine.

Of course I was already heavily using remote X at work; my normal email environment has always run on one of our login servers, not my office desktop, for example. But this past year of working from home I've generally tried to do as much as possible through SSH sessions (because they're more tolerant of my slower network), and that's really driven home the places where remote X has really mattered.

(It's interesting to look back at my entry from several years ago about what I really miss when I don't have remote X and tick off the things that I made work versus the things that I miss but not enough to do anything about. Perhaps I should make it a habit to run all of my remote logins as 'ssh -X' sessions so that I could at least spawn a new xterm from them if I wanted to, even if I mostly used the initial text mode SSH logins.)

Different views of what are basic and advanced Vim features

Although vim is now my most regularly used editor (and it really is vim, not vi), I still have lots to learn about it. Thus, every so often I wind up reading guides to Vim, which often label themselves as being for 'beginning users' or 'intermediate users' or 'advanced users'. One of the interesting things in reading these guides is hitting features labeled as basic that I don't know, or features labeled as advanced that I consider perfectly routine.

(My most recent reading is this one, which I haven't even finished yet.)

For some features that are sometimes considered advanced, like address ranges, I think a lot of this has to do with the path I took through various editors on the way to today. For instance, I've seriously used a version of Unix ed, and it's hard to use ed without getting a fairly thorough exposure to address ranges. This was reinforced by both sysadmin use of sed and my long running fondness for the sam editor. Use of ed and then sam left me completely comfortable with the idea of doing editor operations as (Ex) commands.

(I don't know if marks are generally considered an advanced concept, but as a sysadmin who uses less a lot and jumps around in various sorts of files to find things, marks are very familiar to me and I use them all the time. Sam has the idea of a mark but only a single one, which is easier to keep track of, and it's been so long since I seriously used ed that I can't remember if I really used marks in it.)

When I run into basic vim features that I don't know or haven't mastered, I generally remind myself that I basically drifted into vim instead of actively learning it (including vi). I did at one point long ago read through what was then the vi tutorial, but I did it in another editor (which didn't help get it to stick). This path into vim has left me learning things on a scattershot basis as I stumble over them and decide that they're worth it (not every vim feature is for me).

Part of this is that I use vim primarily as a sysadmin, which is to say that I'm either editing text (such as email) or I'm working with generally small amounts of code, or at least of editing code. My editing sessions tend to be short; I'm definitely not spending all day editing code. If I was, I would probably be more driven to learning vim features that improved that. Plus, right now and likely for the indefinite future, my editor for serious coding is still GNU Emacs for various reasons.

(The pattern of vim things that catches my interest is quite driven by sysadmin interests, from windows while modifying related files at once through mass commenting out references to a machine to changing a bunch of files in the same way at once.)

The mhbuild directives I want for sending MIME attachments with MH

I use and like (N)MH for any number of reasons (it's a very Unix and command line focused mail system, although I often use it with exmh). However, for my sins I sometimes have to send people email with (MIME) attachments of things like PDFs and images, especially these days when I can't just give them physical printouts. This is possible in NMH, but the manual page for mhbuild, the core program you need to do this, is both relatively opaque and full of discussions about things that I mostly don't care about. Since I always wind up picking painfully through the mhbuild manpage when I need it, today I'm going to write down the magic directives you (I) need and also the simpler way that I discovered while writing this entry.

Suppose that we start with a FILE.PDF that we want to send. Our end goal is to wind up with an attachment that as at least the following MIME headers:

Content-Type: application/pdf; name="FILE.PDF"
Content-Disposition: attachment; filename="FILE.PDF"
Content-Transfer-Encoding: base64

In theory the 'name="..."' on the Content-Type is unnecessary and only exists for historical reasons. In practice, it's common practice in MIME messages to put the filename in both places. You can also have a Content-Description, and optionally a Content-ID, which I think is generally irrelevant for attachments.

The simple and often sufficient way to add one or more attachments to your message is to list them as Attach headers in your draft (and then use the 'mime' command in whatnow to activate all of mhbuild's processing). The format of these is simply:

Attach: /path/to/some/FILE.PDF

If NMH can get the MIME type right, this will do just what you want (including providing a Content-Description that's the base filename without its directory). Even if it gets the MIME type wrong, you can fix that by editing the Content-Type: by hand afterward, and for that matter you can edit the file name (in all its places) if you would like your recipient to see a different file name than you have locally. In a lot of cases I think this will be good enough for what I want to attach, especially if I rename my PDFs before hand.

(I didn't discover about Attach until now for many reasons, including that the mhbuild manpage doesn't exactly encourage extensively and carefully reading of all of it. It's very MH manpage-y, for better or worse.)

The more complex way is to use a mhbuild directive to specify everything you want. The mhbuild manpage has a lot to say about directives, but much of it is for complicated and odd cases that I don't care about. The format we want for attaching files is:

#application/pdf; name="FILE.PDF" <> {attachment; filename="FILE.PDF"} /path/argle.pdf

(Notice that this example renames the PDF; it is argle.pdf on the filesystem, but the recipient will see it called FILE.PDF.)

For my future reference, this directive breaks down as follows. The first part becomes the Content-Type. The <> tells mhbuild to not generate a Content-ID, which is generally surplus. The {...} becomes the Content-Disposition. We're specifying the filename MIME parameter here, but you can leave it out if you're happy with the recipient seeing your local file name (this is the same as with Attach). If you leave out the '{ ... }' portion entirely, the result will have no Content-Disposition header.

If you want to go the extra distance you can also also provide a Content-Description by using [...], square brackets, as in '[Amazon purchase invoice #10]'. This goes after the <> and before the {...} (or before the filename, if you leave out the {...}). I don't know how many mail clients show or care about the Content-Description; it's not all that common in my saved mail, and most of the time it's just another copy of the file name.

Given all of this, the minimal version is:

#application/pdf <> {attachment} /path/to/FILE.PDF

This will have no 'name=' parameter on the Content-Type, but will have a 'filename=' parameter on the Content-Disposition so the recipient's mail client will probably let them save it under some useful name. If you're not sure what MIME type some file should be, you can use 'file --mime' to see what it thinks or default to application/octet-stream. If you do the latter, you'll be in good company; we see plenty of attachments on incoming legitimate email that have been defaulted that way even when there are more applicable MIME types.

Talkd and 'mesg n': a story from the old Unix days

I tweeted:

An ancient Unix habit: I often still reflexively run 'mesg n' on my (single-user) workstation before starting screen, even though it's been a very long time since I ran talkd and so had any worries about that.

Back in the old days of Unix there was a program called talk, which actually made it into the POSIX standard (which I was surprised to discover just now). Talk enabled live two way communication, instead of the one way communication of write (which is also a POSIX standard command), but the relevant thing about it was that it generally worked through a daemon, talkd. Your Unix server ran talkd, and when you ran talk it communicated with talkd to notify the person you wanted to talk to and then let them connect with you.

Back in the days, this communication was done over IP, not (say) Unix domain sockets. Since it was the old days of a trusting network environment, your talkd would accept requests from everyone, not just the local machine (and talk would chat across the network), letting anyone on your local network or often the entire Internet try to start up a talk session with you. This meant that even on a single user workstation, there was a reason to run 'mesg n' to avoid having random talk notifications overwrite screen's output and interfere with it.

(On a multi-user machine, other people on the same machine might try to write to you and you'd want to keep that from interfering with your screen session. This isn't an issue on a single user workstation.)

It's been a very long time since my workstation ran talkd, even for requests from the local network. But my reflexes still want to run that 'mesg n' before I start screen.

(I didn't put 'mesg n' in my shell .profile for what is ultimately fuzzy reasons.)

The small oddity in the Unix exec*() family

When I recently wrote about find's -exec option, I casually talked about 'the exec() family of system calls'. This is an odd phrasing and sort of an odd thing to say about Unix system calls, because they don't usually come in families. So let's list off the exec() family:

execv() execve() execvp() execvpe()
execl() execlp() execle()

(This is the list on Linux and OpenBSD; the FreeBSD list has execvP() but not execvpe(). The POSIX version leaves out execvpe() and adds fexecve(), which I don't quite put into this exec() family.)

One of these things is not like the others. Out of the entire list of at least six exec() functions, generally only execve() is a system call; the other exec*() functions are library functions layered on top of it. That there are convenient library functions layered on top of a system call (or a bunch of them) isn't odd; that's what all of stdio is, for example. What makes this situation feel odd is that the names are so close to each other. I have a decent memory for Unix libc function names and most of the time I probably couldn't pick the actual exec() system call out of a lineup like this.

(Right now it's very much in my memory that execve() is the underlying system call on most Unixes, of course.)

This multiplicity goes all the way back to V7 Unix, which documents all of execl(), execv(), execle(), and execve() in its exec(2) manpage. In V7, as is the case today, the underlying system call is execve(), although it had a different system call name. Even V6 had execl() and execv() in the V6 exec(2) manpage.

(The V6 system call was just called exec and took only the program to be executed and argv. When V7 added the environment, it kept the V6 exec call but added a new exece system call that took the environment (well, envp) as an additional argument.)

PS: Some Unixes have underlying system calls that are variants of each other, due to the slow growth and improvement in the system call API over time (for example, to add 64-bit variants of what used to be 32-bit calls). However, usually you only use and think about the most recent version of the system call; they aren't a family of variants the way the exec() family is.