Wandering Thoughts archives

How to make your GNU Emacs commands 'relevant' for M-X

Today I learned about the M-X command (well, key binding) (via), which "[queries the] user for a command relevant to the current mode, and then execute it". In other words it's like M-x but it restricts what commands it offers to relevant ones. What is 'relevant' here? To quote the docstring:

[...] This includes commands that have been marked as being specially designed for the current major mode (and enabled minor modes), as well as commands bound in the active local key maps.

If you're someone like me who has written some Lisp commands to customize your experience in a major mode like MH-E, you might wonder how you mark your personal Lisp commands as 'specially designed' for the relevant major mode.

In modern Emacs, the answer is that this is an extended part of '(interactive ...)', the normal Lisp form you use to mark your Lisp functions as commands (things which will be offered in M-x and can be run interactively). As mentioned in the Emacs Lisp manual section Using interactive, 'interactive' takes additional arguments to label what modes your command is 'specially designed' for; more discussion is in Specifying Modes For Commands. The basic usage is, say, '(interactive "P" mh-folder-mode)'

If your commands already take arguments, life is simple and you can just put the modes on the end. But not all commands do (especially for quick little things you do for yourself). If you have just '(interactive)', the correct change is to make it '(interactive nil mh-folder-mode)'; a nil first argument is how you tell interactive that there is no argument.

(Don't make my initial mistake and assume that '(interactive "" mh-folder-mode)' will work. That produced a variety of undesirable results.)

Is it useful to do this, assuming you have personal commands that are truly specific to a given mode (as I do for commands that operate on MH messages and the MH folder display)? My views so far are a decided maybe in my environment.

First, you don't need to do this if your commands have keybindings in your major mode, because M-X (execute-extended-command-for-buffer) will already offer any commands that have keybindings. Second, my assortment of packages already gives me quite a lot of selection power to narrow in on likely commands in plain M-x, provided that I've named them sensibly. The combination of vertico, marginalia, and orderless let me search for commands by substrings, easily see a number of my options, and also see part of their descriptions. So if I know I want something to do with MH forwarding I can type 'M-x mh forw' and get, among other things, my function for forwarding in 'literal plaintext' format.

With that said, adding the mode to '(interactive)' isn't much work and it does sort of add some documentation about your intentions that your future self may find useful. And if you want a more minimal minibuffer completion experience, it may be more useful to have a good way to winnow down the selection. If you use M-X frequently and you have commands you want to be able to select in it in applicable modes without having them bound to keys, you really have no choice.

The Go 'range over functions' proposal and user-written container types

In Go 1.22, the Go developers have made available a "range over function" experiment, as described in the Go Wiki's "Rangefunc Experiment". Recently I read a criticism of this, Richard Ulmer's Questioning Go's range-over-func Proposal (via). As I read Ulmer's article, it questions the utility of the range over func (proposed) feature based on the grounds that this isn't a significant enough improvement in standard library functions like strings.Split (which is given as an example in the "more motivation" section of the wiki article).

I'm not unsympathetic to this criticism, especially when it concerns standard library functionality. If the Go developers want to extend various parts of the standard library to support streaming their results instead of providing the results all at once, then there may well be better, lower-impact ways of doing so, such as developing a standard API approach or set of approaches for this and then using this to add new APIs. However, I think that extending the standard library into streaming APIs is by far the less important side of the "range over func" proposal (although this is what the "more motivation" section of the wiki article devotes the most space to).

Right from the beginning, one of the criticisms of Go was that it had some privileged, complex builtin types that couldn't be built using normal Go facilities, such as maps. Generics have made it mostly possible to do equivalents of these (generic) types yourself at the language level (although the Go compiler still uniquely privileges maps and other builtin types at the implementation level). However, these complex builtin types still retain some important special privileges in the language, and one of them is that they were the only types that you could write convenient 'range' based for loops.

In Go today you can write, for example, a set type or a key/value type with some complex internal storage implementation and make it work even for user-provided element types (through generics). But people using your new container types cannot write 'for elem := range set' or 'for k, v := range kvstore'. The best you can give them is an explicit push or pull based iterator based on your type (in a push iterator, you provide a callback function that is given each value; in a pull iterator, you repeatedly call some function to obtain the next value). The "range over func" proposal bridges this divide, allowing non-builtin types to be ranged over almost as easily as builtin types. You would be able to write types that let people write 'for elem := range set.Forward()' or 'for k, v := kvstore.Walk()'.

This is an issue that can't really be solved without language support. You could define a standard API for iterators and iteration (and the 'iter' package covered in the wiki article sort of is that), but it would still be more code and somewhat awkward code for people using your types to write. People are significantly attracted to what is easy to program; the more difficult it is to iterate user types compared to builtin types, the less people will do it (and the more they will use builtin types even when they aren't a good fit). If Go wants to put user (generic) types on almost the same level (in the language) as builtin types, then I feel it needs some version of a "range over func" approach.

(Of course, you may feel that Go should not prioritize putting user types on almost the same level as builtin types.)

Understanding a recent optimization to Go's reflect.TypeFor

Go's reflect.TypeFor() is a generic function that returns the reflect.Type for its type argument. It was added in Go 1.22, and its initial implementation was quite simple but still valuable, because it encapsulated a complicated bit of reflect usage. Here is that implementation:

func TypeFor[T any]() Type {
  return TypeOf((*T)(nil)).Elem()
}

How this works is that it constructs a nil pointer value of the type 'pointer to T', gets the reflect.Type of that pointer, and then uses Type.Elem() to go from the pointer's Type to the Type for T itself. This requires constructing and using this 'pointer to T' type (and its reflect.Type) even though we only what the reflect.Type of T itself. All of this is necessary for reasons to do with interface types.

Recently, reflect.TypeFor() was optimized a bit, in CL 555597, "optimize TypeFor for non-interface types". The code for this optimization is a bit tricky and I had to stare at it for a while to understand what it was doing and how it worked. Here is the new version, which starts with the new optimization and ends with the old code:

func TypeFor[T any]() Type {
  var v T
  if t := TypeOf(v); t != nil {
     return t
  }
  return TypeOf((*T)(nil)).Elem()
}

What this does is optimize for the case where you're using TypeFor() on a non-interface type, for example 'reflect.TypeFor[int64]()' (although you're more likely to use this with more complex things like struct types). When T is a non-interface type, we don't need to construct a pointer to a value of the type; we can directly obtain the Type from reflect.TypeOf. But how do we tell whether or not T is an interface type? The answer turns out to be right there in the documentation for reflect.TypeOf:

[...] If [TypeOf's argument] is a nil interface value, TypeOf returns nil.

So what the new code does is construct a zero value of type T, pass it to TypeOf(), and check what it gets back. If type T is an interface type, its zero value is a nil interface and TypeOf() will return nil; otherwise, the return value is the reflect.Type of the non-interface type T.

The reason that reflect.TypeOf returns nil for a nil interface value is because it has to. In Go, nil is only sort of typed, so if a nil interface value is passed to TypeOf(), there is effectively no type information available for it; its old interface type is lost when it was converted to 'any', also known as the empty interface. So all TypeOf() can return for such a value is the nil result of 'this effectively has no useful type information'.

Incidentally, the TypeFor() code is also another illustration of how in Go, interfaces create a difference between two sorts of nils. Consider calling 'reflect.TypeFor[*os.File]()'. Since this is a pointer type, the zero value 'v' in TypeFor() is a nil pointer. But os.File isn't an interface type, so TypeOf() won't be passed a nil interface and can return a Type, even though the underlying value in the interface that TypeOf() receives is a nil pointer.

Go 1.22's go/types Alias type shows the challenge of API compatibility

Go famously promises backward compatibility to the first release of Go and pretty much delivers on that (although the tools used to build Go programs have changed). Thus, one may be a bit surprised to read the following about go/types in the Go 1.22 Release Notes:

The new Alias type represents type aliases. Previously, type aliases were not represented explicitly, so a reference to a type alias was equivalent to spelling out the aliased type, and the name of the alias was lost. [...]

Because Alias types may break existing type switches that do not know to check for them, this functionality is controlled by a GODEBUG field named gotypesalias. [...] Clients of go/types are urged to adjust their code as soon as possible to work with gotypesalias=1 to eliminate problems early.

(The bold emphasis is mine, while the italics are from the release notes. The current default is gotypesalias=0.)

A variety of things in go/types return a Type, which is an interface type that 'represents a type of Go'. Well, more specifically these things return values of type Type, and these values have various underlying concrete types. Some code using go/types and dealing with Type values can handle them purely as interfaces, but other code needs to specifically handle all of the particular types (such as Array and so on). Since Type is an interface, such code will use a type switch that is supposed to be exhaustive over all of the concrete types of Type interface values.

Now we can see the problem. When Go introduces a new concrete type that can be returned as a Type value, those previously exhaustive type switches stop being exhaustive; there's a new concrete type that they're not prepared to handle. This could cause various problems in actual code. And Go has no way of requiring type switches to be exhaustive, so such code would still build fine but malfunction at runtime.

Much like the last time we saw something like this, this change is arguably not an API break, at least in theory; Go never explicitly promised that there was a specific and limited list of go/types types that implemented Type, and so in theory Go is free to expand the list. However, as we can see from the release notes (and the current behavior of not generating these new Alias types by default), the Go authors recognize that this is in practice a compatibility break, one that they're explicitly urging people to be prepared for.

What this shows is that true long term backward compatibility is very hard, and it's especially hard in an area that is inherently evolving, like exposing information about an evolving language. Getting complete backward compatibility requires more or less everything about an exposed API to be frozen, and that generally requires the area to be extremely well understood (and often pushes towards exposing very minimal APIs, which has its own problems).

As a side note, I think that Go is handling this change quite well. They've added the type to go/types so that people can add it to their own code (which will make it require Go 1.22 or later), and also provided a way that people can test the code (by building with gotypesalias=1). At the same time no actual 'Alias' types will appear (by default) until some time in the future; I'd guess no earlier than Go 1.24, a year from now.

One of my MH-E customizations: 'narrow-to-pending' (refiles and deletes)

I recently switched from reading my email with exmh, a graphical X frontend to (N)MH, to reading it with MH-E in GNU Emacs, which is also a frontend to (N)MH. I had a certain amount of customizations to exmh, and for reasons beyond the scope of this entry, I wound up with more for MH-E. One of those customizations is a new MH-E command (and keybinding for it), mh-narrow-to-pending.

Both exmh and MH-E process deleting messages and refiling them to folders in two phases. In the first phase your read your email and otherwise go over the current folder, marking messages to be deleted and refiled; once you're satisfied, you tell them to actually execute these pending actions. MH-E also a general feature to limit what messages are listed in the current folder. In Emacs jargon this general idea is known as narrowing, and there's various tools to 'narrow' the display of buffers to something of current interest. My customization narrows to show only the messages in the current folder that have pending actions on them; these are the messages that will be affected if you execute your pending actions.

So here's the code:

(defun mh-narrow-to-pending ()
  "Narrow to any message with a pending refile or delete."
  (interactive)
  (if (not (or mh-delete-list mh-refile-list))
      (message "There are no pending deletes or refiles.")
    (when (assoc 'mh-e-pending mh-seq-list) (mh-delete-seq 'mh-e-pending))
    (when mh-delete-list (mh-add-msgs-to-seq mh-delete-list 'mh-e-pending t t))
    (when mh-refile-list
      (mh-add-msgs-to-seq
       (cl-loop for folder-msg-list in mh-refile-list
                append (cdr folder-msg-list))
       'mh-e-pending t t))
    (mh-narrow-to-seq 'mh-e-pending)))

(This code could probably be improved, and reading it I've discovered that I've already forgotten what parts of it do and the details of how it works, although the broad strokes are obvious.)

Writing this code required reading the existing MH-E code to find out how it did narrowing and how it marked messages that were going to be refiled or deleted. In the usual GNU Emacs way, this is not a documented extension API for MH-E, although in practice it's unlikely to change and break my code. To the best of my limited understanding of making your own tweaks for GNU Emacs modes like MH-E, this is basically standard practice; generally you grub around in the mode's ELisp source, figure things out, and then do things on top of it.

There are two reasons that I never tried to write something like this for exmh. The first is that exmh doesn't do anywhere near as much with the idea of 'narrowing' the current folder display. The other is that I wound up using the two differently. In MH-E, it's become quite common for me to pick through my inbox (or sometimes other folders) for messages that I'm now done with, going far enough back in one pass that I wind up with a sufficient patchwork that I want to double check what exactly I'm going to be doing before I commit my changes. Since I can easily narrow to messages in general, narrowing to see these pending changes was a natural idea.

(Picking through the past week or more of email threads in my inbox has become a regular Friday activity for me, especially given that MH-E has a nice threaded view.)

It's easy to fall into the idea that any readily extendable program is kind of the same, because with some work you can write plugins, extensions, or other hacks that make it dance to whatever tune you want. What my experience with extending MH-E has rubbed my nose into is that the surrounding context matters in practice, both in how the system already works and in what features it offers that are readily extended. 'Narrow to pending' is very much an MH-E hack.