Wandering Thoughts

How Go's net.DialContext() stops things when the context is cancelled

These days, a number of core Go standard packages support functions that take a context.Context argument and abort their operation if the context is cancelled. This is an interesting trick in Go, because normally you can't gracefully interrupt a goroutine doing network IO (which leads to problems in practice). When I started looking into the relevant standard library code I expected to find that things like net.Dialer.DialContext() had special hooks into the runtime's network poller (netpoller) to do this. This turns out to not be the case; instead dialing uses an interesting and elegant approach that's open to everyone doing network IO.

In order to abort an outstanding dial operation if the context is cancelled, the net package simply sets an expired (write) deadline. In order to do this asynchronously, it starts a background goroutine to listen for the context being cancelled (and then there's some complexity involved to clean everything up properly and handle potential races; races caused a number of issues, eg issue 16523). Setting read and write deadlines is already explicitly documented as affecting currently pending reads (and writes), not just future ones, so dialing is reusing a general mechanism that already needs to exist.

(This reuse is a little bit tricky for dialing, which is taking advantage of a customary and useful property where the underlying OS only reports a network socket as writeable once it's connected. This means that you generally check for a connection having completed by seeing if it's now writeable, and in turn this means you can sensibly limit or abort this check by setting a write deadline.)

Now that I've discovered this use of deadlines in DialContext, it's clear that I can do the same thing to abort outstanding network reads or writes in my own code. As a bonus, this will probably return a fairly distinctive error, or I can wrap this in something that implements 'read with context' or 'write with context', probably with some of the race precautions seen in the net package's code.

PS: I was going to say that this is also how net.ListenConfig.Listen handles its context being cancelled, but then I went to look at the code and now I have no idea how that actually works.

PPS: If the context you pass to DialContext() already has a deadline, DialContext() immediately sets a write deadline on the underlying network connection, in addition to its handling of cancellation. There's also some complexity in the code to stop as soon as possible if the context is cancelled immediately, before it starts up the whole extra goroutine infrastructure to wait.

Things I've stopped using in GNU Emacs for working on Go

In light of my switch to basing my GNU Emacs Go environment on lsp-mode, I decided to revisit a bunch of .emacs stuff that I was previously using and take out things that seemed outdated or that I wasn't using any more. In general, my current assumption is that Go's big switch to using modules will probably break any tool for dealing with Go code that hasn't been updated, so all of them are suspect until proven otherwise. For my own reasons, I want to record everything I remove.

My list, based on an old copy of my .emacs that I saved, is:

• go-guru, which was provided through go-mode; one of the things that I sort of used from it was a minor mode to highlight identifiers. To the extent that I care about such highlighting, it's now provided by lsp-mode.

• gorename and the go-rename bindings for it in go-mode. In practice I never used it to automatically rename anything in my code, so I don't miss it now. Anyway, lsp-mode and gopls do support renaming things, although I have to remember that this is done through the lsp-rename command and there's no key or menu binding for it currently.

• godoctor, which was another path to renaming and other operations. I tried this out early on but found some issues with it, then mostly never used it (just like gorename).

• go-eldoc, which provided quick documentation summaries that lsp-mode will now also do (provided that you tune lsp-mode to your tastes).

• I previously had M-. bound to godef-jump (which comes from go-mode), but replaced it with an equivalent lsp-mode binding to lsp-ui-peek-find-definitions.

• I stopped using company-go to provide autocompletion data for Go for company-mode in favour of company-lsp, which uses lsp-mode as a general source of completion data.

All of these dropped Emacs things mean that I've implicitly stopped using gocode, which was previously the backend for a number of these things.

In general I've built up quite a bunch of Go programming tools from various sources, such as gotags, many of which I installed to poke at and then never got around to using actively. At some point I should go through everything and weed out the tools that haven't been updated to deal with modules or that I simply don't care about.

(The other option is that I should remove all of the Go programs and tools I've built up in ~/go/bin and start over from scratch, adding only things that I turn out to actively use and want. Probably I'm going to hold off on doing this until Go goes to entirely modular builds and I have to clean out my ~/go/src tree anyway.)

I should probably investigate various gopls settings that I can set either through lsp-go or as experimental settings as covered in the gopls emacs documentation. Since I get the latest Emacs packages from Melpa and compile the bleeding edge gopls myself, this is more or less an ongoing thing (with occasional irritations).

Some reasons for Go to not make system calls through the standard C library

One of the recent pieces of news in the Unix world is that as part of its general security work, OpenBSD is moving towards only allowing system calls to be made from the C library, not from any other code (you can read about this in OpenBSD system call origin verification). Right now OpenBSD has an exemption for the code of programs themselves, primarily because Go generally makes system calls directly instead of by calling the C library, but they would like to get rid of that. Other people are not happy about Go making direct system calls; for example, on Solaris and Illumos, the only officially supported method of making system calls is also through the C library (although Go does it itself on those operating systems).

(Update: On Illumos and Solaris, Go actually uses the platform C library to make system calls; I was wrong here.)

On the surface this makes Go sound unreasonable, and you might ask why it can't just make Unix system calls through the system's C library the way pretty much every other Unix language does. Although I don't know exactly why the Go developers chose to do it this way, there are reasons why you might want to avoid the C library in a language like Go, because the standard C library's Unix system call API is under-specified and awkward.

The obvious way that the C library API is under-specified for things like Go is the question of how much free stack space you need. C code (even threaded C code) traditionally allocates large or very large stacks, but Go wants to use very small stacks when it can, on the order of a few KB, in order to keep goroutines lightweight. The C library API makes no promises here, so if you want to be safe you need to call into it with larger stacks and even then you're guessing. The issue of how much stack space you need to call C library system calls has already been a problem for Go. Go solved this for now by increasing the stack size, but since the required stack size is not a documented part of the C library API, it may break in the future (on any Unix, not just Linux for calls to vDSOs).

(Unixes that strongly insist you go through the C library to make system calls generally reserve the right to have those 'system call' library functions do any amount of work behind your back, because the apparent API to system calls may not be the real kernel API. Indeed one reason for Unixes to force this is exactly so they can make changes in the kernel API without changing the 'system call' API that programs use. Such a change in internal implementation can of course cause unpredictable and undocumented changes in how much stack space the C library will demand and use during such function calls.)

For system calls, the most obvious awkward area of the C library API is how the specifics of errors are returned in errno, which is nominally a global variable. Using a global variable was sort of okay in the days before multi-threaded programs and wanting to make system calls from multiple threads, but it's clearly a problem now. Making errno work in the modern world requires behind the scenes magic in the C library, which generally means that you must use the entire C runtime (and yes, C has a runtime) to do things like set up thread local storage, create OS level threads so that they have this thread local storage, and retrieve your thread's errno from its TLS. In the extreme, this may require you to use the C library pthreads API to create any threads that will make system calls, then carefully schedule goroutines that want to make system calls onto those pthreads (likely with large stacks, because of the C library API issues there). All of this is completely unnecessary in the underlying kernel API, which already directly provides the error code to you.

The C global errno exists for historical compatibility and because C has no easy way to return multiple values; the natural modern API is Go's approach of returning the result and the errno, which is intrinsically thread safe and has no pseudo-global variables. Requiring all languages to go through the C library's normal Unix API for system calls means constraining all languages to live with C's historical baggage and limits.

(You could invent a new C library API for all of the system calls that directly wrote the error number to a spot the caller provided, which would make life much simpler, but no major Unix or C library is so far proposing to do this. Everyone wants (or requires) you to go through the traditional Unix API, errno warts and all.)

The Go runtime scheduler's clever way of dealing with system calls

One of Go's signature features is goroutines, which are lightweight threads that are managed by the Go runtime. The Go runtime implements goroutines using a M:N work stealing scheduler to multiplex goroutines on to operating system threads. The scheduler has special terminology for three important entities; a G is a goroutine, an M is an OS thread (a 'machine'), and a P is a 'processor', which at its core is a limited resource that must be claimed by an M in order to run Go code. Having a limited supply of Ps is how Go limits how many things it will do at once, so as to not overload the overall system; generally there is one P per actual CPU that the OS reports (the number of Ps is GOMAXPROCS).

When a goroutine performs network IO or any system call operation that can definitely be done asynchronously, Go has an entire runtime subsystem, the netpoller, that converts what looks like multiple separate synchronous operations into a single wait (using operating system mechanisms like epoll). Rather than actually making a blocking system call, your goroutine goes to sleep waiting for its network socket, just as if it was waiting for a channel to become ready. This is all conceptually straightforward, if tricky to implement efficiently.

However, network IO and similar things are far from the only system calls that Go programs can make, so Go has to deal with blocking system calls as well. The straightforward way to handle blocking system calls is for your goroutine's M to release its P just before it makes the system call and then try to re-acquire a P after the system call resumes. If there's no free P at that time, your goroutine gets parked in the scheduler along with everything else that's waiting to run.

While all system calls are blocking in theory, not all are expected to be blocking in practice. For example, on modern systems the 'system call' to get the current time may not even enter the kernel (see vdso(7) on Linux). Having goroutines go through the full work of releasing their current P and then re-acquiring one for these system calls has two problems. First, there's a bunch of overhead involved in locking (and releasing) all of the data structures involved. Second, if there's more runnable goroutines than Ps, a goroutine that makes this sort of system call won't be able to re-acquire a P and will have to park itself; the moment it released the P, something else was scheduled onto it. This is extra runtime overhead, is sort of unfair, and kind of defeats the purpose of having fast system calls (especially ones that don't go into the kernel).

So the Go runtime and scheduler actually have two ways of handling blocking system calls, a pessimistic way for system calls that are expected to be slow and an optimistic way for ones that are expected to be fast. The pessimistic system call path implements the straightforward approach where the runtime actively releases the P before the system call, attempts to get it back afterward, and parks itself if it can't. The optimistic system call path doesn't release the P; instead, it sets a special P state flag and just makes the system call. A special internal goroutine, the sysmon goroutine, then comes along periodically and looks for P's that have been sitting in this 'making a system call' state for too long and steals them away from the goroutine making the system call. When the system call returns, the runtime code checks to see if its P has been stolen out from underneath it, and if it hasn't it can just go on (if the P has been stolen, the runtime tries to get another P and then may have to park your goroutine).

If everything works out, the optimistic system call path has very low overhead (mostly it requires a couple of atomic compare and swap operations). If things don't work out and there's more runnable goroutines than there are Ps, then one P will be unnecessarily idle for what is probably generally a few tens of microseconds (the sysmon goroutine runs at most once every 20 microseconds, but can run less frequently if there seems to be no need for it). There are probably worst case scenarios possible, but in general this seems to be a worthwhile tradeoff on the part of the Go runtime.

Capturing command output in a Bourne shell variable as a brute force option

Often, the natural form of generating and then processing something in the Bourne shell is as a pipeline:

smartctl -A /dev/$dsk | tr A-Z- a-z_ |
     fgrep -v ' unknown_' | awk '<process more>'

timeout 30s ssh somehost npppctl session brief |
    awk '<generate metrics>'

(Using awk is not necessarily recommended here, but it's the neutral default.)

However, there can be two problems with this. First, sometimes you want to process the command's output in several different ways, but you only want to run the command once (perhaps it's expensive). Second, sometimes you want to reliably detect that the initial command failed, or even not run any further steps if it failed because you don't trust that the output it generates on failure won't confuse the rest of the pipeline and produce bad results.

The obvious solution to this is to write the output of the first command into a temporary file, which you can then process and re-process as many times as you want. You can also directly check the first command's exit status (and results), and only proceed if things look good. But the problem with temporary files is that they're kind of a pain to deal with. You have to find a place to put them, you have to name them, you have to deal with them securely if you're putting them in /tmp (or $TMPDIR more generally), you have to remove them afterward (including removing them on error), and so on. There is a lot of bureaucracy and overhead in dealing with temporary files and it's easy to either miss some case or be tempted into cutting corners.

Lately I've been leaning on the alternate and somewhat brute force option of just capturing the command's output in the shell script, putting it into a shell variable:

smartout="$(smartctl -A /dev/$dsk)"
if [ $? -ne 0 ] ; then
   ....
fi
echo "$smartout" | tr A-Z- a-z_ | ....
echo "$smartout" | awk '<process again>'

(Checking for empty output is optional but probably recommended.)

In the old Unix days when memory was scarce, this would have been horrifying (and potentially dangerous). Today, that's no longer really the case. Unless your commands generate a very large amount of output or something goes terribly wrong, you won't notice the impact of holding the entire output of your command in the shell's memory. In many cases the command will produce very modest amounts of output, on the order of a few Kb or a few tens of Kb, which is a tiny drop in the bucket of modern Bourne shell memory use.

(And if the command goes berserk and produces a giant amount of output, writing that to a file would probably have been equally much of a problem. If you hold it in the shell's memory, at least it automatically goes away if and when the shell dies.)

Capturing command output in shell variable solves all of my problems here. Shell variables don't have any of the issues of temporary files, they let you directly see the exit status of the first command in what would otherwise be the pipeline, and you can repeatedly re-process them through different additional things. I won't say it's entirely elegant, but it works and sometimes that (and simplicity) is my priority.

I use unit tests partly to verify that something works in the first place

A long while ago I read Mostly avoid unit tests (via), in which the author starts out with:

Most of the value of a unit test comes when you change the original (tested) code, but not the test, and can run the test to make sure that all is still well. [...]

This is, broadly, not my experience with a lot of the unit tests that I write. A lot of the time I write tests in large part to verify that the code is correct in the first place, not that it's still correct after I change it; in fact, I may never change the code I'm testing after I write it and verify that it works (ie, that it passes all of my tests). Unit tests are often the easiest way to verify that my code is really working, especially if I want to make sure that corner cases work too, errors are handled correctly, and so on.

Not all code needs this or can be easily tested this way (for example if it interacts with the outside world in complex ways), and it very much helps if you test from inside your modules, not outside of them (often you can only directly reach the complex and potentially incorrect code from inside the module). How much I want unit tests also depends on the language I'm writing in. Languages that offer me a REPL give me a way of exploring whether or not my code works without needing to formalize it as written down tests, so in practice I write fewer tests in them.

(Also, after a certain point it's easier to test if some piece of code works by just running the program and exercising it than it is by writing more tests.)

Even when I could explore whether or not my code works through a REPL or through some ad-hoc scaffolding to run it directly, there's a certain amount of value in writing down a table or group of tests. Writing things down encourages me to more fully check the code's range of inputs and possible problems. It's certainly easier to scan a table to see that I haven't considered some case than it is to think back over checks I've already done by hand and realize that I left something out or that I'm not really checking what I thought I was.

(Much of this is probably obvious, and perhaps there are other ways to do it or these are not really proper unit tests. The modern testing terminology sometimes confuses me.)

Realizing that Go constants are always materialized into values

I recently read Global Constant Maps and Slices in Go (via), which starts by noting that Go doesn't let you create const maps or slices and then works around that by having an access function that returns a constant slice (or map):

const rateLimit = 10

func getSupportedNetworks() []string {
    return []string{"facebook", "twitter", "instagram"}
}

When I read the article, my instinctive reaction was that that's not actually a constant because the caller can always change it (although if you call getSupportedNetworks() again you get a new clean original slice). Then I thought more about real Go constants like rateLimit and realized that they have this behavior too, because any time you use a Go constant, it's materialized into a mutable value.

Obviously if you assign the rateLimit constant to a variable, you can then change the variable later; the same is true of assigning it to a struct field. If you call a function and pass rateLimit as one argument, the function receives it as an argument value and can change it. If a function returns rateLimit, the caller gets back a value and can again change it. This is no different with the slice that getSupportedNetworks() returns.

The difference between using rateLimit and using the return value from getSupportedNetworks is that the latter can be mutated through a second reference without the explicit use of Go pointers:

func main() {
   a := rateLimit
   b := &a
   *b += 10

   c := getSupportedNetworks()
   d := c
   d[1] = "mastodon"

   fmt.Println(a, c)
}

But this is not a difference between true Go constants and our emulated constant slice, it's a difference in the handling of the types involved. Maps and slices are special this way, but other Go values are not.

(Slices are also mutable at a distance in other ways.)

PS: Go constants can't have their address taken with '&', but they aren't the only sorts of unaddressable values in Go. In theory we could make getSupportedNetworks() return an unaddressable value by making its return value be '[3]string', as we've seen before; in practice you almost certainly don't want to do that for various reasons.

(This seems like an obvious observation now that I've thought about it, but I hadn't really thought about it before reading the article and having my reflexive first reaction.)

My common patterns in shell script verbosity (for sysadmin programs)

As a system administrator, I wind up writing a fair number of scripts that exist to automate or encapsulate some underlying command or set of commands. This is the pattern of shell scripts as wrapper scripts or driver scripts, where you could issue the real commands by hand but it's too annoying (or too open to mistakes). In this sort of script, I've wound up generally wanting one of three different modes for verbosity; let's call them 'quiet', 'dryrun', and 'verbose' In 'verbose' mode the script runs the underlying commands and reports what exactly it's running, in 'quiet' mode the script runs the underlying commands but doesn't report them, and in 'dryrun' mode the script reports the commands but doesn't run them.

Unfortunately this three-way setup is surprisingly hard to implement in a non-annoying way in Bourne shell scripts. Now that I've overcome some superstition I wind up writing something that looks like this:

run() {
  [ "$verb" != 0 ] && echo "+ $*"
  [ "$DOIT" = y ] && "$@"
}

[...]

run prog1 some arguments
run prog2 other arguments
[...]

This works for simple commands, but it doesn't work for pipelines and especially for redirections (except sometimes redirection of standard input). It's also somewhat misleading about the actual arguments if you have arguments with spaces in them; if I think I'm likely to, I need a more complicated thing than just 'echo'.

For those sort of more complicated commands, I usually wind up having to do some variant of this code as an inline snippet of shell code, often writing the verbose report of what will be run a little bit differently than what will actually get run to be clear about what's going on. The problem with this is not just the duplication; it's also the possibility of errors creeping into any particular version of the snippet. But, unfortunately, I have yet to come up with a better solution in general.

One hacky workaround for all of this is to make the shell script generate and print out the commands instead of actually trying to run them. This delegates all of the choices to the person running the script; if they just want to see the commands that would be run, they run the script, while if they actually want to run the commands they feed the script's output into either 'sh -e' or 'sh -ex' depending on whether they want verbosity. However, this only really works well if there's no conditional logic that needs to be checked while the commands are running. The moment the generated commands need to include 'if' checks and so on, things will get complicated and harder to follow.

An incorrect superstition about running commands in the Bourne shell

Every so often, I wind up writing some bit of shell script that wants to execute an entire command line that it has been passed (program and all). For years I have written this as follows:

# re-execute the command line
cmd="$1"; shift
"$cmd" "$@"

Some version of this has crept into innumerable shell scripts, partly for reasons beyond the scope of this entry. I've always considered it just a little Bourne irritation that "$@" didn't let you directly run commands and you had to peel off the command first.

Except that that's wrong, as I found out as I was just about to write an entry complaining about it. Both Bash and several other implementations of the Bourne shell are perfectly happy to let you write this as the obvious:

# re-execute the command line
"$@"

I've apparently spent years programming via superstition. Either I got this wrong many years ago and it stuck, or at some point I had to use some version of the Bourne shell where this didn't work. Well, at least I know better now and I can do it the easy way in future scripts.

(One case for this sort of re-execution is if you've got a generic script for running other commands with locking. The script is given a lock name and the command line to run under the lock, and it will wind up wanting to do exactly this operation.)

PS: Since this seems to be so widely supported across various versions of the Bourne shell, I assume that this is either POSIX compatible or even POSIX mandated behavior. I haven't bothered to carefully read the relevant specification to be sure of that, though.

A small irritation with Go's crypto/tls package

I generally like Go's TLS support in the crypto/tls package, to the small extent that I've used it; things pretty much work in straightforward ways as I'd expect and it's easy to use. But I do have one issue with it, and that is that it doesn't supply a .String() method for any of the TLS related constants and types it provides.

The obvious problem is that this leads to duplicated work for everyone who wants to report or log the ciphers, TLS version information, and so on used in connections (and you should capture this information). Unless you want to log the raw hex bytes and throw everyone to the wolves to decode it later, you're going to be converting things to strings yourself, and everyone does it (well, lots of people at least).

So, lots of people create the obvious thing that looks something like this:

var tlsVersions = map[uint16]string{
   tls.VersionSSL30: "SSLv3",
   tls.VersionTLS10: "TLS1.0",
   tls.VersionTLS11: "TLS1.1",
   tls.VersionTLS12: "TLS1.2",
   tls.VersionTLS13: "TLS1.3",
}

Congratulations, you now have three problems. The first problem is that this code doesn't build on older versions of Go, because their older version of crypto/tls doesn't have the tls.VersionTLS13 constant. This issue has been encountered by real code out there in the world. The second problem is that this is going to generate deprecation warnings about the use of tls.VersionSSL30 on modern versions of Go, and may someday fail to build entirely in a future version.

(It's not clear if the Go authors will consider the removal of the VersionSSL30 constant to be an API breakage that is prevented by the Go 1.0 compatibility rules. Similar questions apply for SSL and TLS ciphers that Go drops support for.)

The third problem is that this code will fail to handle new TLS versions that are added in future versions of Go, and which it may encounter when it's compiled under those Go versions and makes connections using those new TLS versions; similar things can happen if you try to have a list of TLS ciphers and names for them. In entirely realistic code (as in, respected programs have done it), this can lead to the program panicing.

In Go programs today, your best option is to ignore those temptingly convenient crypto/tls constants and define them yourself, either as raw numbers (in a tlsVersions map or your equivalent) or as painfully recreated constants of your own that are under your control. Perhaps you can use 'go generate' to automate this from some sort of list.

All of this could be avoided if crypto/tls would convert these to strings for you. Unfortunately it's probably too late to really make this convenient by providing .String() methods, because the relevant types are fixed in stone by the Go 1.0 compatibility promise. However, the package could still provide functions to provide string names for TLS versions and TLS ciphers that are supported by it.

PS: In case you think I'm picking on one specific program here, my own code has this issue too with TLS versions (I was already using raw hex values for my mapping from TLS ciphers to names for them). I'll probably be converting it to use raw hex values for TLS versions, which of course makes it more opaque.