Wandering Thoughts

C was not created as an abstract machine (of course)

Today on the Fediverse I saw a post by @nytpu:

Reminder that the C spec specifies an abstract virtual machine; it's just that it's not an interpreted VM *in typical implementations* (i.e. not all, I know there was a JIT-ing C compiler at some point), and C was lucky enough to have contemporary CPUs and executable/library formats and operating systems(…) designed with its VM in mind

(There have also been actual C interpreters, some of which had strict adherence to the abstract semantics, cf (available online in the Usenix summer 1988 proceedings).)

This is simultaneously true and false. It's absolutely true that the semantics of formal standard C are defined in terms of an abstract (virtual) machine, instead of any physical machine. The determined refusal of the specification to tie this abstract machine in concrete CPUs is the source of a significant amount of frustration in people who would like, for example, for there to be some semantics attached to what happens when you dereference an invalid pointer. They note that actual CPUs running C code all have defined semantics, so why can't C? But, well, as is frequently said, C Is Not a Low-level Language (via) and the semantics of C don't correspond exactly to CPU semantics. So I agree with nytpu's overall sentiments, as I understand them.

However, it's absolutely false that C was merely 'lucky' that contemporary CPUs, OSes, and so on were designed with its abstract model in mind. Because the truth is the concrete C implementations came first and the standard came afterward (and I expect nytpu knows this and was making a point in their post). Although the ANSI C standardization effort did invent some things, for the most part C was what I've called a documentation standard, where people wrote down what was already happening. C was shaped by the CPUs it started on (and then somewhat shaped again by the ones it was eagerly ported to), Unix was shaped by C, and by the time that the C standard was producing drafts in the mid to late 1980s, C was shaping CPUs through the movement for performance-focused RISC CPUs (which wanted to optimize performance in significant part for Unix programs written in C, although they also cared about Fortran and so on).

(It's also not the case that C only succeeded in environments that were designed for it. In fact C succeeded in at least one OS environment that was relatively hostile to it and that wanted to be used with an entirely different language.)

Although I'm not absolutely sure, I suspect that the C standard defining it in abstract terms was in part either enabled or forced by the wide variety of environments that C already ran in by the late 1980s. Defining abstract semantics avoided the awkward issue of blessing any particular set of concrete ones, which at the time would have advantaged some people while disadvantaging others. This need for compromise between highly disparate (C) environments is what brought us charming things like trigraphs and a decision not to require two's-complement integer semantics (it's been proposed to change this, and trigraphs are gone in C23, also).

Dating from when ANSI C was defined and C compilers became increasingly aggressive about optimizing around 'undefined behavior' (even if this created security holes), you could say that modern software and probably CPUs has been shaped by the abstract C machine. Obviously, software increasingly has to avoid doing things that will blow your foot off in the model of the C abstract machine, because your C compiler will probably arrange to blow your foot off in practice on your concrete CPU. Meanwhile, things that aren't allowed by the abstract machine are probably not generated very much by actual C compilers, and things that aren't generated by C compilers don't get as much love from CPU architects as things that do.

(This neat picture is complicated by the awkward fact that many CPUs probably runs significantly more C++ code than true C code, since so many significant programs are written in the former instead of the latter.)

It's my view that recognizing that C comes from running on concrete CPUs and was strongly shaped by concrete environments (OS, executable and library formats, etc) matters for understanding the group of C users who are unhappy with aggressively optimizing C compilers that follow the letter of the C standard and its abstract machine. Those origins of C were there first, and it's not irrational for people used to them to feel upset when the C abstract machine creates a security vulnerability in their previously working software because the compiler is very clever. The C abstract machine is not a carefully invented thing that people then built implementations of, an end in and of itself; it started out as a neutral explanation and justification of how actual existing C things behaved, a means to an end.

I should assume contexts aren't retained in Go APIs

Over on the Fediverse, I said something about some Go APIs I'd run into:

Dear everyone making Go APIs that take a context argument and return something you use to make further API calls (as methods): please, I beg you, document the scope of the context. Does it apply to only the initial setup API call? Does it apply to all further API calls through the object your setup function returned? I don't want to have to read your code to find this out. (Or to have it change.)

I was kind of wrong here. While I still feel that your documentation might want to say something about this, I've come around to realizing that I should assume the default is that contexts are not retained. In fact this is what the context package and the Go blog's article on contexts and structs say you should do.

I'll start with the potential confusion as a consumer of an API. Suppose that the API looks like this:

handle := pkg.ConnectWithContext(ctx, ...)
r, e := handle.Operation(...)
[...]
r, e := handle.AQuery(...)

Your (my) question is how much does the context passed to ConnectWithContext() cover. It could cover only the work to set up the initial connection, or it could cover everything done with the handle in the future. The first allows fine-grained control, while the second allows people to easily configure a large scale timeout or cancellation. What the Go documentation and blog post tell you to do is the first option. If people using the API want a global timeout on all of their API operations, they should set up a context for this and pass it to each operation done through the handle, and thus every handle method should take a context argument.

Because you can build the 'one context to cover everything' usage out of the 'operations don't retain contexts' API but not the other way around, the latter is more flexible (as well as being what Go recommends). So this should be my default assumption when I run into an API that uses contexts, especially if every operation on the handle also takes a context.

As far as documentation goes, maybe it would be nice if the documentation mentioned this in passing (even with 'contexts are used in the standard Go way so they only cover each operation' as part of the general package documentation), or maybe this is now just what people working regularly in Go (and reading Go APIs) just assume. For what it's worth, the net's package DialContext() does mention that the context isn't retained, but then it was added very early in the life of contexts, before they were as well established and as well known as they are now.

I feel the ice is thinner for documentation if the methods on the handle don't take a context (and aren't guaranteed to be instant and always succeed). Then people might at least wonder if the handle retains the original context used to establish it, because otherwise you have no way to cancel or time out those operations. But I suspect such APIs are uncommon unless the API predates contexts and is strongly limited by backward compatibility.

(These days Go modules allow you to readily escape that backward compatibility if you want to; you can just bump your major version to v2 and add context arguments to all the handle methods.)

(Now that I've written this down for myself, hopefully I'll remember it in the future when I'm reading Go APIs.)

Backporting changes is clearly hard, which is a good reason to avoid it

Recently, the Linux 6.0 kernel series introduced a significant bug in 6.0.16. The bug was introduced when a later kernel change was backported to 6.0.16 with an accidental omission (cf). There are a number of things you can draw from this issue, but the big thing I take away from it is that backporting changes is hard. The corollary of this is that the more changes you ask people to backport (and to more targets), the more likely you are to wind up with bugs, simply through the law of large numbers. The corollary to the corollary is that if you want to keep bugs down, you want to limit the amount of backporting you do or ask for.

(The further corollary is that the more old versions you ask people to support (and the more support you want for those old versions), the more backports you're asking them to do and the more bugs you're going to get.)

I can come up with various theories why backporting changes is harder than making changes in the first place. For example, when you backport a change you generally need to understand the context of more code; in addition to understanding the current code before the change and the change itself, now you need to understand the old code that you're backporting to. Current tools may not make it particularly easy to verify that you've gotten all of the parts of a change and have not, as seems to have happened here, dropped a line. And if there's been any significant code reorganization, you may be rewriting the change from scratch instead of porting it, working from the intention of the change (if you fully understand it).

(Here, there is still an net_csk_get_port() function in 6.0.16 but it doesn't quite look like the version the change was made to so the textual patch doesn't apply. See the code in 6.0.19 and compare it to the diff in the 6.1 patch or the original mainstream commit.)

Some people will say that backports should be done with more care, or that there should be more tests, or some other magical fix. But the practical reality is that they won't be. What we see today is what we're going to continue getting in the future, and that's some amount of errors in backported changes, with the absolute number of errors rising as the number of changes rises. We can't wish this away with theoretical process improvements or by telling people to try harder.

(I don't know if there are more errors in backported changes than there are in changes in general. But generally speaking the changes that are being backported are supposed to be the ones that don't introduce errors, so we're theoretically starting from a baseline of 'no errors before we mangle something in the backport'.)

PS: While I don't particularly like its practical effects, this may make me a bit more sympathetic toward OpenBSD's support policy. OpenBSD has certainly set things up so they make minimal changes to old versions and thus have minimal need to backport changes.

My Git settings for carrying local changes on top of upstream development

For years now I've been handling my local changes to upstream projects by committing them and rebasing on (Git) pulls, and it's been a positive experience. However, over the years the exact Git configuration settings I wanted to make this work smoothly have changed (due to things such as Git 2.34's change in fast-forward pull settings), and I've never written down all of the settings in one place. Since I recently switched to Git for a big repository where I carry local changes, this is a good time to write them down for my future reference.

My basic settings as of Fedora's Git 2.39.0 are:

git config pull.rebase true
git config pull.ff true

The second is there to deal with the Git 2.34 changes, since I have a global pull.ff setting of 'only'.

If you're working with a big repository where you don't really care about which files changed when you pull an upstream update and that output is way too verbose, the normal rebase behavior gives you this. You may also want the following:

git config merge.stat off

This is mostly a safety measure, because normally you aren't merging, you're rebasing. If you want to still see 'git pull' like output, you want to set 'git config rebase.stat true' (as I started years ago).

If you have a repository with a lot of active branches, you may also want:

git config fetch.output compact

This mostly helps if branch names are very long, long enough to overflow a terminal line; otherwise you get the same number of lines printed, they're just shorter. Unfortunately you can't easily get tracking upstream repositories to be significantly more quiet.

If your upstream goes through a churn of branches, you (I) will want to prune now-deleted remote branches when you fetch updates. The condensed version is:

git config remote.origin.prune true

With more Git work, it's possible to pull only the main branch. If I'm carrying local changes on top of the main branch and other branches are actively doing things, this may be what I want. It's relatively unlikely that I'll switch to another branch, since it won't have my changes.

(This may turn out to be what I want with the Mozilla gecko-dev repository, but it does have Firefox Beta and Release as well, and someday I may want convenient Git access to them. I'm not sure how to dig out specific Firefox releases, though.)

Some notes to myself on 'git log -G' (and sort of on -S)

Today I found myself nerd-sniped by a bit in Golang is evil on shitty networks (via), and wanted to know where a particular behavior was added in Go's network code. The article conveniently identified the code involved, so once I found the source file all I theoretically needed to do was trace it back in history. Until recently, my normal tool for this is Git's 'blame' view and mode, often on Github because Github has a convenient 'view git-blame just before this commit', which makes it easy to step back in history. Unfortunately in this case, the source code had been reorganized and moved around repeatedly, so this wasn't easy.

Instead, I turned to ''git log -G', which I'd recently used to answer a similar question in the ZFS code base. 'git log -G <thing>' and the somewhat similar 'git log -S <thing>' search for '<thing>' in commits (in different ways). This time around, I used plain 'git log -G <thing>' and then got the full details of likely commits with 'git show <id>'. A generally better option is 'git log -G <thing> -p', which includes the diff for a commit but by default shows you only the file (or files) where the thing shows up in the changes (per gitdiffcore's pickaxe section).

(In this case I had to iterate git-log a few times, because the implementation changed. The answer turns out to be it's been there since the beginning, which I could have found out by reading the other comments.)

'Git log -G' is not exactly the fastest thing in the world, which isn't surprising since it has to generate diffs for all changesets in order to look at things. It's single-threaded, unsurprisingly, and generally CPU bound in my testing. This implies that if I'm probing for multiple things at once on a SMP machine (which is the usual case), it's to my benefit to run multiple git-logs at once in different windows. On sufficiently large repositories it's probably also disk IO bound, although that will depend a lot on the storage involved. Because of this, it seems that it be useful to trim down what file paths Git considers, if you good confidence of where relevant files both are now and were in the past.

'Git log -S' is subtly different from 'git log -G' in a way that may make it less useful than you expect, depending on the repository. As covered in the git-log manual page, the -S option specifically includes binary files as well, while -G implicitly excludes them because they don't normally create patch text (binary files can be included with --text if you really want). In this case the identifier name I was looking for also appeared in some binary files of test data, so some of the commits reported by 'git log -S' puzzled me until I realized that they were being included because they added new versions of the test data, which meant that the number of instances of the identifier name had gone up.

(There may be some way to make -S not search binary files, but if so I couldn't find it when looking through the git-log manual page.)

If I'm hunting for when something was introduced or removed and I'm sure that the repository has no binary files to confuse me, using 'git log -p -S' is probably safe. If there are binary files around to annoy me, I'm probably pragmatically better off using 'git log -p -G'. Using -G also means that I'll spot changes in how something is used, for example making a function call conditional or not conditional (which I believe won't normally show up in -S). Probably my life is better if I standardize on first using 'git log -G' and then switching to -S if I'm getting too many code motion commits.

More use of Rust is inevitable in open source software

Recently, I saw a poll on the Fediverse about making Rust a hard dependency for fwupd. This got me to post a lukewarm take of my own:

Lukewarm take: the spread of Rust in open source software is inevitable, because nothing else fills the niche for 'C/C++ but strongly safe'. We need a replacement C because in general we can't write safe C/C++ at scale.

Rust isn't really my thing, but this shift implies that sooner or later I'm going to have to learn enough to read it and modify it.

I was then asked about my views on Zig in this context:

@williewillus I haven't used Zig, but it (and other C alternatives) doesn't bring the kind of memory, concurrency, and other safety that Rust does. My view is that changes for only small improvements are not going to motivate many OSS developers (although they could get used in some greenfield projects driven by enthused developers).

This issue is similar to how Rust is in our future, but from a different angle. My earlier entry was coming from how quite a few open source programmers like Rust and so naturally were writing things in it. Now I feel that Rust is also inevitable because people trying to add more safety to important software (such as Linux's fwupd) are going to turn to Rust as basically their best option. Zig could become inevitable for the first reason (programmer enthusiasm), but seems unlikely to do so for the second reason, where Rust stands more or less alone.

(In a similar line is Google's Memory Safe Languages in Android 13, although I consider Android only nominally open source software in this sense.)

This shift will inevitably make life harder for smaller and more niche (Unix) operating systems and architectures, since you'll increasingly need a Rust toolchain as well as a C and C++ one in order to bring up various important software. In that way it's just as harmful and also just as inevitable as the migration from HTTP to HTTPS for websites. The security landscape isn't getting much better for C and C++, and at the same time we have a steady increase in the amount of code out there. There are plenty of developers who really want to bend this curve, and asking them to refrain from their best and easiest option in order to help a small fraction of people is not likely to work.

(Telling developers to do better at writing safe C and C++ doesn't work, especially at scale. Doing so is also generally more work than simply writing in Rust, and open source developers have finite amounts of time.)

Detecting missing or bad Go modules and module versions

I recently wrote about the case of a half-missing Go import, where a particular version of a module had been removed upstream but was still available through the (default) Go module proxy, so things didn't really notice. This raises the interesting question of how you find this sort of thing, in all of at least three variations. Unfortunately, right now there are no great answers, but here's what I can see.

The most straightforward case is if a module has been marked as entirely deprecated or a module version has been marked as retracted in the module's go.mod. Marking the latest version of your module as retracted is actually somewhat complex; see the example in retract directive. In both cases, you can find out about this with 'go list -m -u all', although it's mixed in with all of the other modules. I think that today, you can narrow the output down to only retracted and deprecated modules by looking for '(' in it:

go list -m -u all | fgrep '('

However, this will not help you if the upstream has removed the module entirely, or removed a version without doing the right retract directive magic in go.mod. And using fgrep here counts on the Go authors not changing the format of the output for retracted or deprecated modules; as we've all found out in the module era, the Go tools are not a stable interface.

(People who are very clever with Go templates may be able to craft a 'go list -m -f' format that will only list deprecated modules and retracted module versions.)

To detect a missing version or module, you need to defeat two opponents at once; the Go module proxy and your local module cache. Defeating the Go module proxy simple and is done by setting the GOPROXY environment variable to 'direct'. There is no direct way to disable the local module cache; instead, you have to set the GOMODCACHE environment variable to a new, empty scratch directory before you run 'go mod list -m -u all'. This gives you:

export GOPROXY=direct
export GOMODCACHE=/tmp/t-$$
go list -m -u all

This will be slow. If a module version has been removed, you'll get a relatively normal error message of 'invalid version: unknown revision v<...>', which is printed to standard error. If a module has been removed entirely, you will get a much more cryptic error that will probably complain something like 'invalid version: git ls-remote -q origin in <GOMODCACHE area>: <odd stuff>'. I'm not going to put the literal message I get here because I'm not sure anyone else would get the same one.

Afterward, you'll want to remove your temporary GOMODCACHE (which has fortunately not been created with any non-writeable directories). If you intend to check a bunch of modules or programs at once, you don't have to clean and remake your scratch GOMODCACHE between each one; it's good enough that it has relatively recent contents.

The other way to detect missing modules and module versions is to try to build all of your programs with these GOPROXY and GOMODCACHE environment variable settings (and afterward you'll need to do 'go clean -modcache' to clean out this scratch module cache, which also removes your GOMODCACHE top level directory). If all programs build, all of their dependencies are still there (although some may be retracted or deprecated; 'go build' or 'go install' won't warn you about that). In some CI environments you'll start with a completely clean environment every time, with no local module cache, so all you need to do is force 'GOPROXY=direct' (of course, this will slow down builds; the Go module proxy is faster than direct fetching).

It's possible that people have already written some third party programs to do all of this for you, in a more convenient form and with nicer output than you get from the normal Go tools (and to be fair, this isn't really the job of those tools).

PS: The reason 'go build' and 'go install' generally won't warn you that you're using retracted versions or entirely deprecated modules is partly that by and large the retraction or deprecation won't be in the go.mod of the version of the module you're currently using. Instead these will usually be in a later version, which Go will only look at if you ask it to look for module updates.

Go and the case of the half-missing import

One of the Go programs I keep around is gops, which I find handy for seeing if I'm using outdated binaries of Go programs that I should rebuild (for example, locally built Prometheus exporters). I have a script that updates and rebuilds all of my collection of Go programs, gops included, with the latest Go tip build (and the latest version of their code). Recently, rebuilding gops has been failing with an error:

go/pkg/mod/github.com/google/gops@[...]: reading github.com/shirou/gopsutil/go.mod at revision v3.22.11: unknown revision v3.22.11

This only happened on some systems (using one version of my rebuild script), and also didn't happen if I did a 'go install' by hand in my local cloned repository. In fact, after that manual 'go install', my scripted rebuilds started working. It took quite a while before I finally worked out what was going on.

The root cause of this error is that github.com/shirou/gopsutil retracted v3.22.11 after it accidentally slipped out. As I write this, there's no such tag in the repo and there is a commit that specifically marks v3.22.11 as retracted in go.mod (see also this gopsutil pull request about the retraction). However, before v3.22.11 was retracted Github's 'dependabot' noticed the new version and submitted a gops pull request to update its dependency, which was accepted a week later (after the retraction) in this commit.

But if the gopsutil version was retracted, how does gops build at all for anyone, and why did it start building for me? The first answer is the (default) Go module proxy, and the second answer is the module cache. The Go module proxy caches modules it's seen, so as long as one person fetched gopsutil v3.22.11 through the proxy cache before it was retracted, that version will be there for more or less all time and anyone can get a copy of it from the proxy cache. Although I had forgotten it, one set of my Go rebuild scripts deliberately turn off the module proxy so that I can spot exactly this sort of issue. When I ran 'go install' by hand, it was now talking to the proxy cache and so it got a copy of gopsutil v3.22.11. Once I had a copy, that copy was saved away in my local module cache, where it was found by the 'go' command run by my rebuild script.

(Setting GOPROXY=direct bypasses the Go module proxy and its cache, but doesn't bypass your local module cache.)

All of the individual pieces here are more or less working as designed, but the end result appears undesirable. Some of this seems to have happened because of automatic release tagging (cf), which makes that a foot-gun too, especially if things like Dependabot and Continuous Integration checks cause the default Go module proxy to fetch and cache newly tagged versions very rapidly after they become visible.

(This elaborates on a Fediverse post, and I filed a gops issue, although who knows what other Go programs and projects have this particular gopsutil issue too.)

Floating point NaNs as map keys in Go give you weird results

The last time around I learned that Go 1.21 may have a clear builtin partly because you can't delete map entries that have a floating point NaN as their key. Of course this isn't the only weird thing about NaNs as keys in maps, although you can pretty much predict all of them from the fact that NaNs never compare equal to each other.

First, just like you can't delete a map entry that has a key of NaN, you can't retrieve it either, including using the very same NaN that was used to add the entry:

k := math.NaN()
m[k] = "Help"
v, ok := m[k]

At this point v is empty and ok is false; the NaN key wasn't found in the map although there is an entry with a key of NaN.

Second, you can add more than one entry with a NaN key, in fact you can add the same NaN key to the map repeatedly:

m[k] = "Help"
m[k] = "Me"
m[k] = "Out"
// Now m has at least three entries with NaN
// as their key.

You can retrieve the values of these entries only by iterating both the keys and values of the map with a ranged for loop:

for k, v := range m {
  if math.IsNaN(k) {
    fmt.Println(k, v)
  }
}

If you iterate only the keys, you run into the first issue; you can't use the keys to retrieve the values from the map. You have to extract the values directly somehow.

I don't think this behavior is strictly required by the Go specification, because the specification merely talks about things like 'if the map contains an entry with key x' (cf). I believe this would allow Go to have maps treat NaNs as keys specially, instead of using regular floating point equality on them. Arguably the current behavior is not in accordance with the specification (or at least how people may read it); in the cases above it's hard to say that the map doesn't contain an entry with the key 'k'.

(Of course the specification could be clarified to say that 'with key x' means 'compares equal to the key', which documents this behavior for NaNs.)

Incidentally, Go's math.NaN() currently always returns the same NaN value in terms of bit pattern. We can see this in src/math/bits.go. Unsurprisingly, Go uses a quiet NaN. Also, Go defers to the CPU's floating point operations to determine if a floating point number is a NaN:

// IEEE 754 says that only NaNs satisfy f != f.
// [...]
return f != f

If you want to manufacture your own NaNs with different bit patterns for whatever reason and then use them for something, see math.Float64frombits() and math.Float64bits() (learning the bit patterns of NaNs is up to you).

Go 1.21 may have a clear(x) builtin and there's an interesting reason why

Recently I noticed an interesting Go change in the development version, which adds type checking of a 'clear' builtin function. This was the first I'd heard of such a thing, but the CL had a helpful link to the Go issue, proposal: spec: add clear(x) builtin, to clear map, zero content of slice, ptr-to-array #56351. The title basically says what it's about, but it turns out that there's a surprising and interesting reason why Go sort of needs this.

On the surface you might think that this wasn't an important change, because you can always do this by hand even for maps. While there's no built in way to clear a map, you can use a for loop:

for k := range m {
   delete(m, k)
}

This for loop is less efficient than clearing a map in one operation, but it turns out that there is a subtle tricky issue that makes it not always work correctly. That issue is maps with floating point NaNs as keys (well, as the value of some keys). The moment a NaN is a key in your map, you can't delete it this way.

(Really, you can see it in this playground example using math.NaN().)

The cause of this issue with NaNs in maps is that Go follows the IEEE-754 standard for floating point comparison, and under this standard a NaN is never equal to anything, even another NaN or even itself. Although formally speaking delete() isn't defined in terms of specific key equality (in general maps don't quite specify things like that), in practice it works that way. Since delete() is implicitly based on key equality and NaNs never compare equal to each other, delete() can never remove key values that are NaNs, even if you got the key value from a 'range' over the map.

Of course you're probably not going to deliberately use NaN as a key value in a map. But you may well use floating point values as keys, and NaNs might sneak into your floating point values in all sorts of ways. If they do, and you have code that clears a map this way, you're going to get a surprise (hopefully a relatively harmless one). There's no way to fix this without changing how maps with floating point keys handle NaNs, and even that opens up various questions. Adding a clear() builtin is more efficient and doesn't open up the NaN can of worms.

This NaN issue was a surprise to me. Had you asked me before I'd read the proposal, I would have expected 'clear()' to be added only for efficiency and clarity. I had no idea there was also a correctness reason to have it.

(While the current change will likely be in Go 1.20, a clear builtin likely won't appear until Go 1.21 at the earliest.)

PS: If you suspect that this implies interesting and disturbing things for NaNs as key values in maps, you're correct. But that's for another entry.