Wandering Thoughts

2021-04-21

Go 1.17 will allow converting a slice to an array pointer (some of the time)

In Go, slices contain a reference to their backing array, whether this is an array that exists somewhere as a distinct variable of its own or is simply an anonymous array that was allocated to support the slice (that it can be either case can sometime make slices seem like two data structures in one). The presence of this backing array can lead to interesting memory leaks or just surprising changes to your slices, but in Go through 1.16 you can't get direct access to the backing array without using the reflect and unsafe packages. However, it's possible to do this safely in some cases, and there's been a long standing Go enhancement request that it be possible in issue #395 (which dates from 2009, shortly after Go was announced and well before Go 1.0 was released).

In Go 1.17 this will be possible, due to a series of changes starting with commit 1c268431f4, which updates the specification. The specification's description of this is straightforward:

Converting a slice to an array pointer yields a pointer to the underlying array of the slice. If the length of the slice is less than the length of the array, a run-time panic occurs.

It's harmless for the slice to be longer than the length of the array (although, as usual, this will keep the entire backing array of the slice alive). A longer slice than the array only means that your array won't be able to access all of the original slice's backing array.

The specification has some examples (all comments are from the specification):

s := make([]byte, 2, 4)
s0 := (*[0]byte)(s)      // s0 != nil
s2 := (*[2]byte)(s)      // &s2[0] == &s[0]
s4 := (*[4]byte)(s)      // panics: len([4]byte) > len(s)

var t []string
t0 := (*[0]string)(t)    // t0 == nil
t1 := (*[1]string)(t)    // panics: len([1]string) > len(s)

The discussion around s2 shows that this conversion doesn't (and can't) allocate a new array, making it guaranteed to be efficient. The cases of s0 and t0 are interesting; converting a non-empty slice to a 0-length array must give you a valid pointer, even though you can't do anything with it, but converting an empty slice gives you a nil.

At the moment there's no way to do this conversion with a check to see if it would panic, the way you can with type assertions. If you think you might have a too-short slice, you need to use an if.

The reflect package has also been updated to support conversions from slices to array pointers, in commit 760d3b2a16. If you're working with reflect for this, you should read the caution from the commit:

Note that this removes an invariant:

v.Type().ConvertibleTo(t) might return true, yet v.Convert(t) might panic nevertheless.

This is a fairly unavoidable consequence of the decision to add the first-ever conversion that can panic.

ConvertibleTo describes a relationship between types, but whether the conversion panics now depends on the value, not just the type.

(I believe that you can always use reflect.ArrayOf() to create a type with the right number of elements, and then reflect.PtrTo() to create the necessary 'pointer to array of size X' type.)

GoConvertSliceToArray written at 23:46:14; Add Comment

2021-04-07

Rust's rustup tool is surprisingly nice and well behaved

I don't code in Rust (it's not my thing), but I do periodically look into other people's Rust code, most recently Julia Evans' dnspeep. When I had reason to poke into dnspeep, I decided it was time I got a LSP-enabled GNU Emacs Rust environment. Following Robert Krahn's guide, one part of this rust-analyzer, which you can either download a binary of or build it yourself, which is the option I chose. On my Fedora 33 desktops, this was a simple git clone and a 'cargo xtask install --server'. On work's Ubuntu 18.04 servers, it turned out that the stock Ubuntu 18.04 version of Rust and Cargo were too old to build rust-analyzer.

(Our login servers are still running Ubuntu 18.04 because of still ongoing world and local events.)

When I poked around in my work Ubuntu account I discovered that at some point I'd gotten a copy of rustup, the Rust toolchain installer (this is foreshadowing for another entry). As a system administrator I have a reflexive aversion to black box tools that promise to magically install something on my system, partly because I don't want my system changed (also), and this is what the rustup website presents rustup as. However, some more reading told me that rustup was much better behaved than that, and this is how it proved to be in practice. I had apparently never initialized a rustup setup in my $HOME and it took me some time to figure out what to do, but once I did, rustup was straightforward and confined itself (more or less) to $HOME/.cargo.

(As I found out later, the starting command I probably wanted was 'rustup default stable'. I went through a more elaborate chain of things like 'rustup toolchain stable' and then figuring out how to make it the default.)

That rustup doesn't insist on mangling the system shouldn't surprise me by now, because this is relatively normal behavior for other modern language package managers like Python's pip, NPM, and Go. But it's still nice and appreciated. As a sysadmin and a Unix user, I appreciate things that will install and operate on a per-user basis, especially if they confine all of their activities to a single directory hierarchy in $HOME.

One thing about rustup is that all of the binaries it manages in ~/.cargo/bin are hardlinks to itself, and so their installation time never changes (presumably unless you update rustup itself). This was a bit disconcerting when I was playing around with toolchain settings and so on; I would change things in rustup, look in .cargo/bin, and nothing would seem to have changed. Behind the scenes rustup was changing stuff in $HOME/.rustup, but that was less visible to me.

I'm not sure how I got my rustup binary in the first place. I think I probably got it through the "other installation methods" chapter, undoubtedly with at least --no-modify-path. It's a pity there's no straightforward bootstrap with cargo that I can find mentioned, because various Linuxes supply Cargo and Rust but not necessarily rustup.

(This elaborates on a tweet of mine and will hopefully get this pleasant side of rustup to stick in my mind for any future needs. Because Ubuntu 18.04 is so long in the tooth by now, I also had to shave some other yaks to get a modern Rust LSP environment set up in GNU Emacs.)

RustupFairlyNice written at 21:51:55; Add Comment

2021-04-05

A stable Unix updating its version of Go isn't straightforward

Recently, I was implicitly grumpy at Ubuntu 18.04 LTS for not updating its Go version from Go 1.10. The reason to want an update is straightforward; Go 1.10 doesn't support modules and you need module support. However, after thinking about it more I had the realization that a stable Linux or Unix distribution updating Go versions within a single release isn't as straightforward as it looks. In fact, Go 1.10 is a particularly awkward version to update from (as will be Go 1.15).

Most distributions that are going to update something within a stable release want the new version of whatever it is to be a more or less drop in and transparent replacement for the old version. On the surface, the Go 1 compatibility guarantee seems like Go should be fine here, since new versions of Go will compile your code so that it does the same thing as your current version. However, that isn't all that people care about. In order to be a drop in replacement, people need to be able to develop with a new version of Go in the same way and it should have the same broad runtime behavior (or better). Neither of these are covered by the Go 1 compatibility guarantee, and in fact the Go team have explicitly stated that the Go toolchain is not.

For Ubuntu 18.04 and Go 1.10, there are two issues. First, going from Go 1.15 and below to Go 1.16 is a breaking change in tool behavior for existing setups, because the default behavior changed to modular mode even under $GOPATH/src. In Go 1.10, everything was in $GOPATH/src and you had to work there (and any go.mod files were ignored); from Go 1.16 onward, none of that works by default. Even Go 1.13 might be a behavior change, since it enables module mode in the presence of a go.mod file even when you're under $GOPATH/src, which means that some programs may build differently between the two versions.

Second and smaller, from Go 1.12 through Go 1.15 Go programs reported a higher RSS, which could (and did) cause issues in environments where these were either monitored or limited. Going from this range to Go 1.16 improves things (which is a change but not one people are likely to object to), but of course that gives you the Go 1.16 module behavior change. This change in runtime behavior from Go 1.10 (or Go 1.11) might be considered an acceptable change in some Unix distributions, but there are others that would probably reject it as too much.

If both behavior changes are unacceptable in a stable release, then Go 1.10 could only be updated to Go 1.11, which is hardly worth it (1.11 is far out of support and I'm not sure how good its module support was). Ubuntu 20.04 has Go 1.13, which could probably be updated to Go 1.15 but certainly not to Go 1.16. Debian stable has Go 1.11 which could theoretically go to Go 1.12 but probably not Go 1.13 given Debian's cautious attitude on changes.

(I haven't checked CentOS 8 but it's probably similarly constrained in Go versions. I don't know if FreeBSD ports are considered stable in this same way.)

To full time Go programmers, much of this may seem picky on the part of Linux distributions. To the users of the Linux distributions that advertise stability (and to the distributions themselves), it's not. People count on being able to routinely apply package updates without careful testing and surprise changes in behavior that break workflows and other things.

(In a way this is another aspect of frequent versus infrequent developers. If you're a frequent Go developer, you probably already have your own toolchain. It's the infrequent Go developer that is probably using whatever Go toolchain comes with their OS because it's usually good enough.)

PS: I'm still going to be sad that Ubuntu 18.04's standard version of Go isn't really good enough any more, but that's just because it would make my life at work slightly easier if it was.

GoVersionsAndStableUnixes written at 23:08:39; Add Comment

2021-04-04

You need a version of Go with module support (ideally good support)

Linux distributions and other Unixes often package versions of languages for you. This has good bits, such as easy installation and someone else worrying about the packaging, and bad bits, such as the versions potentially being out of date. For many languages (C being a great example), the exact version doesn't matter too much, and most any version installed by a Linux distribution will be fine for most people. Unfortunately Go's current strong transition to modules makes it not one of those languages; you now need a version of Go with (good) module support. Which leads to this tweet of mine:

It makes me a bit sad that Ubuntu 18.04's packaged version of Go is 1.10, which is one version too old to have any modules support. It's very Ubuntu but I still wish they'd updated since release.

On operating systems with versions of Go that are too old, such as Ubuntu 18.04, you should get or build your own local copy of a good version of Go, which probably should be the latest version. Fortunately this is an easy process; unfortunately it will complicate simply working with programs that written in Go and that you need to build yourself (possibly including locally written programs). Even if you can sort of work with the system version of Go when it lacks module support, I don't think you should try to do so; it's not going to be worth the extra pain and work.

(Related to this, you might want to start installing third party Go programs with 'go install ...@latest' or 'go get ...@latest', since both of these force modular mode if they work at all.)

There are at least three reasons to use a version of Go that supports Go modules and to use Go modules yourself. First, it's the way that the ecosystem is moving in general. I expect that an increasing number of development tools and general programs that work with Go code are going to require and assume modules, or at least work much better with modules. Like it or not, traditional $GOPATH Go development is going away as people drop code for it or the old code quietly stops working.

Second, Go modules are the mostly the better way to develop your own local programs. They complicate your life if you have your own local packages that you don't want to publish somewhere, but if you don't use local, un-published packages then they're a great way to get self-contained Go program source. And of course modules make it far less likely that random changes in outside packages that you use will break your programs.

Third, it's likely that a steadily increasing number of third party programs will stop building without modules, because they're set up to use older versions of third party packages (or older versions of their own packages). You could generally build them by hand with a carefully constructed $GOPATH environment, but using a version of Go with module support is much easier.

GoModuleSupportNeed written at 01:08:15; Add Comment

2021-03-29

Nil in Go is typed in theory and sort of untyped in practice

I was recently reading Down the Golang nil Rabbit Hole (via), and ran into the statement part way through:

So here’s a fun fact, Go has multiple “types” of nil. Specifically, there are both typed and untyped nil variables.

Some of you reading this are now reaching for your keyboards (cf), and that was my first reaction too. But I think it's more interesting to talk about how someone could wind up feeling that nil is sometimes untyped, because Go's nil is a little bit slippery when you combine it with other features and a necessary API for some things. So let's start with what nil is in Go and how it behaves in some circumstances.

In code, a literal of 'nil' effectively works as if it was an untyped constant like '0' (although it's not a constant; it's a predeclared identifier with special semantics that are sprinkled all through the specification). However, just like untyped constants, when you use nil by assigning it to a variable or passing it as a function argument, the resulting Go value is always typed, whether it is a pointer, function, slice, map, channel, or interface of some type. For concrete types that can be nil (pointers, functions, slices, maps, and channels), the type of even these nil values is readily observed. The fmt package's %T verb will report it, for example, and you can examine this type through reflect. However this is not quite the case for interfaces.

Like most everything else in Go, interfaces are typed, including the famous empty interface, interface{}, and as a result so are interface values. An interface value that is nil still has a type. However, this interface type (of interface values) is very hard to observe dynamically (in code) because of another property of Go: when you convert one interface type to another interface type, the original interface type is lost. If an interface value is not nil, it has an underlying concrete type, which is preserved when the interface value is converted to another interface type (although the original interface type is lost). But if the interface value is nil, there's no underlying concrete type and so there's no other type information to be preserved; you can't tell a nil of one interface type from a nil of another interface type.

Now we get to the subtle Go trap. Neither fmt nor reflect have a magic exemption from the Go type system in order to accept arbitrary Go types. Instead, they must use the Go escape hatch of the empty interface, interface{}, which everything can be converted to. But this means that when you call fmt or reflect on an interface value, you're performing an interface conversion, and as a result you lose the type of the original interface value. For instance, suppose you write code like this:

func fred(e error) {
  fmt.Printf("%#v %T\n", e, e)
}

The error type is an interface type. When you call fmt.Printf, the error interface value is converted to the interface{} type, and fmt.Printf now has no access to the fact that e was an error; all it can report is the concrete underlying type, if there is one. If you call 'fred(nil)', fmt winds up being passed a nil value of type interface{} and can only shrug about what type that value originally was.

(By extension, the fmt '%T' verb can never tell you what interface type was being used. If you want to know that, you must pass a pointer to the interface value and then '%T' that, which will report things like '*error'. See this playground example, which demonstrates that this does preserve the interface type for a nil interface value, as you'd expect.)

In this sense, nil interface values are untyped, in that their type is very hard to observe dynamically and check in practice in ordinary Go code. Due to this, a lot of perfectly normal Go code will tell you that such values have no type (fmt's '%T' verb will report their type as '<nil>', for example).

(Perhaps their type should be reported as 'interface{}', but that ship has sailed and it wouldn't be all that much more useful in practice.)

The other way to put this is that anything that takes interface{} arguments can't distinguish between nil values of different interface types, and by extension also can't distinguish between called with a literal nil and being called with an interface value that happens to be nil (for example, because it's a value of (interface) type error and there was no error). Both fmt and reflect take interface{} arguments and so are limited by this. If you're not aware of this indistinguishability, things can look pretty puzzling (or you can get caught out by it).

(This is another facet of how sometimes a nil is not a nil, which is sufficiently common to have a FAQ about it.)

GoNilIsTypedSortOf written at 22:20:53; Add Comment

2021-03-22

Portability has ongoing costs for code that's changing

I recently had a hot take on Twitter:

Hot take: no evolving project should take patches for portability to an architecture that it doesn't have developers using, or at least a full suite of tests and automated CI on. Especially if the architecture is different from all supported ones in some way (eg, big-endian).

Without regular usage or automated testing, it's far too likely that the project will break support on the weird architecture. And keeping the code up to date for the weird architecture is an ongoing tax on other development.

Some proponents of alternate architectures like to maintain that portability is free, or at least a one time cost (that can be paid by outside contributors in the form of a good patch to 'add support' for something). It would be nice if our programming languages, habits, and techniques made that so, but they don't. The reality is that maintaining portability to alternate environments is an ongoing cost.

(This should not be a surprise, because all code has costs and by extension more code has more costs.)

To start with, we can extend the general rule that 'if you don't test it, it doesn't work'. If alternate environments (including architectures) aren't tested by the project and probably used by developers, they will be broken sooner or later. Not because people are necessarily breaking them deliberately, but because people overlook things and never have the full picture. This happens even to well intentioned projects that fully want to support something that they don't test, as shown by Go's self-tests. Believing that an alternate architecture will work and keep working without regular testing goes against everything we've learned about the need for regular automated testing.

(I believe that for social reasons it's almost never good enough to just have automated tests without developers who are using the architecture and are committed to it.)

Beyond the need for extra testing and so on, portability is an ongoing cost for code changes. The alternate architecture's differences and limits will need to be considered during programming, probably complicating the change, then some changes will be broken only on the alternate architecture and will require more work to fix. Sooner or later some desired changes won't be possible or feasible on the alternate architecture, or will require extra work to implement options, alternatives, or additional things. In some cases it will require extreme measures to meet the project's quality of implementation standards for changes and new features. When something slips through the cracks anyway, the project will have additional bug reports, ones that will be difficult to deal with unless the project has ongoing easy access both to the alternate environment and to experts in it.

More broadly, the alternate architecture is implicitly yet another thing that programmers working on the project have to keep in mind. The human mind has a limited capacity for this; by forcing people to remember one thing, you make it so that they can't remember other things.

The very fact that an alternate architecture actually needs changes now, instead of just building and working, shows that it will most likely need more changes in the future. And the need for those changes did not arise from nothing. Programmers mostly don't deliberately write 'unportable' code to be perverse. All of that code that needs changes for your alternate architecture is the natural outcome of our programming environment. That code is the easiest, most natural, and fastest way for the project to work. The project has optimized what matters to it, although not necessarily actively and consciously.

(My tweets and this entry were sparked by portions of this, via. The whole issue has come up in the open source world before, but this time I felt like writing my own version of this rejoinder.)

PortabilityOngoingCosts written at 23:53:56; Add Comment

2021-03-10

The tradeoffs of Go version behavior in go.mod module files

As covered in the official documentation, the go.mod file that contains various information about a Go module can specify a Go version. This specified version does not actually do much in practice, at least so far, and in particular older versions of Go will still attempt to build the module. This behavior and others I mentioned yesterday may strike people as peculiar or even crazy. However, any behavior here has tradeoffs and I think that Go has chosen what is a sensible option.

The core tradeoff is between what Go version is assigned to a newly created module and what older versions of Go will do with modules that name a newer version. If a newly created module gets the current version of Go and older versions of Go will not build newer modules, then your newly created module is not useful for anyone who is using an older Go version (even though there's probably nothing in the code of the module that requires the latest Go version). This encourages people to specify older versions by hand or just to keep using older Go versions to avoid accidents. It's also a recipe for projects to get annoying issue reports of 'can you change your go.mod to set an older version of Go that this works fine on (and do a new release)'.

On the other hand, if a newly created module gets an older version of Go, then you can't immediately use any new features of your version of Go in your new code without a confusing extra manual step. This at least discourages people from using new Go features they could benefit from. And if older versions of Go won't build modules with newer Go versions, people still may not be able to build your module on some Go versions that they may want to use (such as the one supplied by their operating system, which may be sadly out of date), unless the default Go version in modules is all the way back at Go 1.12 (where the 'go' directive was introduced).

(Setting the Go version back also creates a battleground of people arguing how far back it should be. No one is going to be happy.)

If you allow older versions of Go to build modules claiming to require newer versions of Go, you more or less have to not change existing behavior, only add new behavior and new features that were previously some sort of error. If you change existing behavior, older Go versions will silently mis-compile the code, possibly introducing both bugs and security issues. If the version of Go in go.mod is a hard requirement, you can change existing behavior; if a module's author increases the Go version, it's their responsibility to make sure the result really works right. However, changing existing Go behavior is at least against the spirit of the Go 1 compatibility guarantee.

In addition, Go already has a way to implement hard Go version requirements, in the form of build directives. Build directives are also per file, not per package or module, which potentially allows you to write a module that uses new Go features but will still build and work for people with older Go versions.

The path Go has chosen (partly implicitly) allows people to use new Go features immediately and without nagging in new Go code (or at least new modules) but makes a best effort attempt to let such new code work with older versions of Go. If the code genuinely needs new Go features, it will most likely fail to build on older versions of Go with actual errors, and you'll know that the go.mod requirement is a real one.

Sidebar: The engineering reason to not change existing Go behavior

If the behavior of existing valid Go code depends on the version of Go it's for, someone reading the code in the future must both know what version of Go the code is for and remember that there's a difference in behavior in that particular area of Go between versions (and between what versions). But the version of Go is only specified in the go.mod file, not even in the Go source code file you're reading, and people are bad at remembering the exact details of changed behavior or even that it exists. Mistakes and bugs would be practically guaranteed.

GoModulesGoVersionWhy written at 00:19:05; Add Comment

2021-03-09

Go version directives in go.mod files: some notes and crude usage numbers

Go modules use a go.mod file to specify various things. One of the directives that they can include is the go directive, which sets 'the expected language version of the module', to quote the specification's words. The specification goes on to say a bit about what it's used for.

A 'go' line in go.mod does not limit what versions of Go will (try) to build the module (well, packages in the module), although of course if the module uses something that's not in older versions of Go, building will probably fail. This is important behavior, because in current versions of Go, running 'go mod init' will write the current version of Go into your new go.mod file as the 'go 1.nn' version. This is reasonably behavior and probably won't cause you problems, but you might want to watch out for it. The Go version specified in the go.mod can predate modules; in my assortment of Go programs (and vendored packages), I have one that has 'go 1.9' and two that have 'go 1.10'.

I have a collection of various people's Go programs, their vendored modules, and the dependent packages for the remaining non-modular programs sitting around. In that collection, the distribution of Go versions in go.mod files is this:

    31 go 1.13
    23 go 1.14
    20 go 1.12
    15 go 1.15
    15 go 1.11
     2 go 1.10
     1 go 1.9

(A real study should try to do some measure of the most frequently imported or used Go packages.)

I don't think 'go' directives are ever automatically updated by Go module commands, so this represents some combination of what Go version the modules were created with and what Go version people have explicitly decided to call for (whether to insure backward compatibility or to set a minimum version).

I expect that in the future we're going to see a burst of 'go' directives that specify Go 1.17 or whatever Go version modules become mandatory in, as people are forced to turn long ignored programs into modules. Some of that may be happening now in Go 1.16, since Go will now prod you about this if you haven't explicitly set $GO111MODULE to something.

PS: The development version of Go generally considers itself to be one Go version in the future for this sort of thing, so right now if you do a 'go mod init' with a development version it will create a go.mod with 'go 1.17' in it. This is reasonable behavior, but might be slightly surprising.

GoModulesGoVersions written at 00:08:55; Add Comment

2021-02-05

Limiting what branches I track from an upstream Git repository

I track a number of upstream Git repositories for software we use or that I'm interested in, where by 'track' I mean that I keep a local copy and update it periodically. I've been growing more and more unhappy with how noisy this process has been getting, so recently I did some work on making this tracking quieter. Sadly this left me with one remaining pain point, the repository for Grafana.

Grafana's repository is unusual because the Grafana developers work in a profusion of short lived feature branches that appear in the main repository. When I do a 'git pull' of the Grafana repository after only a few days, I get a shower of newly created branches and another shower of recently deleted remote branches. As far as I can see, 'git fetch' has no particularly good way to suppress this output through a .git/config option; the most you can do is run 'git fetch' entirely quietly.

In order to deal with this, I've now switched to tracking only a limited range of upstream branches. Ideally I would only need to track one upstream branch, the main one, but in practice the Grafana developers create a separate branch for each significant release and I also want to track those branches, just in case. Because of this need to track additional branches combined with Grafana's branch naming practices, the result is a little messy and imperfect.

The normal Git refspec for an upstream 'origin' repo is, in .git/config format:

[remote "origin"]
  fetch = +refs/heads/*:refs/remotes/origin/*

(The plus sign is a bit magical but is probably okay here.)

To track only a limited subset of branches, we need to change this to one or more 'fetch =' lines that are more specific. Fortunately, you can have multiple 'fetch =' lines; unfortunately, Git's support for wildcard matching here is relatively limited, so we can only do so much given Grafana's branch naming scheme. What I use is:

[remote "origin"]
  fetch = +refs/heads/master:refs/remotes/origin/master
  fetch = +refs/heads/v*:refs/remotes/origin/v*

Grafana's version branches are called things like 'v7.4.x', but it also has a few other branches that start with 'v' and we can't use a wildcard match of 'v[1-9]*'. I could list out all of the current major versions that I care about, but that would leave me having to manually change things every so often (such as when 'v8.0.x' is created, which the Grafana people are working on). For now I've decided to accept a few extra branches (currently four); if more unwanted 'v*' branches show up, I may change my mind.

(More recent versions of Git than any version I have access to allows for negative refspecs in 'git fetch', and even documents wildcards a bit in the git-fetch manpage.)

This change of 'fetch =' lines limits what will be pulled in the future, but it doesn't do anything to prune the unwanted feature branches I currently have (and I'm not certain git will still notice deleted upstream branches, since I've probably told it to not look up information on them). To get rid of them, we need to manually delete the local 'origin/<...>' branches using 'git branch -d -r origin/<...>'. I generated a list of branches to remove by starting with 'git branch -r' and then filtering out remote branches I didn't want to remove, then used a big shell for loop to do the work.

Possibly there are better Git ways to do this. I am somewhat of a brute force person when it comes to Git, because I don't know the ins and outs; when I want to do something new, I make a strategic expedition into the manpages and Internet searches, looking for a relatively obvious answer.

PS: Grafana could make my life easier here by putting all of their release branches in a specific namespace, like 'release/v7.4.x'. Then I could have a fetch spec of 'refs/heads/release/*' and it wouldn't match any other feature branches, because of course you wouldn't put feature branches in release/.

GitUpstreamLimitedTracking written at 22:39:11; Add Comment

2021-02-01

Go 1.16 will make system calls through libc on OpenBSD

One of the unusual things about Go is that it started out with the approach of directly making system calls on Unix, instead of calling the standard C library functions that correspond to those system calls. There are reasonably good reasons for Go to make direct system calls and this works well on Linux, but other Unixes are different. The official API for Illumos and Solaris system calls requires you to use their C library, and OpenBSD wants you to do this as well for security reasons (for OpenBSD system call origin verification). Go has used the C library on Solaris and Illumos for a long time, but through Go 1.15 it made direct system calls on OpenBSD and so current released versions of OpenBSD had a special exemption from their system call origin verification because of it.

The news of the time interval for Go 1.16 is that this is changing. To quote from the current draft release notes (which are probably soon to be the official release notes):

On the 64-bit x86 and 64-bit ARM architectures on OpenBSD (the openbsd/amd64 and openbsd/arm64 ports), system calls are now made through libc, instead of directly using the SYSCALL/SVC instruction. This ensures forward-compatibility with future versions of OpenBSD. In particular, OpenBSD 6.9 onwards will require system calls to be made through libc for non-static Go binaries.

As far as I know, Go programs that look up host names or do a few other operations are very likely to not be statically linked. You can force static linking (and you'll normally get it if you cross-build), but it has some drawbacks for hostname lookups in some configurations and you can't do some other operations at all.

At one level everything is okay with this situation. OpenBSD 6.9 will almost certainly include Go 1.16 in its ports collection, since it will be the only version of Go that works on it, and from there you can build Go programs that will run fine on 6.9. At another level, any dynamically linked Go program you have will need to be rebuilt with Go 1.16 before you can run it on OpenBSD 6.9. Hopefully you have the source code and can still build it (in what will be a 'modular by default' world in Go 1.16). This is nothing really new for OpenBSD, which has always made it clear that they don't promise ABI or even API compatibility; you always need to be prepared to rebuild your programs for new OpenBSD versions, and perhaps to update them to more secure APIs.

(Statically linked Go programs built by Go 1.15 or earlier will likely keep working on OpenBSD 6.9, assuming that there are no other ABI changes that affect them. But you should probably plan to rebuild them with Go 1.16 just to be sure. I don't know what the situation will be if you want to create Go binaries that work across a range of OpenBSD versions.)

As the release notes say, Go 1.16 will make system calls through libc for all programs, whether they're dynamically linked or statically linked. Right now OpenBSD only requires this for dynamically linked programs (well, will require it), but always calling via libc is simpler than to maintain two sets of system call code. And someday OpenBSD may do something more elaborate so that making system calls via libc is required even for statically linked programs.

Go116OpenBSDUsesLibc written at 22:02:23; Add Comment

(Previous 10 or go back to January 2021 at 2021/01/29)

Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.