Wandering Thoughts archives

Some notes on "closed interfaces" in Go

One reaction to basic proposals for union types in Go is to note that "closed interfaces" provide a lot of these features (cf). When I saw this I had to refresh myself about what such a closed interface is, and then think about some of the potential issues and limitations involved, leaving me with some things I want to note down for my later reference.

What I've seen called a closed interface is an interface that requires an unexported method:

type Closed interface {
  isClosed()

  NormalMethod1(...) ...
  NormalMethod2(...) ...
  [...]
}

Although it's not spelled out exactly in the Go language specification, an interface with an unexported method like this can't be implemented by any type outside of its package, because such an external type can't have the right 'isClosed()' method on it (since it's not exported, the identifier for the method is unique to the package, or at least I believe that's how the logic flows). The 'isClosed()' method doesn't do anything and need never be called, it just has to be declared as an (empty) method function on everything that is going to be part of the Closed interface.

This means that there is a finite and known list of types that implement the Closed interface, instead of the potentially unknown list of them that could exist for an open interface. Go tooling can use this knowledge to see, for example, that a type switch on Closed is exhaustive, or that a given type assertion will never succeed. However, in the current state of godoc, I don't believe that generated package documentation will automatically tell you which types implement Closed; you'll have to remember to document that explicitly.

(I don't know if there's any Go tooling that does this today, especially when run in the context of other packages instead of the package that defines the Closed interface.)

Code in other packages can still construct a nil Closed interface value, or zero values of any exported types that implement Closed (and then turn such a zero value into a non-nil Closed interface value). If you want to be extra tricky, you can make all the types that implement Closed be unexported; at that point the only way people outside your package can easily create or manipulate those types is through an instance of Closed that you give them, and implicitly only through its methods. The Closed interface is not merely a closed interface, it has become an opaque interface (short of people getting out the chainsaw of reflect and perhaps unsafe).

However, this is also a limitation of the Closed interface, and of closed interfaces in general. For good reason, you can't add methods to types declared outside your package, so you can't make an outside type be a member of the Closed interface, even though you control the interface's definition in your package. In order to induct a useful outside type into the world of Closed, you must wrap it in a type from your package, and this type must be a concrete type, even if what you want to pull in is another interface. I believe that under at least some circumstances, this will cost you some extra memory. More broadly, I don't think you can really have two separate packages that cooperate so each defines some types that are part of Closed. One way or another you have to put all the types in one package.

In my view, this means that a closed interface isn't really useful to document inside your package that this particular interface will only ever contain a limited number of outside types (or a mixture of outside types and inside types), including outside types from a related package. You can use it for this but there's a chunk of bureaucracy involved for each outside type you want to pull in. If you go to this effort, presumably you have tooling that can deduce what's going on and take advantage of this knowledge.

(These days you could define a generic type that wraps another type and implements your Closed interface for it, making this sort of bureaucracy easier, at least.)

I think Go union type proposals should start with their objectives

At this point I've skimmed a number of relatively serious union type proposals for Go (which is to say, people who were serious enough to write something substantial in the Go issue tracker). One of the feelings I've wound up with as a result of this is that any such union type proposal should probably start out by describing what its objectives are, not what its proposed syntax is.

There are a number of different things union types in Go could do, ranging from creating an interface type that only allows a limited range of concrete types to (potentially) reducing the amount of space needed to store one of several types of values to requiring people to do specific things to extract an interior value from a union value, implicitly forcing them to check for errors (or panic). Some of these objectives will be more or less complete by themselves, but some others will really want (or at least benefit from) additional changes to things like 'go vet', or in some cases be incomplete without other language changes.

Proposals that bury their objectives (or don't clearly spell them out) invite people to guess at what they really want, and then argue about it. Such proposals are also harder to evaluate, since a reader can't judge whether the proposal would have problems achieving its objectives, or even to what extent the objectives are reasonable for Go. Without objectives, people are left to discuss what the proposal does and doesn't achieve without understanding which of those things are important (and need to be kept in any changes) and which aren't (and could be altered or removed).

Presumably the proposer had an objective in mind and so can write it down. If they haven't actually considered their objectives, they should definitely start with that, not with the syntax or language semantics. I won't say that objectives are the most important thing about a change to Go, because the syntax and semantics matter too, but a clearly expressed objective is (in my view) necessary. And if you convince people that the objective is a good one, you have a much better chance of having that objective achieved somehow, even if it happens in a different way than you thought of.

(One reason that people may be reluctant to write down their objective is for fear that others won't like it and will be against the proposal as a result. One of my views is that if you can't persuade people of the objectives, your proposal should fail, and trying to sneak it in is a bad look.)

PS: One of the reasons I'm writing this down is for myself, because I was briefly tempted to write up a straw-person union types idea here on Wandering Thoughts. Had I done so, it definitely wouldn't have started from objectives, but from semantics and some effects. It's easy to have a clever idea (or at least a clever looking idea, one that sounds good for long enough for me to write a blog entry about it), but mere clever ideas are at best a distraction.

Good union types in Go would probably need types without a zero value

One of the classical big reason to want union types in Go is so that one can implement the general pattern of an option type, in order to force people to deal explicitly with null values. Except this is not quite true on both sides. The compiler can enforce null value checks before use already, and union and option types by themselves don't fully protect you against null values. Much like people ignore error returns (and the Go compiler allows this), people can skip over that they can't extract an underlying value from their Result value and return a zero value from their 'get a result' function.

My view is that the power of option types is what they do in the rest of the language, but they can only do this if you can express their guarantees in the type system. The important thing you need for this is non-nullable types. This is what lets you guarantee that something is a proper value extracted from an error-free Result or whatever. If you can't express this in your types, everyone has to check, one way or another, or you risk a null sneaking in.

Go doesn't currently have a type concept for 'something that can't be null', or for that matter a concept that is exactly 'null'. The closest Go equivalent is the general idea of zero values, of which nil pointers (and nil interfaces) are a special case (but you can also have zero value maps and channels, which also have special semantics; the zero value of slices is more normal). If you want to make Result and similar types particularly useful in Go, I believe that you need to change this, somehow introducing types that don't have a zero value.

(Such types would likely be a variation of existing types with zero values, and presumably you could only use values or assign to variables of that type if the compiler could prove that what you were using or assigning wasn't a zero value.)

As noted in a comment by loreb on my entry on how union types would be complicated, these 'union' or 'sum' types in Go also run into issues with their zero value, and as Ian Lance Taylor's issue comment says, zero values are built quite deeply into Go. You can define semantics for union types that allow zero values, but I don't think they're really particularly useful for anything except cramming some data structures into a few less bytes in a somewhat opaque way, and I'm not sure that's something Go should be caring about.

Given that zero values are a deep part of Go and the Go developers don't seem particularly interested in trying to change this, I doubt that we're ever going to get the powerful form of union types in Go. If anything like union types appears, it will probably be merely to save memory, and even then union types are complicated in Go's runtime.

Sidebar: the simple zero value allowed union type semantics

If you allow union types to have a zero value, the obvious meaning of a zero value is something that can't have a value of any type successfully extracted from it. If you try the union type equivalent of a type assertion you get a zero value and 'false' for all possible options. Of course this completely gives up on the 'no zero value' type side of things, but at least you have a meaning.

This makes a zero value union very similar to a nil interface, which will also fail all type assertions. At this point my feeling is that Go might as well stick with interfaces and not attempt to provide union types.

Union types ('enum types') would be complicated in Go

Every so often, people wish that Go had enough features to build some equivalent of Rust's Result type or Option type, often so that Go programmers could have more ergonomic error handling. One core requirement for this is what Rust calls an Enum and what is broadly known as a Union type. Unfortunately, doing a real enum or union type in Go is not particularly simple, and it definitely requires significant support by the Go compiler and the runtime.

At one level we easily do something that looks like a Result type in Go, especially now that we have generics. You make a generic struct that has private fields for an error, a value of type T, and a flag that says which is valid, and then give it some methods to set and get values and ask it which it currently contains. If you ask for a sort of value that's not valid, it panics. However, this struct necessarily has space for three fields, where the Rust enums (and generally union types) act more like C unions, only needing space for the largest type possible in them and sometimes a marker of what type is in the union right now.

(The Rust compiler plays all sorts of clever tricks to elide the enum marker if it can store this information in some other way.)

To understand why we need deep compiler and runtime support, let's ask why we can't implement such a union type today using Go's unsafe package to perform suitable manipulation of a suitable memory region. Because it will make the discussion easier, let's say that we're on a 64-bit platform and our made up Result type will contain either an error (which is an interface value) or an int64[2] array. On a 64-bit platform, both of these types occupy 16 bytes, since an interface value is two pointers in a trenchcoat, so it looks like we should be able to use the same suitably-aligned 16-byte memory area for each of them.

However, now imagine that Go is performing garbage collection. How does the Go runtime know whether or not our 16-byte memory area contains two live pointers, which it must follow as part of garbage collection, or two 64-bit integers, which it definitely cannot treat as pointers and follow? If we've implemented our Result type outside of the compiler and runtime, the answer is that garbage collection has no idea which it currently is. In the Go garbage collector, it's not values that have types, but storage locations, and Go doesn't provide an API for changing the type of a storage location.

(Internally the runtime can set and change information about what pieces of memory contain pointers, but this is not exposed to the outside world; it's part of the deep integration of runtime memory allocation and the runtime garbage collector.)

In Go, without support from the runtime and the compiler the best you can do is store an interface value or perhaps an unsafe.Pointer to the actual value involved. However, this probably forces a separate heap allocation for the value, which is less efficient in several ways that the compiler supported version that Rust has. On the positive side, if you store an interface value you don't need to have any marker for what's stored in your Result type, since you can always extract that from the interface with suitable type assertion.

The corollary to all of this is that adding union types to Go as a language feature wouldn't be merely a modest change in the compiler. It would also require a bunch of work in how such types interact with garbage collection, Go's memory allocation systems (which in the normal Go toolchain allocate things with pointers into separate memory arenas than things without them), and likely other places in the runtime.

(I suspect that Go is pretty unlikely to add union types given this, since you can have much of the API that union types present with interface types and generics. And in my view, union types like Result wouldn't be really useful without other changes to Go's type system, although that's another entry.)

PS: Something like this has come up before in generic type sets.