Wandering Thoughts archives

The impact on middleware of expanding APIs with Go's interface smuggling

Recently, the Go blog had Keeping your Modules Compatible which is about doing exactly that as you add features and want to expand your module's API. When the module's API involves interfaces, one of the approaches they suggested is what I've called interface smuggling and what other people have called interface upgrades. Let me quote the article:

When you run into a case where you want to add a method to an existing interface, you may be able to follow this strategy. Start by creating a new interface with your new method, or identify an existing interface with the new method. Next, identify the relevant functions that need to support it, type check for the second interface, and add code that uses it.

This is a quite popular approach, one used by many packages in Go's standard library and third party packages. However, it has a dark side, and that is its unfortunate effects on middleware.

The problem for middleware is best illustrated by the most common sort of middleware, which is things that interpose in the chain of HTTP handlers to modify the results. Much middleware wants to look at or act on some aspect of the HTTP reply, for example to gather metrics based on the result code, which means that it must modify and proxy the http.ResponseWriter passed to child http.Handlers. Over time the http package has acquired a whole collection of smuggled interfaces on ResponseWriters, such as http.CloseNotifier (which is deprecated), http.Flusher, http.Hijacker, and http.Pusher. In the future there will probably be more.

(In addition, the ResponseWriter may or may not support io.ReaderFrom.)

If you're a piece of middleware, the ResponseWriter you're passed may support some, many, or all of these additional APIs. However, Go provides you no good way to pass this support through your proxy ResponseWriter that you're going to pass to children Handlers. The Prometheus people try hard to do it anyway, and the result is rather messy and involves a combinatorial explosion of the possible combinations of APIs. As the case of io.ReaderFrom shows, these additional APIs don't even necessarily come from the http package. A smuggled interface from anywhere may need to be supported.

One answer to this is that you just don't support these additional APIs in your middleware, or you only support a few of them. The problem with this is that the ResponseWriter and the client code that people are trying to use your middleware with well have been developed, tested, and normally used in an environment where these expanded APIs are used, not cut off. As we all know, if you don't test it it doesn't work. Your middleware may be the first code to try to pass the next hop a ResponseWriter with a genuinely narrow API, because such narrow APIs may mostly come from middleware. And of course if there are any bugs in the result, people will blame your middleware.

None of this is insurmountable. But beyond the problems and the hassles, it means that expanding your API with interface smuggling is decidedly not transparent if people use middleware with it. And as a practical matter, some amount of the time your new API will not be usable until middleware is expanded to cope with it (if it ever is).

Another problem is that this expansion of middleware to cope with your new API can't happen until your new API itself is pervasive. Go currently provides no support for conditional building based on the version of other packages or the state of their API, so middleware can't include any use of your new API interfaces until it doesn't have to build against versions of your package that predate them.

(People can work around this for HTTP middleware because they can make files build only on specific minimum versions of Go. Your package doesn't have this magical power; it's something available only for new APIs in the Go standard library.)

Because nothing is new under the sun, this problem was noticed back in 2014's Interface Upgrades in Go, which is one of the earliest places to call this pattern an 'interface upgrade'. The article notes the proxy problem and ends with a call to use interface upgrades sparingly. This is good advice in my opinion, but is very much at odds with the idea of routinely using interface upgrades to expand your API.

(Via.)

Ubuntu, building current versions of Firefox, and snaps

Today on Twitter, I said:

Given that Ubuntu's ostensible logic for putting Chrome in a snap is 'it makes maintaining it easier', my cynical side expects Ubuntu to also do this with Firefox before too long (due to Firefox's need for steadily increasing versions of Rust).

(The context here is LWN's Linux Mint drops Ubuntu Snap packages, via. For 'Ubuntu', you can read 'Canonical', since Canonical is driving this.)

Ubuntu ships current versions of Firefox (at the moment Firefox 78) in Ubuntu LTS releases, which means that they must build current versions of Firefox on all supported Ubuntu LTS versions. Firefox is built partly with Rust (among other things), and new releases of Firefox often require relatively recent versions of Rust; for instance, right now Firefox Nightly (which will become Firefox 80 or 81) requires Rust 1.43.0 or better. Nor is Rust the only thing that Firefox has minimum version requirements for. Firefox 78, the current release, requires nasm 2.14 or better if you want to build the AV1 codecs, and I'm sure there are others I just haven't tripped over yet.

This is a problem for Ubuntu because Ubuntu famously doesn't like updating packages on Ubuntu LTS (or probably any Ubuntu release, but I only have experience with LTS releases). Today, the need to build current Firefox versions on old Ubuntu LTS releases means that Ubuntu 16.04 has been dragged up to Rust 1.41.0 (the same Rust version that's on 18.04 and 20.04). If current versions of Rust weren't required to build Firefox, Rust on 16.04 would probably be a lot like Go, where the default is version 1.6 (that's the golang package version) and the most recent available one is Go 1.10 (which actually dates from 2018, which is modern for an LTS release from 2016). When Firefox 80 or so is released and requires Rust 1.43.0 or better, Ubuntu will have to update Rust again on all of the still supported LTS versions, which will probably still include 16.04 at that point.

Canonical can't like this. At the same time, they have to ship Firefox and they have to keep it current, for security reasons. Shipping Firefox as a Snap would deal with both problems, because Canonical would no longer need to be able to build the current Firefox from source on every supported Ubuntu release (LTS and otherwise, but the oldest ones are generally LTS releases). Given that Canonical wants to shove everyone into Snaps in general, I rather expect that they're going to do this to Firefox sooner or later.

PS: I'm not looking forward to this, because Snaps don't work with NFS mounted home directories or in our environment in general. Ubuntu moving Firefox to a Snap would probably cause us to use the official Mozilla precompiled binaries in the short term, and push us more toward another Linux release in the longer term (probably Debian).