Wandering Thoughts

Go's net package doesn't have opaque errors, just undocumented ones

I tweeted:

I continue to be irritated by how opaque important Go errors are. I should not have to do string comparisons to discover that my network connection failed due to 'host is unreachable'.

The standard library net package has a general error type that's returned from most network operations. If you read through the package documentation straightforwardly, as I did in this tweet, you will likely conclude that the only reasonable way to see if your net.Dial() call to something has failed because your Unix is reporting 'no route to host' is to perform a string match against the string value of the error you get back.

(You want to do that string match against net.OpError.Err, since that's what gets you the constant error string without varying bits like the remote host and port you're trying to connect to.)

As I discovered when I started digging into things in the process of writing a different version of this entry, things are somewhat more structured under the hood. In fact the error that you get back from net.Dial() is likely to be all officially exported types and you can do a more precise check than string comparisons (at least on Unix), but you have to reach through several layers to see what is going on. It goes like this:

• net.Dial() is probably returning a *net.OpError, which wraps another error that is stored in its .Err field.

• if you have a connection failure (or some other specific OS level error), the *net.OpError.Err value is probably an *os.SyscallError. This is itself a wrapper around an underlying error, in .Err (and the syscall that failed is in .Syscall; you could verify that it's "connect").

• this underlying error is probably a *syscall.Errno, which can be compared against the various E* errno constants that are also defined in syscall. Here, I'd want to check for EHOSTUNREACH.

So we have a *syscall.Errno inside an *os.SyscallError inside a *net.OpError. This wrapping sequence is not documented and thus not covered by any compatibility guarantees (neither is the string comparison, of course). Since all of these .Err fields are declared as type error instead of concrete types, unwrapping the whole nesting requires a bunch of checked type casts.

If I was doing this regularly, I would probably bother to write a function to check 'is this errno <X>', or perhaps a list of errnos. As a one-off check, I don't feel particularly guilty about doing the string check even now that I know it's possible to get the specific details if you dig hard enough. Pragmatically it works just as well, it's probably just as reliable, and it's easier.

(You still need to do a checked type cast to *net.OpError, but that's as far as you need to go. If you don't even want to bother with that, you could just string-ify the whole error and then use strings.HasSuffix(). For my purposes I wanted to check some other parts of the *net.OpError, so I needed the type cast anyway.)

In my view, the general shape of this sequence of wrapped errors should be explicitly documented. Like it or not, the relative specifics of network errors are something that people care about in the real world, so they are going to go digging for this information one way or another, and I at least assume that Go would prefer we unwrap things to check explicitly rather than just string-ifying errors and matching strings. If there are cautions about future compatibility or present variations in behavior, document them explicitly so that people writing Go programs know what to look out for.

(Like it or not, the actual behavior of things creates a de facto standard, especially if you don't warn people away. Without better information, people will code to what the dominant implementation actually does, with various consequences if this ever changes.)

How I want to use Go's versioned modules

I thought that I understood Go's 'vgo' versioned modules after reading things like Russ Cox's "Go & Versioning" series. Then I started poking at them for the usage cases I'm interested in, and now I'm more confused than before.

The available Go documentation makes it pretty clear how to work with Go modules for your own code, and there are walk-throughs like Dave Cheney's Taking Go modules for a spin. But much of my use of Go is in fetching and building other people's Go programs (eg, a handy TLS certificate inspector, this handy program that I should use more often, and of course my favorite Let's Encrypt client). It's not clear how Go modules interact with this in a future world where these packages have go.mod files that specify what versions of their dependencies they should be built with, and certainly there doesn't seem to be any interaction now.

When I'm building Go programs like this, I'm acting as what I've called an infrequent developer and I basically want things to just work. If a program has a go.mod file, the most likely way to have things just work is to use the dependency version information from go.mod (it's what the program is advertising as right, after all). For usability I want this to happen automatically on plain 'go get <package>' or something very like it, because that's what I and many other people are going to use.

(I absolutely will not be manually cloning VCS repos to somewhere, cding to it, and running 'go build' just so I can have Go respect the package's go.mod. It's an extremely useful feature of Go that I can go from nothing to an installed program with a single command.)

All of this leads me to want a model of go.mod usage where Go commands respect go.mod if one is present but still work and behave traditionally if there isn't one. I want this to happen whether or not the package in question is in $GOPATH/src, partly because that means I don't have to care whether any particular program has added a go.mod yet. The Go developers don't seem to have any interest in supporting this approach, though; perhaps they consider it too unpredictable.

(I consider it very predictable; I will get whatever the authors of the module think is best. If they like go.mod, I'll automatically use that; if they vendor some or all things, I'll use that; otherwise, I'll use the Go default of 'the latest version of everything', which they're presumably fine with since they left their program that way.)

PS: Given that the latest Go tip still doesn't seem to have any way of using a package's go.mod if you just do 'go get <package>', I suspect that the Go developers consider handling this in any way to be out of scope for the first version of Go modules. These days I'm not sure they even like 'go get <package>', or if they've switched over to considering it a mistake that they're more or less locked in to supporting to some degree.

PPS: For existing packages you have fetched, you could get what I want by writing a cover script for go that manipulates $GO11MODULE based on whether or not there's a go.mod file in an appropriate spot. Having to write this cover script seems wasteful, though, since Go is already perfectly capable of checking for itself.

(At least according to the documentation, setting $GO11MODULE to on doesn't quite do this. Instead it claims to make use of go.mod mandatory, at least as I'm reading the 'go help modules' documentation. The actual behavior in my tests doesn't necessarily match this, so the whole thing leaves me confused.)

One advantage of Go modules will be less mess in $HOME/go/src

One of the things coming in Go 1.11 is (experimental) support for 'vgo' package versioning; you can read some (in-progress) documentation here and here. One of the reasons I'm looking forward to projects adopting it is that it holds great promise for cleaning up my $HOME/go/src tree.

The problem with my $HOME/go/src currently is that it mingles together programs (and sometimes packages) that I'm actively using and tracking along with their various, assorted, and ever-changing dependencies. I care about the programs but not the dependencies, but with them co-mingled it's hard to keep either straight and thus hard to notice and clean up when a dependency is no longer needed. I generally only notice when I spot pending updates that are still there even after I 'go get -u' all of the actual programs I'm using. In a 'vgo' world, the dependencies will get downloaded and stored in a separate area from the actual packages I'm building; I can purge that area every so often if I want to.

(Of course doing this purge is slightly risky, since a dependency might have removed its canonical master version. It would be nice if Go would track what dependencies in my cache are needed by what (and how recently), but I don't really expect that to happen.)

I was going to say something about how the new module aware behavior of 'go get' is not ideal for my case, but now that I read the documentation I'm not sure of that because I'm not sure I understand how using 'go get' to just fetch and build a program is supposed to work now. I suspect that there's going to be some behavior changes as people talk to the Go developers (and I also rather suspect that the Go developers will find broad objections to getting rid of support for things without go.mod files). In any case it looks like I won't be able to make any sort of near term transition; this is instead going to be a long term kind of thing.

(Modules would directly help us with something at work, except that we're going to be on Go 1.10 for years to come because that's what Ubuntu 18.04 comes with.)

(I've sort of written about the clutter issue before in this entry.)

Sidebar: How I want a module-aware 'go get' to behave

Basically I want a mode where Go prefers being module enabled if there's a go.mod available but falls back to the current behavior if there isn't. For 'go get', this would mean downloading the package to $GOPATH/src as it does today (because there's no other good place to put it), then either getting its dependencies as modules (when it has a go.mod) or fetching them into $GOPATH/src as today (if there's no go.mod), then building using the appropriate bits (either module aware or not). This would provide an essentially transparent upgrade path as Go programs that I build start migrating to using Go modules and having a go.mod.

(Right now it's not clear to me how you're supposed to correctly fetch and build a Go program that uses go.mod.)

You should probably write down what your math actually means

Let's start with my tweet:

I have no brain, but at least I can work out what this math in my old code actually means, something I didn't bother to do when I wrote the original code & comments years ago. (That was a mistake, but maybe I didn't actually understand the math then, it just looked good.)

For the almost authentic experience I'm going to start with the code and its comments that I was looking at and explain it afterward.

       # if we are not going to protect a lot more data
       # than has already been protected, no. Note that
       # this is not the same thing as looking at the
       # percentages, because it is effectively percentages
       # multiplied by the number of disks being resilvered.
       # Or something.
       if rdata < psum*4 or rdata < esum*2:
               return (False, "won't protect enough extra data")

ZFS re-synchronizes redundant storage in an unusual way, which has some unfortunate implications some of the time. I will just quote myself from that entry:

So: if you have disk failure in one mirror vdev, activate a spare, and then have a second disk fail in another mirror and activate another spare, work on resilvering the first spare will immediately restart from scratch. [...]

My code and comment is from a function in our ZFS spares handling system that is trying to decide if it should activate another spare even though it will abort an in-progress spare replacement, or if it should let the first spare replacement finish.

The problem with this comment is that while it explains the idea of this check to a certain extent, it doesn't explain the math at all; the math is undocumented magic. It's especially undocumented magic if you don't know what rdata, psum, and esum are and where they come from, as I didn't when I was returning to this code for the first time in several years (because I wanted to see if it still made sense in a different environment). Since there's no explanation of the math, we don't know if it actually express the comment's idea or if it's making some sort of mistake, perhaps camouflaged by how various terms are calculated.

(It's not that hard to get yourself lost in a tangle of calculated terms. See, for example, the parenthetical discussion of how svctm is calculated in this entry.)

In fact when I dug into this, it turns out that my math was at least misleading for us. I'll quote some comments:

# psum: repaired space in all resilvering vdevs in bytes
# esum: examined space in all resilvering vdevs in bytes
[...]
# NOTE: psum == esum for mirrors, but not necessarily for
# raidz vdevs.

Our ZFS fileservers only have mirrors and none of our spares handling code has ever been tested on raidz pools. Using both psum and esum in my code was at best a well intentioned brain slip, but in practice it was misleading. Since both are the same, the real condition is the larger one, ie 'rdata < psum*4'. rdata itself is an estimate of how much currently unredundant data we're going to add redundancy for with our new spare or spares.

To start, let's rewrite that condition to be clearer. Ignoring various pragmatic math issues, 'rdata < psum*4' is the same as 'rdata/4 < psum'. In words and expanding the variables out, this is true if we've already repaired at least as much data as one quarter of the additional data we'd make redundant by adding more spares.

Is this a sensible criteria in general, or with these specific numbers? I honestly have no idea. But at least I now understand what the math is actually doing.

In fact it took two tries to get to this understanding, because it turns out that I misinterpreted the math the first time around, when I made my tweets. Only when I had to break it down again to write this entry did I really work out what it's doing. This really shows very vividly that the moment you understand your math (or think you do), write that understanding of your math down. Be specific. It's not necessarily going to be obvious to you later.

(If you work on some code all the time, or if the math is common knowledge in the field, maybe not; then it falls into the category of obvious comments that are saying 'add 2 + 2'. Also, perhaps better variable names could have helped here, as well as avoiding the too-clever use of a multiplication instead of a division.)

PS: Since I wrote 'Or something.' even in the original comment, I clearly knew at the time that I was waving my hands at least a bit. I should have paid more attention to that danger sign back then, but I was probably too taken with my own cleverness. When it comes ot this sort of math and calculation work, this is an ongoing issue and concern for me.

The downsides of processing files using too large a buffer size

I was recently reading this entry by Ben Boyter, part of which is his discussion of some attempts to optimize file IO. In these attempts he varied his buffer sizes, from reading the entire file more or less at once to reading in smaller buffers. As I thought about this, I had a belated realization about file buffer sizes when you're processing the result as a stream.

My normal inclination when picking read buffer sizes for file IO is to pick a large number. Classically this has two good effects; it reduces system call overhead (because you make fewer of them) and it gets the operating system to do IO to the underlying disks in larger chunks, which is often better. However, there is a hidden drawback to large buffer sizes, namely that reading data into a buffer is a synchronous action as far as your program is concerned; under normal circumstances, the operating system can't give your program back control until it's put the very last byte of the buffer into place. If you ask for 16 Kb, your program can start work once byte 16,384 has shown up; if you ask for 1 Mbyte, you get to wait until byte 1,048,576 has shown up, which is generally going to take longer. The more you try to read at once, the longer you're going to stall.

On the surface this looks like it reduces the time to process the start of the file but not necessarily the time to process the end (because to get to the end of a 1 Mbyte file, you still need to wait for byte 1,048,576 to show up). However, reading data is not necessarily a synchronous action all the way to the disk. If you're reading data sequentially, all OSes are going to start doing readahead. This readahead means that you're effectively doing asynchronous disk reads that at least partially overlap with your program's work; while your program is processing its current buffer of data, the OS is issuing readaheads and may be able to satisfy your program's next read by just copying things around in RAM, instead of waiting for the disk.

If you attempt to read the entire file before processing any of it, you don't get any of these benefits. If you read in quite large buffers, you probably only get moderate benefits; you're still waiting for relatively large read operations to finish before you can start processing data, and the OS may not be willing to do enough readahead to cover the next full buffer. For good results, you don't want your buffer sizes to be too large, although I don't know what a good size is these days.

(Because I like ZFS and the normal ZFS block size for many files is 128 Kb, I think that 128 Kb is a good starting point for a read buffer size. If you strongly care about this, you may want to benchmark on your specific environment, because it's going to depend on how much readahead your OS is willing to do for you.)

PS: This also depends on your processing of the file not taking too long. If you can only process the file at a rate far lower than the speed of your IO, IO time has a relatively low impact on things and so it may not matter much how you read the file.

(In retrospect this feels like a reasonably obvious thing, but it didn't occur to me until now and as mentioned I've tended to reflexively do read IO in quite large buffer sizes. I'm probably going to be changing that in the future, at least for programs that process what they read.)

What I use Github for and how I feel about it

In light of recent events (or at least rumours) of Microsoft buying Github, I've been driven to think about my view of Github and how I'd feel about it changing or disappearing. This really comes in two sides, those of someone who has repos on Github and those of someone who uses other people's repos on Github, and today I feel like writing about the first (because it's simpler).

Some people probably use Github as the center of their own work, and to a certain extent Github tries hard to make that inevitable if you have repositories that are popular enough to attract activity from other people (because they'll interact with your Github presence unless you work hard to prevent that). In my case, I don't have things set up that way, at least theoretically. Github doesn't host the master copy of any of my repositories; instead I maintain the master copies on my own machines and treat the Github version as a convenient publicly visible version (one that presents, more or less, what I want people to be using). If Github disappeared tomorrow, I could move the public version to another place (such as Gitlab or Bitbucket), or perhaps finally get around to setting up my own Git publishing system.

Well, except for the bit where most or all of my public projects currently list their Github URL in the README and so on, and I have places (such as my Firefox addon's page) that explicitly list the Github URLs. All of those would have to be updated, which starts to point out the problem; those updates would have to propagate through to any users of my software somehow. The reality is that I've been sort of lazy in my README links and so on; they tend to point only to Github, not to Github plus anywhere else. What they should really do is point to Github plus some page that I run (and perhaps additional public Git repo URLs if I establish them on Gitlab or wherever).

There's some additional things that I'd lose, too. To start with, any issues that people have filed and pull requests that people have made (although I think that one can get copies of those, and perhaps I should). I'd also lose knowledge of people's forks of my Github repos and the ability to look at any changes that they may have made to them, changes that either show me popular modifications or things that perhaps I should adopt.

All of these things make Github sticky in a soft way. It's not that you can't extract yourself from Github or maintain a presence apart from it; it's that Github has made its embrace inviting and easy to take advantage of. It's very easy to slide into Github tacitly being your open source presence, where people go to find you and your stuff. If I wanted to change this (which I currently don't), I'm honestly not sure how I'd make it clear on my Github presence that people should now look elsewhere.

I don't regret having drifted into using Github this way, because to be honest I probably wouldn't have public repositories and a central point for them without Github or some equivalent. At the same time I'm aware that I drift into bad habits because they're easy and it's possible that Github is one such bad habit. Am I going to go to the effort of changing this? Certainly not right away (especially to my own infrastructure). Probably, like many people who have their code on Github, I'm going to wait and see and above all hope that I don't actually have to do anything.

(I'm also not convinced that there is any truly safe option for having other people host the public side of my repositories. Sourceforge serves as a cautionary example of what can happen to such places, and it's not like Gitlab, Bitbucket, and so on are obviously safer than Github is or was; they're just not (currently) owned by Microsoft. The money to pay for all of the web servers and disk space and so on has to come from somewhere, and I'm probably going to be a freeloader on any of them.)

Some notes on Go's runtime.KeepAlive() function

I was recently reading go101's "Type-Unsafe Pointers" (via) and ran across a usage of an interesting new runtime package function, runtime.KeepAlive(). I was initially puzzled by how it was used, and then me being me I had to poke into how it worked.

What runtime.KeepAlive() does is that it keeps a variable 'alive', which means that it (and what it refers to) will not be garbage collected and any finalizers it has won't be run. The documentation has an example of its use. My initial confusion was why the use of runtime.KeepAlive() was so late in the code; I had sort of expected it to be used early, like finalizers are set, but then I realized what it is really doing. In short, runtime.KeepAlive() is using the variable. A variable is obviously alive right up to the end of its last use, so if you use a variable late, Go must keep it alive all the way there.

At one level, there's nothing magical about runtime.KeepAlive; any use of the variable would do to keep it alive. At another level there is an important bit of magic about runtime.KeepAlive, which is that Go guarantees that this use of your variable will not be cleverly optimized away because the compiler can see that nothing actually really depends on your 'use'. There are various other ways of using a variable, but even reasonably clever ones are vulnerable to compiler optimization and aggressively clever ones have the downside that they may accidentally defeat Go's reasonably clever escape analysis, forcing what would otherwise be a local stack variable to be allocated on the heap instead.

The other special magic trick in runtime.KeepAlive() is in how it's implemented, which is that it doesn't actually do anything. In particular, it doesn't make a function call. Instead, much like unsafe.Pointer, it's a compiler builtin, set up in ssa.go. When your code uses runtime.KeepAlive(), the Go compiler just sets up a OpKeepAlive SSA thing and then the rest of the compiler knows that this is a use of the variable and keeps it alive through to that point.

(Reading this ssa.go initialization function was interesting. Unsurprisingly, it turns out that there are a number of nominal package function calls that are mapped directly to instructions that will be placed inline in your code, like math.Sqrt. Some of these are platform-dependent, including a bunch of math.bits.)

That runtime.KeepAlive is special magic has one direct consequence, which is that you can't take its address. If you try, Go will report:

./tst.go:20:22: cannot take the address of runtime.KeepAlive

I don't know if Go will too-cleverly optimize away a function that only exists to call runtime.KeepAlive, but hopefully you're never going to need to call runtime.KeepAlive indirectly.

PS: Although it's tempting to say that one should never need to call runtime.KeepAlive on a stack allocated local variable (including arguments) because the stack isn't cleaned up until the function returns, I think that this is a dangerous assumption. The compiler could be sufficiently clever to either reuse a stack slot for two different variables with non-overlapping lifetimes or simply tell garbage collection that it's done with something (for example by overwriting the pointer to the object with nil).

Bad versions of packages in the context of minimal version selection

Recently, both Sam Boyer and Matt Farina have made the point that Go's proposed package versioning lacks an explicit way for packages to declare known version incompatibilities with other packages. Suppose that you have a package A and it uses package X, initially at v1.5.0. The package X people release v1.6.0, which is fine, and then v1.7.0, where they introduce an API behavior change that is incompatible with how your package uses the API (Matt Farina's post has a real world example of this). By the strict rules of semantic versioning this is a no-no, but in real life it happens for all sorts of reasons. People would like the ability to have their own package say 'I'm not compatible with v1.7.0 (and later versions)', which Russ Cox's proposal doesn't provide.

The first thing to note is that in a minimal version selection environment, this incompatibility doesn't even come up as long as you're only building the package or something using the package that has no other direct or indirect dependencies on package X. If you're only using package A, package A says it wants X@v1.5.0 and that's what MVS picks. MVS will never advance to the incompatible version v1.7.0 on its own; it must be forced to do so. Even if you're also using package B and B requires X@v1.6.0, you're still okay; MVS will advance the version of X but only to v1.6.0, the new minimal version.

(This willingness to advance the version at all is a pragmatic tradeoff. We can't be absolutely sure that v1.6.0 is really API compatible with A's required X@v1.5.0, but requiring everyone to use exactly the same version of a package is a non-starter in practice. In order to make MVS useful at all, we have to hope that advancing the version here is safe enough (by default, and if we lack other information).)

So this problem with incompatible package versions only comes up in MVS if you also have another package B that explicitly requires X@v1.7.0. The important thing here is that this version incompatibility is not a solvable situation. We cannot build a system that works; package A doesn't work with v1.7.0 while package B only works with v1.7.0 and we need both. The only question is whether MVS or an MVS-like algorithm will actually tell us about this problem, aborting the build, or whether it will build a system that doesn't work (if we're lucky the system will fail our tests).

To me, this changes how critical the problem is to address. Failure to build a working system where it's possible would be one thing, but we don't have that; instead we merely have the question of whether you're going to get told up front that what you want isn't possible.

The corollary to this is that when package A publishes information that it's incompatible with X version v1.7.0, it's doing so almost entirely as a service for other people, not something it needs for itself. Since A's manifest only requires X@v1.5.0, MVS will generally use v1.5.0 when building A alone (let's assume that none of A's other dependencies also use X and will someday advance to requiring X@v1.7.0). It's only when A gets bundled together with B that problems happen, and so this is mostly when A's information about version incompatibility is useful. Should this information be published in a machine readable form? Well, I think it would be nice, but it depends on what else we have to give up for it.

(The developers of A may want to leave themselves a note about the situation in their version manifest, of course, just so that no developer accidentally tries advancing X's version and then gets surprised by the results.)

PS: There is also an argument that such incompatible version blocks should only be advisory warnings or the like. As the person building the overall system, you may actually know that the end result will work anyway; perhaps you've taken steps to compensate for the API incompatibility in your own code. Since the failure is an overall system failure, package A can't necessarily be absolutely sure about things.

(Things might be easier to implement as advisory warnings. One approach would be to generate the MVS versions as usual, then check to see if anyone declared an incompatibility with the concrete versions chosen. Resolving the situation, if it's even possible, would be up to you.)

'Minimal version selection' accepts that semantic versioning is fallible

Go has been quietly wrestling with package versioning for a long time. Recently, Russ Cox brought forward a proposal for package versioning; one of the novel things about it is what he calls 'minimal version selection', which I believe has been somewhat controversial.

In package management and versioning, the problem of version selection is the problem of what version of package dependencies you'll use. If your package depends on another package A, and you say your minimum version of A is 1.1.0, and package A is available in 1.0.0, 1.1.0, 1.1.5, 1.2.0, and 2.0.0, version selection is picking one of those versions. Most package systems will pick the highest version available within some set of semantic versioning constraints; generally this means either 1.1.5 or 1.2.0 (but not 2.0.0, because the major version change is assumed to mean API incompatibilities exist). In MVS, you short-circuit all of this by picking the minimum version allowed; here, you would pick 1.1.0.

People have had various reactions to MVS, but as a grumpy sysadmin my reaction is positive, for a simple reason. As I see it, MVS is a tacit acceptance that semantic versioning is not perfect and fails often enough that we can't blindly rely on it. Why do I say this? Well, that's straightforward. The original version number (our minimum requirement) is the best information we have about what version the package will definitely work with. Any scheme that advances the version number is relying on that new version to be sufficiently compatible with the original version that it can be substituted for it; in other words, it's counting on people to have completely reliably followed semantic versioning.

The reality of life is that this doesn't happen all of the time. Sometimes mistakes are made; sometimes people have a different understanding of what semantic versioning means because semantic versioning is ultimately a social thing, not a technical one. In an environment where semver is not infallible (ie, in the real world), MVS is our best option to reliably select package versions with the highest likelihood of working.

(Some package management systems arrange to also record one or more 'known to work' package version sets. I happen to think that MVS is more straightforward than such two-sided schemes for various reasons, including practical experience with some Rust stuff.)

I understand that MVS is not very aesthetic. People really want semver to work and to be able to transparently take advantage of it working (and I agree that it would be great if it did work). But as a grumpy sysadmin, I have seen a non-zero amount of semver not working in these situations, and I would rather have things that I can build reliably even if they are not using all of the latest sexy bits.

Sorting out some of my current views on operator overloading in general

Operator overloading is a somewhat controversial topic in programming language design and programming language comparisons. To somewhat stereotype both sides, one side thinks that it's too often abused to create sharp-edged surprises where familiar operators do completely surprising things (such as << in C++ IO). The other side thinks that it's a tool that can be used to create powerful advantages when done well, and that its potential abuses shouldn't cause us to throw it out entirely.

In general, I think that operator overloading can be used for at least three things:

1. implementing the familiar arithmetic operations on additional types of numbers or very strongly number-like things, where the new implementations respect the traditional arithmetic properties of the operators; for example + and * are commutative.

2. implementing these operations on things which already use these operators in their written notation, even if how the operators are used doesn't (fully) preserve their usual principles. Matrix multiplication is not commutative, for example, but I don't think many people would argue against using * for it in a programming language.

3. using these operators simply for convenient, compact notation in ways that have nothing to do with arithmetic, mathematical notation, or their customary uses in written form for the type of thing you're dealing with.

I don't think anyone disagrees with the use of operator overloading for the first case. I suspect that there is some but not much disagreement over the second case. It's the third case that I think people are likely to be strongly divided over, because it's by far the most confusing one. As an outside reader of the code, even once you know the types of objects involved, you don't know anything about what's actually happening; you have to read the definition of what that type does with that operator. This is the 'say what?' moment of << in C++ IO and % with Python strings.

Languages are partly a cultural thing, not purely a technical one, and operator overloading (in its various sorts) can be a better or a worse fit for different languages. Operator overloading probably would clash badly with Go's culture, for example, even if you could find a good way to add it to the language (and I'm not sure you could without transforming Go into something relatively different).

(Designing operator overloading into your language pushes its culture in one direction but doesn't necessarily dictate where you wind up in the end. And there are design decisions that you can make here that will influence the culture, for example requiring people to define all of the arithmetic operators if they define any of them.)

Since I'm a strong believer in both the pragmatic effects and aesthetic power of syntax, I believe that even operator overloading purely to create convenient notation for something can be a good use of operator overloading in the right circumstances and given the right language culture. Generally the right circumstances are going to be when the operator you're overloading has some link to what the operation is doing. I admit that I'm biased here, because I've used the third sort of operator overloading from time to time in Python and I think it made my code easier to read, at least for me (and it certainly made it more compact).

(For example, I once implemented '-' for objects that were collections of statistics, most (but not all) of them time-dependent. Subtracting one object from another gave you an object that had the delta from one to the other, which I then processed to print per-interval statistics.)

In thinking about this now, one thing that strikes me is that an advantage of operators over function calls is that operators tend to be written with whitespace, whereas function calls often run everything together in a hard to read blur. We know that whitespace helps readability, so if we're going to lean heavily on function calls in a language (including in the form of method calls), perhaps we should explore ways of adding whitespace to them. But I'm not sure whitespace alone is quite enough, since operators are also distinct from letters.

(I believe this is where a number of functional languages poke their heads up.)