Wandering Thoughts archives

Some thoughts on Go compiler directives being in source comments

Recently, I've been reading some commotion about how Go compiler directives being in source code comments is, well, not the 'elegant design' that Go's creators may feel it is. As it happens I have some sympathies for Go here, so let's talk about what I see as the issues involved.

First, let's differentiate between what I'll arbitrarily call 'broad' and 'narrow' compiler directives. In a nutshell, what I'm calling a broad compiler directive is something that changes the meaning of the source code such that every compiler implementation must handle it. In C, #include and #define are broad directives. Broad directives are effectively part of the language and as such I feel that they deserve first class support as an explicit element in language syntax.

(Broad directives don't have to use a new language syntax element. Python's 'from __future__ import ...' is such a broad directive, but it uses a standard language element.)

By contrast, narrow directives only apply to a specific compiler or tool. Since they're only for a specific program they should be namespaced, ie you need some way of saying 'this uninterpreted blob of text is only for <X>' so that other compilers can ignore it. This requires either a specific element of language syntax to say 'this following text is only for <X>' or hijacking a portion of some existing syntax where you can add arbitrary namespaced text. The easiest existing syntax to hijack is comments.

Since narrow directives do not change the language itself (at least in theory), it seems at least a bit odd to give them an explicit syntax element. In effect you're creating another escape hatch for language-meaningless text that sits alongside comments; one is sort of for people (although it may be interpreted by tools, for example for documentation) and one is a slightly structured one for tools.

(If a narrow directive changes the semantics of the code being compiled, it's actually changing the language the compiler is dealing with from 'language <X>' to 'something similar to <X> but not quite it'. Problems often ensue here in the long run.)

As far as I know, all of the existing Go compiler directives are narrow directives. They're either used by specific non-compiler tools or they're internal directives for one specific Go compiler (admittedly the main 'go' compiler). As far as I'm concerned this makes them pretty much fair game to be implemented without a specific element of language syntax. Other people may disagree and feel that even narrow directives should have some sort of specific language syntax support.

PS: There may well be standard terminology in the programming language community for what I'm calling broad versus narrow directives here.

(This elaborates on some tweets I made, because Twitter forces condensed and sometimes opaque writing.)

Sidebar: The problem with non-namespaced narrow directives

If you don't namespace your narrow directives you wind up with the C #pragma problem, which is 'what do you do when you encounter a #pragma that you don't recognize?'. If you do error out, you cause problems for people who are using you to compile source code with #pragmas for some other compiler. If you don't error out, you cause problems for people who've accidentally misspelled one of your #pragmas and are now having it be more or less silently ignored.

(You can try to know about the #pragmas of all other compilers, but in practice you're never going to know absolutely all of them.)

A Bash test limitation and the brute force way around it

Suppose that you are writing a Bash script (specifically Bash) and that you want to get a number from a command that may fail and might also return output '0' (which is a bad value here). No problem, you say, you can write this like so:

sz=$(zpool list -Hp -o size $pool)
if [ $? -ne 0 -o "$sz" -eq 0 ]; then
   echo failure
   ....
fi

Because you are a smart person, you test that this does the right thing when it fails. Lo and behold:

$ ./scrpt badarg
./scrpt: line 3: [: : integer expression expected

At one level, this is a 'well of course'; -eq specifically requires numbers on both sides, and when the command fails it does not output a number (in fact $sz winds up empty). At another level it is very annoying, because what we want here is the common short-circuiting logical operators.

The reason we're not getting the behavior we want is that test (in the built in form in Bash) is parsing and validating its entire set of arguments before it starts determining the boolean values of the overall expression. This is not necessarily a bad idea (and test has a bunch of smart argument processing), but it is inconvenient.

(Note that Bash doesn't claim that test's -a and -o operators are short-circuiting operators. In fact the idea is relatively meaningless in the context of test, since there's relatively little to short-circuit. A quick test suggests that at least some versions of Bash check every condition, eg stat() files, even when they could skip some.)

My brute force way around this was:

if [ $? -ne 0 -o -z "$sz" ] || [ "$sz" -eq 0 ]; then
   ....
fi

After all, [ is sort of just another program, so it's perfectly valid to chain [ invocations together with the shell's actual short circuiting logical operators. This way the second [ doesn't even get run if $sz looks bad, so it can't complain about 'integer expression expected'.

(This may not be the right way to do it. I just felt like using brute force at the time.)

PS: Given that Bash emits this error message whether you like it or not, it would be nice if it had a test operator for 'this thing is actually a number'. My current check here is a bit of a hack, as it assumes zpool emits either a number or nothing.

(Updated: minor wording clarification due to reddit, because they're right, 'return 0' is the wrong way to phrase that; I knew what I meant but I can't expect other people to.)

Exceptions as aggregators of error handling

As is becoming traditional, I'll start with the tweets:

@jaqx0r: People complaining about "if err != nil" density in #golang don't check errors in other languages. Discuss.

@thatcks: In exception-based languages, you often get to aggregate a bunch of similar/same error handling together in one code bit.

As usual on Twitter, this is drastically crunched down to fit into 140 characters and thus not necessarily all that clear. To make it clear, I'll start with a really simplistic example of error checking in a routine in Go:

func (c *MyConn) reply(code int, msg string) error {
   emsg, e := utils.encode(msg)
   if e != nil {
      return e
   }
   e = c.writecode(code)
   if e != nil {
      return e
   }
   e = c.writemsg(emsg)
   if e != nil {
      return e
   }
   e = c.logger.logreply(code, msg)
   return e
}

If it runs into an error, all that this code does is abort further processing and returns the error to the caller. There is a slightly less verbose way to write this but regardless of how you write it, you have to explicitly check errors each time even though you're handling them all the same way.

In an exception-based language like Python we can write the code so that all of these 'if error, abort' checks are aggregated together. In a slightly more elaborate example we might write something like:

def reply(self, code, msg):
   try:
      emsg = utils.encode(msg)
      self.writecode(code)
      self.writemsg(emsg)
      self.logger.logreply(code, msg)
   except (A, B, C) as exp:
      ... cleanup ...
      raise MyException(exp)

Since the action taken on errors is the same for all cases, we've been able to aggregate it together into a single exception handling block with a single set of code. This clearly results in less code here.

The more that your code wants to do basically the same thing on errors (perhaps with some variations in messages or the like), the more you will benefit from error aggregation through exceptions and the more onerous and annoying it is to repeat yourself every single time. Conversely, the more different your error handling is, the less you benefit; in fact in some environments the exception-based version can wind up more verbose. My perception is that often you have essentially the same handling of a lot of errors and so the aggregation offered by exceptions will wind up saving you code and repetition.

(There are tricks to avoid repeating yourself with explicit error checking but at least some of them wind up distorting the flow of your code, for example by inserting an intermediary function whose only real purpose is to aggregate the error handling actions into one place.)

So, I think that people who complain about the density of explicit error checks in Go (and similar languages) have a real case here. I don't think Go will ever solve it, but these people are not necessarily writing sloppy code in other languages, just efficiently structured code.

(This is basically what I wrote back in Exceptions as efficient programming.)

Sidebar: Some pragmatic arguments for the Go way

There are arguments about code readability here, of course. The Go version makes it explicit that the only action done on error each time is to return immediately from the function, while the Python version moves that to a separate block of code. It's not very distant in this contrived version, but you can imagine a more realistic one where there was a substantial amount of code that was being wrapped inside a single exception handler (thus moving the two much further apart). As a structural matter, the Go example also clearly handles all error cases that can arise from each function call, while in the Python one we've assumed that we know all of the exceptions that can be raised underneath us.

Go's tradeoffs here strike me as the correct ones given Go's overall goals, even if I'm not sure I like them. Still, I can't say that handling explicit errors has been any particular pain in my Go code to date.

My current views on Rust (the programming language)

Let's start with my tweet:

Modern languages like Rust and Haskell often give me the feeling that I am not smart enough and determined enough to program in them.

In the geeky programmer circles of the Internet that I fish for information in, Rust is very much the latest hot thing. It's generally presented as a better C/C++ that is (or should be) just as fast while being far more memory safe. All of this sounds very nice; who wouldn't want a C with better memory safety, after all?

My problem with Rust is that, as far as I can see from reading about it in passing, it is a fairly complicated language (due in large part to its determination to be a memory safe systems programming language). To read about Rust code is to be immersed in mutable and immutable variables, transferable references, borrows, macros, apparently complex types, and many other things. I'm probably smart enough to code in Rust if I was dedicated enough to learn all of the things that you need to know to write and read Rust code fluently. In practice I'm not.

One reason I'm not that dedicated is that it turns out I no longer have a use for 'like C but with better memory safety' (and a modern type and macro system, I think). And this is because I no longer write 'systems programs', broadly construed. If I just need a program, I can write it in Go or Python. If I need a program that goes fast or uses relatively little resources, I'll write it in Go. I write C programs these days only in two circumstances: if I need an absolutely minimal sized executable, or if I need a program that is right down to the metal of Unix system calls.

(One use of 'down to the metal' programming is benchmarking and testing, where I want to be sure exactly what the program is asking the kernel to do.)

But that's the sober rational reason. The emotional reason and my gut reaction to Rust is that it's too complicated to be appealing to me. More than that, it's complicated about things I no longer give a damn about, such as memory management without garbage collection. Rust is a language for smart people who care very deeply about certain issues, and I do not.

(I can completely see why Mozilla wants Rust and would not consider Go an adequate substitute. But I am never going to write a browser.)

And despite everything I've written here, there's still a part of me that wants to like Rust because damnit, it's cool and in theory it is right up my 'systems programming' alley and any number of smart people think it's a great thing. So I keep reading Rust news every so often (in much the same way I keep reading Haskell news). No one ever said I was a completely sensible person.

I don't commit changes in my working repos

I build a number of open source projects from source for various reasons of my own, using clones of the upstream master repos. With some of them, I make changes for my own reasons, often changes I will never be attempting to push upstream. Whenever I make changes this way, I do not commit my changes to the repos. This is the case whether the project uses Git or Mercurial.

The plain and simple truth is that if you're going to perpetually carry modest changes to an upstream project that changes frequently (and that you're going to re-pull frequently), not committing your changes is by far the easiest way to operate. If you have uncommitted changes, almost all of the time a simple 'git pull' or 'hg pull -u' will quietly get you up to date. If this doesn't work you can stash your changes in various ways, update to the latest version from a pristine setup, and then re-patch your changes. You spend a minimum amount of time interacting with the version control system, the actual checked in state of your repo exactly matches the master repo, and any time you want to know what you changed you can check 'git status' or 'hg diff' or so on. It's also really easy to temporarily or permanently set aside a particular change or set of them.

If you commit the changes, well, you have a bunch more work and your repo has now diverged from master. Every time you pull an upstream update, you must either merge or rebase your commits; even if the VCS will do this for you automatically this time, the merges or rebases are still there in repo history. If you do merges, any bisection is probably going to be more exciting. If you do rebases, your repo steadily accumulates a bunch of clutter in the corners. In either case monitoring the state of your changes is somewhat less straightforward, as is manipulating them.

It's my view that all of the stuff you have to do if you commit your changes is essentially make-work. There are cases where it's (potentially) useful, but those cases have been few and far between for me, and in the mean time I'd be doing a lot of VCS commands that I don't currently have to.

(I find the situation unfortunate but as far as I know neither Git nor Mercurial has extensions or features that would improve the situation. Git autorebase is sort of what I want, but not quite. Life is complicated by the fact that upstreams sometimes screw up their repos so that my local mirror repo diverges from their new reality, in which case I want to abandon my repo and just re-clone from upstream (and apply my changes again).)

(I mentioned this yesterday but I suspect it's a sufficiently odd way of working that I wanted to explain it directly.)

I don't find Github pull requests an easy way to submit patches

(This is a grumble. You've been warned.)

A lot of open source projects are hosted on Github these days and many of them prefer to get patch submissions in the form of Github pull requests. On the one hand I admire Github for making the process of pull requests relatively painless for the maintainer(s) in my brief experience. On the other hand, I don't find pull requests to be very attractive from the other side, as a (potential) submitter of changes, and I suspect that a lot of other potential change submitters are going to feel the same way.

In the moderately old days, submitting changes was really easy. You started with a current, checked out copy of the project's source repo and you carried your relevant changes on top of it as uncommitted changes (in a CVS or SVN based project you didn't have any choice about that and in general it was usually the easiest approach). When you wanted to submit your change, you ran '<whatever> diff' to get a unified diff relative to the latest master, dumped the output in a file, and emailed the result off to the project with an explanatory note. If you were running your modifications yourself, you could do this from your live working repo.

Creating a Github pull request is not so simple in practice. As far as I can tell, the workflow I'd want to use is something like the following:

• fork a copy of the project into my own Github account.
• 'git clone' that copy to a scratch repo locally. Create a new branch in the repo and switch to it.
• 'git diff' my working repo with the uncommitted change then use patch to apply the diff to the scratch repo.
• 'git commit' my change in the scratch repo (on my branch) and push it to my Github fork.
• go to Github and submit a pull request from my Github fork to the master.
• after the pull request is merged, delete my Github fork and my local scratch repo.

If I was being thorough, I should actually build the project out of this scratch repo and maybe do a test install (even though I've already built and installed from my working repo).

I have no idea what I'd do to keep clean commits if the master repo moves forward between my pull submission and when it gets merged. Maybe Github manages this transparently; maybe I need to update my scratch repo, 'git rebase' to get a new clean commit (maybe on a new branch), and push it back to Github to make a new pull request. It's a fun new frontier of extra work.

(None of this is difficult with the 'email a patch' approach. You 'git pull' to update your working repo, maybe fiddle with your change, then send another email with a newly generated 'git diff' if you think you need to or get asked for it.)

Note that there are some simplifications to this that could be done if I contributed a lot to specific projects, which is what I suspect Github pull requests are good for. But I rather feel that they're not so good for essentially one-off contributions from people, which is the category I'm most likely to fall into. So I'd sure like it if Github based projects still made it easy to send them patches by email (and mentioned this in any 'how to contribute' documentation they have). Unfortunately patches by email don't integrate very well with Github issues (of course), while Github pull requests work great there. I'm sure that this is a not insignificant factor pushing projects towards pull requests.

In Go, you need to always make sure that your goroutines will finish

Yesterday I described an approach to writing lexers in Go that pushed the actual lexing into a separate goroutine, so that it could run as straight-line code that simply consumed input and produced a stream of tokens (which were sent to a channel). Effectively we're using a goroutine to implement what would be a generator in some other languages. But because we're using goroutines and channels, there's something important we need to do: we need to make sure the lexer goroutine is run to completion, so that the goroutine will actually finish.

Right now you may be saying 'well of course the lexer will always be run to the end of the input, that's what the parser does'. But not so fast; what happens if the parser runs into a parse error because of a syntax error or the like? The natural thing to do in the parser is to immediately error out without looking at any further tokens from the lexer, which means that the actual lexer goroutine will stall as it sits there trying to send the next token into its communication channel, a channel that will never be read from because it's been abandoned by the parser.

The answer here is that the parser must do something to explicitly run the lexer to completion or otherwise cause it to exit, even if the tokens the lexer are producing will never be used. In some environments having the lexer process all of the remaining input is okay because it will always be small (and thus fast), but if you're lexing large bodies of text you'll want to arrange some sort of explicit termination signal via another channel or something.

This is an important way in which goroutines and channels aren't a perfect imitation of generators. In typical languages with generators, abandoning a generator results in it getting cleaned up via garbage collection; you can just walk away without doing anything special. In Go with goroutines, this isn't the case; you need to consider goroutine termination conditions and generally make sure it always happens.

You might think that this is a silly bug and of course anyone who uses goroutines like this will handle it as a matter of course. If so, I regret to inform you that I didn't come up with this realization on my own; instead Rob Pike taught it to me with his bugfix to Go's standard text/template module. If Rob Pike can initially overlook this issue in his own code in the standard library, anyone can.

Go goroutines as a way to capture and hold state

The traditional annoyance when writing lexers is that lexers have internal state (at least their position in the stream of text), but wind up returning tokens to the parser at basically random points in their execution. This means holding the state somewhere and writing the typical start/stop style of code that you find at the bottom of a pile of subroutine calls; your 'get next token' entry point gets called, you run around a bunch of code, you save all your state, and you return the token. Manual state saving and this stuttering style of code execution doesn't lend itself to clear logic.

Some languages have ways around this structure. In languages with generators, your lexer can be a generator that yields tokens. In lazy evaluation languages your lexer turns into a stream transformation from raw text to tokens (and the runtime keeps this memory and execution efficient, only turning the crank when it needs the next token).

In Rob Pike's presentation on lexing in Go, he puts the lexer code itself into its own little goroutine. It produces tokens by sending them to a channel; your parser (running separately) obtains tokens by reading the channel. There are two ways I could put what Rob Pike's done here. The first is to say that you can use goroutines to create generators, with a channel send and receive taking the place of a yield operation. The second is that goroutines can be used to capture and hold state. Just as with ordinary threads, goroutines turn asynchronous code with explicitly captured state into synchronous code with implicitly captured state and thus simplify code.

(I suppose another way of putting it is that goroutines can be used for coroutines, although this feels kind of obvious to say.)

I suspect that this use for goroutines is not new for many people (and it's certainly implicit in Rob Pike's presentation), but I'm the kind of person who sometimes only catches on to things slowly. I've read so much about goroutines for concurrency and parallelism that the nature of what Rob Pike (and even I) were doing here didn't really sink in until now.

(I think it's possible to go too far overboard here; not everything needs to be a coroutine or works best that way. When I started with my project I thought I would have a whole pipeline of goroutines; in the end it turned out that having none was the right choice.)

An interesting trick for handling line numbers in little languages

One of the moderately annoying issues you have to deal with when writing a lexer for a language is handling line numbers. Being able to report line numbers is important for passably good error messages, but actually doing this can be a pain in the rear end.

The usual straightforward way is to have your lexer keep track of the current line number and make it available to higher levels on demand. One problem this runs into is that the lexer's current position is not necessarily where the error actually is. The simple case is languages that don't allow multi-line constructs, but even here you can wind up off by a line in some situations.

A more sophisticated approach is to include the line number (and perhaps the position in the line) as part of what you return for every token. Both the parser and the lexer can then use this to report accurate positions for everything without any problems, although the lexer still has to keep counting lines and so on.

Somewhat recently I wound up writing a lexer in Go as part of a project, and I modeled it after Rob Pike's presentation on lexing in Go. Pike's lexer uses an interesting low-rent trick for handling line numbers, although it's one that's only suitable for use with a little language. Pike's lexer is given the entire source code to lex at once, so rather than explicitly tracking line numbers it just tracks the absolute character position in the source code (which it needs anyways) and includes this absolute character position as part of the tokens. If you turn out to need the line number, you call back to the lexer with the character position and the lexer counts how many newlines there are between the start of the source and the position.

Ever since I saw it this has struck me as a really clever approach if you can get away with it. Not only is it really easy to implement, but it's optimized for the common case of not needing the line number at all because you're parsing something without errors. Now that I've run into it, I'll probably reuse it in all future little language lexers.

Note that this isn't a general approach for several reasons. First, serious lexers are generally stream lexers that don't first read all of the source code into memory. Second, many languages routinely want line number information for things like profiling, debugging, and exception traces (and all of these uses are well after lexing has finished). That's why I say Pike's approach here is best for little languages, where it's common to read all of the source in at once for simplicity and you generally don't have those issues.

(If I was dealing with a 'bigger' language, I think that today I would take the approach of returning the line number as part of every token. It bulks up the returned token a bit but having the line number information directly in the token makes your life simpler in the long run, as I found out from the Go parser I wrote.)

In practice, programmers mostly understand complexity by superstition

A while back I said that a significant amount of programming is done by superstition. A corollary of that is that a great deal about programming is also understood primarily through superstition and mythology, as opposed to going to the actual deep technical detail and academic definitions of things. Today I want to point specifically to complexity.

You are a good, well educated programmer, so you know what 'constant time' or O(1) in the context of hash tables really means (for example). You probably know many of the effects that can distort hash table operations from being fast; there's that 'constant time' really only refers to 'relative to the number of entries', there's pathological cases that violate this (like extensive hash collisions), there's the practical importance of constant factors, there's the time taken by other necessary operations (which may not be constant time themselves), and then there's the low-level effects of real hardware (RAM fetches, TLB fills, L2 cache misses, and so on). This is an incomplete list, because everything is complex when you dig in.

Most programmers either don't know about this or don't think about it very much. If something is called 'constant time', they will generally expect it to be fast and to be consistently so under most conditions (certainly normal conditions). Similar things hold for other broad complexity classes, like linear or n log n. In fact you're probably doing well if a typical programmer can even remember what complexity class things are in. Generally this doesn't matter; as before, the important thing is the end result. If you're writing code with a significantly wrong understanding of something's practical behavior and thus time profile, you're probably going to notice.

(Not always, though; see Accidentally Quadratic, especially the rationale. You can choose to see this as a broad problem with the mismatch between superstition and actual code behavior here, in that people are throwing away performance without realizing it.)