Go goroutines as a way to capture and hold state
The traditional annoyance when writing lexers is that lexers have internal state (at least their position in the stream of text), but wind up returning tokens to the parser at basically random points in their execution. This means holding the state somewhere and writing the typical start/stop style of code that you find at the bottom of a pile of subroutine calls; your 'get next token' entry point gets called, you run around a bunch of code, you save all your state, and you return the token. Manual state saving and this stuttering style of code execution doesn't lend itself to clear logic.
Some languages have ways around this structure. In languages with generators, your lexer can be a generator that yields tokens. In lazy evaluation languages your lexer turns into a stream transformation from raw text to tokens (and the runtime keeps this memory and execution efficient, only turning the crank when it needs the next token).
In Rob Pike's presentation on lexing in Go, he puts the lexer
code itself into its own little goroutine. It produces tokens by
sending them to a channel; your parser (running separately) obtains
tokens by reading the channel. There are two ways I could put what
Rob Pike's done here. The first is to say that you can use
goroutines to create generators, with a channel send and receive
taking the place of a
yield operation. The second is that
goroutines can be used to capture and hold state. Just as with
ordinary threads, goroutines turn
asynchronous code with explicitly captured state into synchronous
code with implicitly captured state and thus simplify code.
(I suppose another way of putting it is that goroutines can be used for coroutines, although this feels kind of obvious to say.)
I suspect that this use for goroutines is not new for many people (and it's certainly implicit in Rob Pike's presentation), but I'm the kind of person who sometimes only catches on to things slowly. I've read so much about goroutines for concurrency and parallelism that the nature of what Rob Pike (and even I) were doing here didn't really sink in until now.
(I think it's possible to go too far overboard here; not everything needs to be a coroutine or works best that way. When I started with my project I thought I would have a whole pipeline of goroutines; in the end it turned out that having none was the right choice.)