Go goroutines as a way to capture and hold state

May 13, 2015

The traditional annoyance when writing lexers is that lexers have internal state (at least their position in the stream of text), but wind up returning tokens to the parser at basically random points in their execution. This means holding the state somewhere and writing the typical start/stop style of code that you find at the bottom of a pile of subroutine calls; your 'get next token' entry point gets called, you run around a bunch of code, you save all your state, and you return the token. Manual state saving and this stuttering style of code execution doesn't lend itself to clear logic.

Some languages have ways around this structure. In languages with generators, your lexer can be a generator that yields tokens. In lazy evaluation languages your lexer turns into a stream transformation from raw text to tokens (and the runtime keeps this memory and execution efficient, only turning the crank when it needs the next token).

In Rob Pike's presentation on lexing in Go, he puts the lexer code itself into its own little goroutine. It produces tokens by sending them to a channel; your parser (running separately) obtains tokens by reading the channel. There are two ways I could put what Rob Pike's done here. The first is to say that you can use goroutines to create generators, with a channel send and receive taking the place of a yield operation. The second is that goroutines can be used to capture and hold state. Just as with ordinary threads, goroutines turn asynchronous code with explicitly captured state into synchronous code with implicitly captured state and thus simplify code.

(I suppose another way of putting it is that goroutines can be used for coroutines, although this feels kind of obvious to say.)

I suspect that this use for goroutines is not new for many people (and it's certainly implicit in Rob Pike's presentation), but I'm the kind of person who sometimes only catches on to things slowly. I've read so much about goroutines for concurrency and parallelism that the nature of what Rob Pike (and even I) were doing here didn't really sink in until now.

(I think it's possible to go too far overboard here; not everything needs to be a coroutine or works best that way. When I started with my project I thought I would have a whole pipeline of goroutines; in the end it turned out that having none was the right choice.)

Written on 13 May 2015.
« It's time to stop coddling software that can't handle HTTPS URLs
In Go, you need to always make sure that your goroutines will finish »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed May 13 02:19:32 2015
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.