Wandering Thoughts archives

I need to use getopts sooner (and more often) in Bourne shell scripts

I have this flaw in my Bourne shell scripting; in shell scripts and shell scripts alone, I find it all too easy and tempting to reach for ad-hoc option processing when I add the first option to a script. Inevitably, handling the first option will look something like:

[...]
SOMEVAR=0
if [ "$1" = "-L" ]; then
   SOMEVAR=1; shift
fi
[...]

When a second option shows up, it sometimes winds up getting wedged awkwardly into the same sort of code. Maybe the options conflict so they'll never get used together in practice, or I'll tell myself that I'll always remember the order, or some equally bad excuse.

Doing this is a mistake, and I need to accept that. Rather than ever writing ad-hoc option processing code, I should be using getopts in my scripts right from the start even in tiny little scripts. Even using getopts for a single option is not that much more code over the simple version above, and it has better error handling. And of course, adding more options is much simpler and works much better if I'm using getopts already.

(Probably I should just memorize and master a standard getopts stanza so I can drop it into everything by reflex. But that's going to take time, and also not reflexively reaching for the quick solution for single options.)

I'm not entirely sure why I'm so reluctant in practice to use getopts, but for whatever reason it's not part of my shell landscape most of the time. This is silly, as getopts has existed for a long time and is a Single Unix Specification standard. As a practical matter it's long since been supported everywhere that I'm ever going to run a shell script. I need to get with the times and update my Bourne shell habits.

(This entry is brought to you by me updating some scripts today, trying to add a second option to them in a terrible way, and then telling myself that I should be doing this right and switching them over to using getopts. The resulting shell script code is both better and shorter.)

Sometimes, doing a bunch of programming can be the right answer

I like doing programming, and on top of that I can be a bit obsessive about it; for instance, if there are obvious features for a program to have, I want to add them even if they may not be strictly necessary. If left to myself, I would write plenty of programs for plenty of things and enjoy it a fair bit. The problem with this is that locally written programs are often an overhead and a long term burden, as xkcd has famously pointed out. Sure, it's nice to write code to solve our problems, but often that's not the right answer. I'm very conscious of this every time I'm tempted to write a program, and as a result I wind up sitting on my hands a lot.

We have a long standing local program to sort of deal with the pain of the Ubuntu package update process. It was a relatively minimal program, but it worked, and so for a long time I suppressed my urge to make it shinier and let it be. A couple of weeks ago I reached the limits of my tolerance after one too many extended end-of-day update runs. Never mind being sensible, I was going to change things because I couldn't take the current situation any more, and it didn't matter if this was objectively a bad use of my time.

I spent about a week working over most of the code, substantially growing the program in the process. The result is faster and more convenient, but it is also a lot more than that. The old update process had a lot of limitations; for example, it didn't really notice if updating one machine had problems, and if updating one machine hung there was no way to go see what was going on and maybe rescue the situation. The new program fixes these issues. This makes it substantially more complicated, but also much more useful (and less dangerous). There are a whole host of things we can do now because I got annoyed enough at the state of affairs to do something that wasn't strictly sensible (and then carry on further).

There's two lessons I draw from this. The first is that sometimes writing the code is the right answer after all. To put it one way, not everything that feels good is a bad idea. The second is that I actually should have done this years ago. This problem and its parade of irritations and workarounds is not new; we've been annoyed at the hassles of Ubuntu updates probably for as long as we've been running Ubuntu machines, and there's nothing in my code that couldn't have been done years ago. Had I done this coding much earlier, well, we could have been enjoying its improvements for quite some time by now.

(The meta-lesson here is that the earlier you make a change or an improvement with a likely long lifetime, the higher your payoff is. From the start we were pretty certain we'd be running Ubuntu machines for a long time to come, so clearly we could have forecast that a good update handling program had a big potential long-term payoff.)

I'm often an iterative and experimental programmer

I've been doing a significant amount of programming lately (for a good cause), and in the process I've been reminded that I'm often fundamentally an iterative and explorative programmer. By this I mean that I flail around a lot as I'm developing something.

In theory, the platonic ideal programmer plans ahead. They may not write more than they need now, but what they do write is considered and carefully structured. They think about the right data structures and code flow before they start typing (or at the latest as they start typing) and their code is reasonably solid from the start.

I can work this way when I understand the problem domain I'm tackling and how I want to approach it well enough to know in advance what's probably going to work and how I want to approach it. This works even (or especially) for what people sometimes consider relatively complicated cases, like recursive descent parsers. But put me in a situation where I don't know in advance what's going to work and roughly how it's all going to come out, and things get messy fast.

In a situation of uncertainty my approach is not to proceed cautiously and carefully, but instead to bang something ramshackle together to get experience and figure out if I can get an idea to work at all. My first pass code is often ugly and hacky and almost entirely the wrong structure (and often contains major duplication or badly split functionality). Bits and pieces of it evolve as I work away, with periodic cleanup passes that usually happen after I get some piece of functionality fully working and decide that now is a good time to deal with some of the mess. Entire approaches and user interfaces can be gutted and replaced with things that are clearly better ideas once I have a better understanding of the problem; entire features can sadly disappear because I realize that in retrospect they're bad ideas, or just unnecessary.

(It's very common for me to get something working and then immediately gut the working code to rebuild it in a much more sensible manner. I have an idea, I establish that the idea can actually be implemented, and then I slow down to figure out how the implementation should be structured and where bits and pieces of it actually belong.)

Eventually I'll wind up with a solid idea of what I want from my program (or code) and a solid understanding of what it takes to get there. This is the point where I feel I can actually write solid, good code. If I'm lucky I have the time to do so and it's not too difficult to transmogrify what remains of the first approach into this. If I'm unlucky, well, sometimes I've done a ground up rewrite and sometimes I've just waited for the next time I tackle a similar problem.

Whether or not to use cgo for Go packages, illustrated in a dilemma

Recently Dave Cheney wrote cgo is not Go where he very strongly advocates for not getting functionality into Go by using cgo to interface to existing C libraries. More directly, he writes:

I believe that when faced with reimplementing a large piece of C code in Go, programmers choose instead to use cgo to wrap the library, believing that it is a more tractable problem. I believe this is a false economy.

He goes on to run down a laundry list of problems that using cgo causes. All of them are real problems and they do bite, and I very much appreciate it when people write pure Go packages for what could otherwise easily be cgo-based interfaces to existing C libraries (eg, there's pure Go bindings for the X protocol).

But, well, as it happens I have a cgo using package, and today I want to talk about the problems of making it a pure Go package. My package is a Go interface to a Solaris/Illumos/etc facility for getting kernel statistics, called 'kstat'. Kstat is not officially provided as a documented kernel interface, just as a C library with a specific API. Of course the C library talks to the kernel under the hood, but this interface to the kernel is documented only in source code comments (and is not necessarily stable).

This could be reimplemented from scratch in pure Go (although using the unsafe package). Go can make ioctl() calls and it's actually not all that difficult to reverse engineer the kernel ioctls that libkstat is using (partly because the kernel interface is fairly close to the user level API). While in theory the kernel API might change, it seems unlikely to do so. A pure Go implementation wouldn't make this package any less OS-specific, but it would avoid the other problems with cgo.

On the other hand, a pure Go version has potential problems of its own. One is testing that it properly handles various things that the kernel does with changing kstats, some of which only occur in odd corner cases, especially since I would be writing it from necessarily imperfect understanding of how things work. Another is that this is more or less explicitly not supported by the platform; on Solaris and Illumos, libc and other OS-supplied shared libraries are the API, and the kernel interfaces is officially an implementation detail. If I have problems, I can pretty much count on all of the Illumos people telling me 'don't do that'.

(Go goes behind the back of this libc API already, which may hurt it at some point.)

So, I could rewrite to be a pure Go package. But I'd get a collection of new problems in the process and I'm not sure that the package would be as solid and as probably portable as it is now (as it is, for example, it probably works on Solaris 11 too). Is this a good tradeoff? I don't know. I feel it's a genuine dilemma.

PS: Over the course of writing this entry, I think I've talked myself into believing that a pure Go rewrite is at least reasonably possible and thus worth attempting (in theory). In practice I have no idea if I'll ever have the time and energy to do one, since the current situation does work for me.

Sidebar: The two pragmatic reasons that this is a cgo based package today

When I decided that I wanted to be able to get kstats in a Go program, I had neither the energy for a grounds up reverse engineering of the libkstat library (which I didn't even understand when I started out) nor the spare time a grounds up (re)write would have required. I was interested in solving a small problem, which meant not a lot of time and not very much frustration. Talking to libkstat in a Go program was only slightly more work than talking to it in a C program, and the Go program was nicer than a C program would have been.

(My other option was Perl, but I was even less interested in Perl than in C.)

I suspect a bunch of people are going to keep making this particular tradeoff and writing cgo-based packages. Some of them will throw the packages up on Github because, well, why not, that's the friendly way.

Memory-safe languages and reading very sensitive files

Here is an obvious question: does using modern memory safe languages like Go, Rust, and so on mean that the issues in what I learned from OpenSSH about reading very sensitive files are not all that important? After all, the fundamental problem in OpenSSH came from C's unsafe handling of memory; all of the things I learned just made it worse. As it happens, my view is that if you are writing security sensitive code you should still worry about these things in a memory safe language, because there are things that can still go wrong. So let's talk about them.

The big scary nightmare is a break in the safety of the runtime and thus the fundamental language guarantees, resulting in the language leaking (or overwriting) memory. Of course this is not supposed to happen, but language runtimes (and compilers) are written by people and so can have bugs. In fact we've had a whole series of runtime memory handling bugs in JIT'd language environments that caused serious security issues; there have been ones in JavaScript implementations, the JVM, and in the Flash interpreter. Modern compiled languages may be simpler than these JIT environments, but they have their own complexities where memory bugs may lurk; Go has a multi-threaded concurrent garbage collector, for example.

I'm not saying that there will be a runtime break in your favorite language. I'm just saying that it seems imprudent to base the entire security of your system on an assumption that there will never be one. Prudence suggests defense in depth, just in case.

The more likely version of the nightmare is a runtime break due to bugs in explicitly 'unsafe' code somewhere in your program's dependencies. Unsafe code explicitly can break language guarantees, and it can show up in any number of places and contexts. For example, the memory safety of calls into many C libraries depends on the programmer doing everything right (and on the C libraries themselves not having memory safety bugs). This doesn't need to happen in the code you're writing; instead it could be down in the dependency of a dependency, something that you may have no idea that you're using.

(A variant is that part of the standard library (or package set) either calls C or does something else marked as unsafe. Go code will call the C library to do some sorts of name resolution, for example.)

Finally, even if the runtime is perfectly memory safe it's possible for your code to accidentally leak data from valid objects. Take buffer handling, for example. High performance code often retains and recycles already allocated buffers rather than churning the runtime memory allocator with a stream of new buffer allocations, which opens you up to old buffer contents leaking because things were not completely reset when one was reused. Or maybe someone accidentally used a common, permanently allocated global temporary buffer somewhere, and with the right sequence of actions an attacker can scoop out sensitive data from it. There are all sorts of variants that are at least possible.

The good news is that the scope of problems is narrower at this level. Since what you're leaking is the previous state of a specific valid object, an attacker needs the previous state of that sort of object to be something damaging. You don't get the kind of arbitrary crossovers of data that you do with full memory leaks. Still, leaks in a central type of object (such as 'generic byte buffers') could be damaging enough.

The bad news is that your memory safe language cannot save you from this sort of leak, because it's fundamentally an algorithmic mistake. Your code and the language is doing exactly what you told it to, and in a safe way; it is just that what you told it to do is a bad idea.

(Use of unsafe code, C libraries, various sorts of object recycling, and so on is not necessarily obvious, either. With the best of intentions, packages and code that you use may well hide all of this from you in the name of 'it's an implementation detail' (and they may change it over time for the same reason). They're not wrong, either, it's just that because of how you're using their code it's become a security sensitive implementation detail.)

Things I learned from OpenSSH about reading very sensitive files

You may have heard that OpenSSH had an exploitable issue with some bad client code (which is actually two CVEs, CVE-2016-0777 and CVE-2016-0778). The issue was reported by Qualys Security, who released a fascinating and very detailed writeup on the issues. While the direct problem is basically the same as in Heartbleed, namely trusting an attacker-supplied length parameter and then sending back whatever happened to be sitting in memory, Qualys Security identified several issues that allowed private keys to leak through this issue despite OpenSSH's attempts to handle them securely. The specific issues are also fascinating in how they show just how hard it is to securely read sensitive files.

So here is what I have learned from OpenSSH about this:

• Do not use any sort of library level buffered IO. OpenSSH read private keys with stdio, which left copies of them behind in stdio buffers that were later free()'d. If your data is sensitive enough that you are going to explicitly erase it later, you must insure that it never passes through buffers that you do not control (and then you zero the buffers afterwards).

  (I suspect that I have Go code that fumbles this, although doing this in Go in general is at least a bit tricky.)

• Do not use any convenient form of memory or buffer handling that magically reallocates a new buffer and copies data when you ask a buffer to grow. The other way OpenSSH leaked private keys into memory was through realloc(), which may of course free the buffer you handed it and give you another one.

  There are all sorts of convenient auto-growing buffers and objects in all sorts of languages that are going to be vulnerable to this. You need to avoid them all. Once again, explicitly handling all storage yourself is required (and explicitly erasing all of it).

• Do not use general but over-powerful facilities in security sensitive code. OpenSSH apparently used a general 'load keypair' function that read the private key too even though it only needed the public key, which resulted in OpenSSH bringing into memory (and exposing) the private keys for all public keys it checked, not just the private key of the public key it was going to use with the server.

  It's easy to see how this happened, and strictly from a programming perspective it's the right answer. We have to load keypairs some times, so rather than have a 'load keypair' function and a 'load public key' function and so on, you just have a fully general 'load keypair' function and throw away the parts of results you don't need. But the result is that we loaded key data we didn't need into memory and then it leaked.

  (Admittedly the issue in OpenSSH is somewhat complex, since you can remove the .pub file and force it to decrypt your private key file to recover the public key (see the comments for this entry).)

(There is also an additional issue where overly clever C compilers can eliminate 'unnecessary' memset() operations that are supposed to erase the key data.)

The thing that scares me is that all of these are really easy mistakes to make, or rather really easy issues to overlook. I could have written code that did all three of them without even thinking about it, and I might have done so even if I was also writing code to carefully erase key data later.

Part of how they are so easy to overlook is that we are trained to think in terms of abstractions, not the details underneath them. Stdio gives you efficient IO (and maybe you remember that it involves buffering), realloc() just magically grows the allocated space (and oh yeah, sometimes it gives you a new area of memory, which means the old one got freed), and so on. And in fact all of these would have been harmless or mostly harmless in the absence of the core 'server can get a copy of random memory' bug.

A defense of C's null-terminated strings

In certain circles that I follow, it's lately been popular to frown on C's choice of simple null-terminated strings. Now, there are certainly any number of downsides to C's choice of how to represent strings (including that it makes any number of string operations not all that efficient, since they need to scan the string to determine its length), but as it happens I think that C made the right choice for the language it was (and is), and especially for the environment it was in. Namely, C is a relatively close to machine code language, especially in the beginning, and it was used on very small machines.

The other real option for representing strings is for them to have an explicit length field at a known spot (usually the start), plus of course their actual data. This is perfectly viable in C (people write libraries to do this), but putting it in the language as the official string representation adds complications, some of them immediately obvious and others somewhat more subtle. So, let's talk about those complications as a way of justifying C's choice.

The obvious issue is how big you make the length field and possibly how you encode it. A fixed size length field commits you to a maximum string size (if you make it 32 bits you're limited to 4GB strings, for example) and also potentially adds extra overhead for short strings (of which there are many). If you use a variable sized length encoding, you can somewhat avoid overhead and size limits at the cost of more complications when you need to look at the size or work out how much memory a particular sized string needs. C's null byte terminator imposes a fixed and essentially minimal size penalty regardless of the length of the string.

The indirect issue is that moving to a length-first string representation essentially forces you to create two new complex types in the language. To see why this is so, let's write some code in a Go-like language with inferred types:

string s = "abcdef";

void frob() {
   char *cp

   p := s+1    // refer to 'bcdef'
   cp = &s[2]  // refer to 'cdef'
   if (*cp == 'c') {
      cp++;
      [...]
   }
   [...]
}

Clearly s is a normal length plus data string; it takes up six bytes plus however large the length field is. In the simple case it is basically:

typedef string struct {
   strsize_t _len
   char      _data[]
}

But:

• what type is p, and what is its machine representation?
• is the assignment to cp valid or an error? Is cp a simple pointer, or does it have a more complicated representation?

One possible answer is that p is a char * and that a char * is a simple pointer. However, this is going to be very inconvenient because it means that any time you refer to something inside a string you lose the length of what you're dealing with and need to carry it around separately. You don't want to make p a string itself, because that would require allocating a variable amount of space and copying string data when p is created. What you need is what Go calls a slice type:

typedef strslice struct {
   strsize_t  _len
   char      *_dptr
}

This slice type carries with it the (adjusted) string length and a pointer to the data, instead of actually containing the data. From now on I'm going to assume that we have such a thing, because a language without it is pretty inconvenient (in effect you wind up making a slice type yourself, even if you just explicitly pass around a char * plus a length).

(In the traditional C style, it's your responsibility to manage memory lifetime so you don't free the underlying string as long as slices still point to some of it.)

You could make this slice type be what char * is, but then you'll want to create a new type that is a bare pointer to char, without a string length attached to it. And you probably want such a type, and you want to be able to get char * pointers from strings so that you can do operations like stepping through strings with maximum efficiency.

Our p type has a subtle inconvenience: because it fixes the string length, a p reference to a string does not resize if the underlying string has been extended (in the C tradition it doesn't notice if the string has been shrunk so the p is now past the end, either). Extending a string in place is a not uncommon C idiom, partly because it is more efficient than constantly allocating and freeing memory; it's a pity that our setup doesn't support that as it stands.

Since p is a multi-word type, we also have the problem of how to pass it around when making function calls. We either want our neo-C to have efficient struct passing or to always pass basic pointers to p's, with called functions doing explicit copying if they want to keep them. Oh, and clearly we're going to want a neo-C that has good support for copying multi-word types, as it would be terrible usability to have to write:

slice s2
memcpy(&s2, &p, sizeof(p))

(I already kind of assumed this when I wrote 'p := s+1'.)

The third level of issues is that this makes all of the str* routines relatively special magic, because they must reach inside this complex string type to manipulate the individual fields. We will probably also want to make a lot of the str* functions take slices (or pointers to slices) as arguments, which means either implicitly or explicitly creating slices from (pointers to) strings in various places in the code that people will write.

We could get around some of these problems by saying that the string type is actually what I've called the slice type here, ie all strings contain a pointer to the string data instead of directly embedding it. But then this adds the overhead of a pointer to all strings, even constant strings.

(It doesn't require separate memory allocations, since you can just put the string data immediately after the slice structure and allocate both at once.)

You can make all of this work and modern languages do (as do those string libraries for C). But I think it's clear that a neo-C with explicit-length strings would have been significantly more complicated from the get go than the C that we got, especially very early on in C's lifetime. Null terminated strings are the easiest approach and they require the least magic from the language and the runtime. Explicit length strings would also have been less memory efficient and more demanding on what were very small machines, ones where every byte might count.

(They are not just more demanding in data memory, they require more code to do things like maintain and manipulate string and slice lengths. And this code is invisible, unless you make programmers code it all explicitly by eg not having a slice type.)

I like Magit, the GNU Emacs package for Git

As part of slowly improving my GNU Emacs environment, I recently decided to experiment with magit, a very well regarded Emacs package for working with Git. I'm not attempting to adopt it as my primary interface to Git (I'm not a 'do everything from inside Emacs' person); instead, I have relatively specific situations where I was hoping magit would be the best way to work with Git. Although I haven't used it extensively yet, I can definitely say that Magit has been great in the areas where I was hoping that it would be.

For me, the big Magit feature and selling point is selectively adding things to the Git index (that is, staging them before committing them). Ordinary command line Git has 'git add -p', but this is a relatively awkward interface that I have had blow up in my face before. Magit offers a very powerful interface for selectively staging changes; you can easily stage not just individual chunks of the diff but even individual lines and collections of them. This is a very powerful way of straightening out a tangle of changes into logical changesets, even if they don't come out in separate chunks.

Part of why Magit makes this relatively easy is that it gives you a good live display of the staged and unstaged diffs as you are working away. This means you get immediate feedback if something looks wrong (or incomplete), and you can also reverse a decision right away. With 'git add -p' I found that I didn't really get this, since it's pretty much a linear process through all the chunks of diffs. In fact being non-linear is another advantage of Magit here; you can see the full set of diffs and skip around it to spot bits that you want to commit first.

Magit also has a relatively nice interface for just making commits. You get an Emacs buffer to edit the commit message (with standard general Emacs features like spell checking) and it shows you the staged diff that will be committed at the same time, which is handy. I can pretty much get all of this at the command line with multiple windows, but if I'm already in Emacs for editing files I often might as well commit from Emacs as well.

I have much less exposure to other Magit features, at least some of which I'm not terribly interested in (at least right now, I may change my mind later). How you invoke Magit commands initially struck me as a bit weird and confusing, but I'm getting used to it with time. People who regularly use Magit will likely find it completely natural.

So, in summary: Magit rocks for selectively staging changes, so much so that I think it's worth turning to GNU Emacs with Magit for this if you need it, even if you regularly edit in something else. If you regularly use GNU Emacs anyways I think the other parts of Magit are useful enough to learn at least the basics of looking at diffs and making straightforward commits.

(Although I spend a lot of time in vi, GNU Emacs remains my favorite editor for working on code in most languages for reasons beyond the scope of this entry.)

Some things that force Go to call the C library for name resolution on Linux

Traditionally on Unix systems there is no official standard for how to do various sorts of network name lookups, or rather the official standard is 'call the functions in the system C library'. There is generally no actual specification for how name lookup works at the level that would permit you to create an independent implementation (although there is generally documentation on how to configure it). This presents a problem for people who are creating non-C languages; they must either arrange to call the C library (through the C library's limited interfaces for this) or write their own versions that may not resolve things exactly like the C library does, so that you get inconsistent behavior between C programs and programs written in the other language.

Go kind of takes both approaches. As covered in the net package's documentation, Go can both call the C library routines (using cgo) and do its own name lookups in pure Go. Go normally tries to use the pure Go approach as much as possible because it's considered better (partly because cgo calls can be relatively expensive). In theory the pure Go approach should give you the same results as the cgo approach; in practice, the two can behave somewhat differently in some situations, sometimes because of oversights.

(Although the net package's documentation talks only about DNS related lookups, this also affects how at least net.LookupPort() works.)

Go normally attempts to be pretty hyper-intelligent about whether or not it can use its pure Go lookup functions. It makes this decision in part by reading through your /etc/resolv.conf and /etc/nsswitch.conf to see if you're using anything that it doesn't think it can handle. This raises the question of what things in either of these files can accidentally force Go to use cgo calls to the C library, instead of its own more efficient (and more consistent across systems) pure Go version. For /etc/resolv.conf, Go understands all of the common things but anything else will force it to cgo, including any mistakes you may have lurking in there. For /etc/nsswitch.conf, Go looks at the 'hosts' line and a few complications can be common on modern Linuxes:

• if your hosts includes myhostname, only lookups of names with dots in them can be done in pure Go. Because of an implementation quirk, this currently means that net.LookupPort() is forced to use the C library.

  (Some other things are also forced to use the C library, but arguably they should in this situation because they involve hostnames.)

• if your hosts includes mymachines, all lookups go to the C library. This is probably common on modern systemd-based Linux distributions.

If you're using Go programs and you don't use containers or don't need the magic functionality of mymachines, you may want to strip it out of your nsswitch.conf. If you're like me, you may even be surprised to find it there in the first place. You may not want myhostname either, especially if your host has IP aliases that are most definitely not included in what a name to IP lookup for its hostname should return.

Note that contrary to what you might think, net.LookupPort() (and things that call it to get ports, like net.ResolveTCPAddr()) does not look at the services line in /etc/nsswitch.conf, only the hosts line. And of course the pure Go port lookup only looks at /etc/services (and may not parse it exactly like the C library does). At the moment a missing or unreadable /etc/services seems not to force a fallback to the C library but instead uses a tiny built in default version.

(This probably doesn't affect anyone. I can't imagine that there are very many people who use NIS or otherwise do something funny for services lookups and not hosts lookups.)

You can see what sort of lookup is used for specific queries by setting the GODEBUG environment variable to a verbosity of 2 or more, ie 'GODEBUG=netdns=2'. The resulting report may look something like this:

go package net: dynamic selection of DNS resolver
go package net: hostLookupOrder() = cgo
go package net: hostLookupOrder(smtp.cs) = files,dns

This is on a Linux machine where I set hosts to include myhostname but not mymachines. The first hostLookupOrder() is for looking up the port number for smtp; here the presence of myhostname forced it to resort to cgo. A blank argument to hostLookupOrder() is used by several sorts of lookups, including net.LookupAddr() and net.LookupCNAME().

(This is of course an implementation detail and may change at some point.)

Goroutines, network IO streams, and the resulting synchronization problem

These days, I use my Go take on a netcat-like program for all of my casual network (TCP) service testing. One of the things that makes it convenient is that it supports TLS, so I can just do things like 'call tls imap.cs imaps' instead of having to dabble in yet another netcat-like program. However, this TLS support has an important and unfortunate limit right now, which is that it only supports protocols where you do TLS from the start. A certain number of protocols support a during-connection upgrade to TLS, generally via some form of what gets called STARTTLS; the one most relevant to me is SMTP.

I would like to support STARTTLS in call, but this exposes a structural issue. Call is a netcopy program and these are naturally asynchronous in that copying from standard input to the network is (and often should be) completely decoupled from copying from the network to standard output. Call implements this in the straightforward Go way by using two goroutines, one for each copy. This works quite well normally (and is the only real way to support arbitrary protocols), but the moment you add STARTTLS you have a problem.

The problem is that STARTTLS intrinsically requires the input and the output to synchronize (and really you need them to merge). The moment you send STARTTLS, the previously independent to-network and from-network streams must be synchronized and yoked together as TLS exchanges messages with the server at the other end of the connection. In a program based around poll() or select(), this is easy; you can trivially shift from asynchronous IO streams back to synchronous ones while you set up TLS, and then go back to asynchronous streams over the TLS connection. But in a goroutine based system, it's far from clear how to achieve this (especially efficiently), for two reasons.

The first problem is simply establishing synchronization between the sending and receiving sides in some reasonably elegant way. One approach is some sort of explicit variable for 'starting TLS' that is written by the sending side and checked at every IO by the receiving side. In theory another approach is to switch to a model with more goroutines, with one for every IO source and sink and a central manager of some sort. The explicit variable seems not very Go-y, while the central manager raises the issue of how to keep slow IO output to either the network or stdout from blocking the other direction.

The second problem is that mere synchronization is not enough, because there is no way to wake up on incoming network IO without consuming some of it. This matters because when we send our STARTTLS, we want to immediately switch network input handling from its normal mode to 'bring up TLS' mode, without reading any of the pending network input. If we cannot switch right away, we will wake up having .Read() some amount of the STARTTLS response, which must then be handled somehow.

(Hopefully all STARTTLS-enabled protocols have a server response in plaintext that comes before any TLS protocol messages. If the first thing we may get back from the network in response to our STARTTLS is part of the server TLS handshake, I'd have to somehow relay that initial network data to tls.Client() as (fake) network input. The potential headaches there are big.)

I don't have any answers here and I don't know what the best solution is; I just have my usual annoyance that Go forces asynchronous (network) IO into a purely goroutine based architecture.

(The clear pragmatic answer is to not even try to support STARTTLS in call and to leave that to other tools, at least until I wind up with a burning need for it. But that kind of annoys me.)