Wandering Thoughts archives

Some things for enumerated constants in Go

The standard way to do C style enums in Go is with iota and a typed set of constants. This looks like this:

type SmtpCmd int

const (
    EHLO SmtpCmd = iota
    MAILFROM
    RCPTTO
    ....
)

This set of constants is perfectly fine in one sense, but it has a non-obvious drawback (at least for someone coming from a Python world): you can't tell an uninitialized instance of SmtpCmd from an EHLO value. Like all Go types, SmtpCmd has a zero value and since it's ultimately an integer the zero value is, well, 0. iota starts numbering from 0 so EHLO is also 0. Having your zero value overlap with a valid value that you can see normally has the possibility of hiding errors like uninitialized fields or unset return values. Instead of being glaringly obvious it will just look like your code is somehow incorrectly determining that you have an EHLO.

The simple fix for this is to introduce a dummy first value:

const (
    unsetCmd SmtpCmd = iota
    EHLO
    ....
)

(We make unsetCmd start with a lower case letter so that it won't be visible from outside the package. This may or may not be a good idea depending on whether outside people generate instances of SmtpCmd or just look at them.)

The other thing you can do in Go with a typed set of constants is to pretty-print them for the purposes of the %v formatting operator. All this takes is a String() method function defined for your type. Doing a minimal version that annotates the string value with type information is simple but perhaps not completely useful:

func (v SmtpCmd) String() string {
    return fmt.Sprintf("<SmtpCmd %d>", v)
}

More sophisticated versions are possible if you have either a source of number to name mapping information or just build one yourself. You probably don't need it to be strikingly efficient because you probably won't be using %v on your enumerated types very often.

(One thing you can do in even a simple version is specifically annotate unsetCmd and perhaps anything that's out of range for your actual defined constants.)

Sidebar: A potential trick with String() methods

In many cases being able to use %v on your package's values is most useful when debugging test failures. In addition, printing useful information about them may involve crusty code that you don't really like sitting around in the main package source.

As far as I can tell, you can actually put the String() method definitions in your *_test.go files and have them work. The *_test.go packages are an internal part of your package when built for tests, but I believe that stuff defined in them is not part of your package's exported functionality and the crufty and internals specific pretty-printing code can live with the rest of your testing code.

(Of course this may cause a certain amount of confusion someday, when your package internal tests print out nice looking and informative %v values for things while in your outside code they just gets eg basic numbers. This is your call to make. I honestly don't know which side I come down on right now, although I get a half-pass because I'm not writing a package I expect anyone else to ever use.)

Why Java is a compiled language and Python is not

A commentator on Charles Marsh's article Why are there so many Pythons asked an interesting question:

[...] If both Java and Python produces Bytecodes which in-turn is run by VM's why Java is called as Compiled language and Python is called Interpreted language? [...]

One comment's answer was 'marketing', which in a sense is correct; one reason we call Java a compiled language is that that's what Sun called it from the start. Another comment noted that Java has an explicit compilation phase that is separate from having the JVM execute your Java program by interpreting the bytecodes. All of this points us towards what I feel is the real answer:

In Java, bytecode is a first class object. In Python it's an internal implementation detail.

You've always been able to find specific documentation on the JVM and its bytecodes; as far as I know they were released along side the first definitions of Java the language. JVM bytecode has never been marked 'for internal use only' and in fact it's explicitly been a stable, public representation of Java programs. For example you don't download the source code for Java applets into your browser, you download the JVM bytecode (and in general JVM bytecode is the common delivery method of Java code even on servers). And the JVM guarantees that this will work regardless of what sort of machine you have (and in theory guarantees to make it work securely even in the face of maliciously generated bytecode).

This public visibility, this treatment of bytecode as a real part of the language environment with its own documentation and guarantees and specifications, makes JVM bytecode a first class object in the overall Java ecosystem. In turn this makes it accurate to say that Java programs are (usually) compiled to JVM bytecode and then executed by an interpreter of that bytecode. This is especially so when the language's implementation makes these two explicitly separate steps with a whole collection of artifacts that are this compiled bytecode.

In Python there is nothing like this; CPython's use of bytecode is just an internal detail of the interpreter implementation. CPython bytecode is not part of Python semantics, it is not really documented, and you are not really supposed to generate your own. There's no guarantee of any cross-version compatibility, as CPython bytecode is expected to be consumed by the same interpreter that generated it. And so on. CPython bytecode is an interesting curiosity, not a first class part of (C)Python.

(It's technically accurate to say that CPython compiles Python source to bytecodes and then interprets these bytecodes. But this is a narrow technicality, not how people really see CPython working, and can be said about any number of languages that we normally think of as interpreted.)

The advantages of editors over database programs for modifying your data

Yesterday I mentioned that one advantages of using plain text files is that you could modify your data with an editor instead of having to use some sort of database program (whether a SQL shell, some shell scripting on top of that, or custom code you write). One reason this is an advantage is that someone else has already written your editor of choice and you already know how to use it, but another part of this is that editors are often genuinely better environments to modify your data in.

There are at least two aspects of this. The first is that any decent editor has not just undo but multi-level undo that crosses over file save boundaries. In most editors you can revert your entire set of modifications all the way up until you quit out of the editor while at the same time making your modifications visible to other programs (by saving the file or files). Actual database programs generally can't match this level of support for 'oh oops, better revert that' moments, partly by design.

(Most editors are implicitly built around being the sole modifier of the files you're working on. Databases are generally built around assuming multiple people are modifying things on an ongoing basis.)

The second is that editors make your changes fully visible and also show you the surrounding context (or at least a screen's worth). You literally see your data sitting there on the screen. This makes it much easier to notice undesirable changes and other accidents, especially large scale ones; if you mangled your data, you have a chance to see that. Did you change a field you didn't want to or accidentally delete half your data? You'll probably spot that. By contrast, working through a database access program can literally make your accidents invisible since you often don't see the post-change or post-addition records.

The two aspects combined are great for avoiding accidents and fixing them when they happen during modifications (provided that you notice at the time). This goes especially well with version control, not just because version control lets you revert things but also because it can explicitly show you what you changed. Databases, well, databases don't do diff very much without a lot of outside assistance.

Why I don't use relational databases in practice

I started out planning to write an entry I was going to title 'when I reach for a relational database'. The problem with this implicit question is that in practice the answer is 'when the system I'm building on basically insists on it'. Otherwise I basically don't use one, even when the problem more or less naturally fits into a relational world. On a purely logical basis, this is crazy; using a relational database for relational things is much easier than the alternatives and SQLite makes this trivial.

In thinking about it, I've come around to the idea that my problem with relational databases is that using a database makes my data relatively opaque and as a result complicates my life. The easy option for a relational database is an SQL based database, probably SQLite, and all of those store my data in unreadable and uneditable binary files. In order to get at my data I need to use the database server's tools (or write code). The direct consequence of this is that I can't inspect and go through my data by firing up less or grep or any number of other things and I can't (if I want to) put things under version control in any easy way.

The slightly more indirect consequence is that I need to write more tools. When my data is in plain text files I can modify my data with an editor or other simple Unix programs; add something, delete something, modify something, shuffle things around, it's all something that gets done with existing Unix programs and maybe some thin scripts. When my data is in a SQLite database I have to manipulate the data through SQLite in some way, either in more complicated shell scripts that drive the SQLite shell or through actual little utility programs that I write. In practice the latter is much safer and so what I'm likely to wind up with. Either way it's more work and more dedicated tools that I have to keep track of in various ways.

For me all of this adds friction to using a real relational database, enough friction that I'll endure some level of pain from hacking things together with plain text files instead. This is not completely rational, but evidently I put a high premium on being able to see things with less and edit them with vi.

(Note that some people have violent objections to relational databases, either in general or for uses where they aren't 'needed' in some sense. I'm not one of them; I don't have any strong feelings that they're inelegant or consume too many resources or whatever.)

How dynamic language code gets optimized

Yesterday I mentioned in passing that code that used Python dictionaries in predictable struct-like ways could probably be optimized just as similar uses of Python class instances is (in JIT environments like eg PyPy). This may raise some people's eyebrows; after all, aren't Python dictionaries too complex and flexible to get optimized?

It's my understanding that this is based on a misunderstanding of how modern dynamic languages are optimized. To simplify, the core of optimizing dynamic languages like Javascript, Python, and so on is the fact that most actual programs are not very dynamic. Instead in practice they're mostly static programs with mostly static types and so on. Optimizing a dynamic language program mostly consists of finding the static code that exists inside it and directly implementing that. In other words, dynamic languages are optimized by recognizing behavior. In theory it doesn't matter where this behavior happens or what language-level constructs it uses; all that matters is that you can reliably spot the behavior and optimize it.

(Sometimes this behavior is recognized dynamically, as the code executes, and sometimes it's recognized statically as the code is inspected. And of course there are schemes that do both.)

But there's still two reasons to care about what language level features and data structures get used for something, because all of this optimization is a heuristic. First, you care about how likely it is that your attempts to find behavior will work and this is related to how people actually use the language. If most people use classes as structures instead of dictionaries, then it's must more likely that your structure-recognition heuristics will succeed if you look at class usage. Time spent carefully tracking dictionary usage to spot structure usage patterns may generally be wasted time.

Second, even when you think you recognize a pattern of behavior you may turn out to be wrong; after behaving 'right' for a while the code may then turn around and do something that invalidates the pattern. This can waste some or all of your effort in optimizing for the pattern, so you also care about how often this happens with any particular sort of object. Again this will be related to how people use the language; if most people use dictionaries flexibly, you're much more likely to have false positives on temporary 'structure like' behavior with a particular dictionary and section of code that's invalidated later.

Lack of freedom in the language for how something is used certainly helps optimization, in part because it means that you don't even have to consider or worry about certain possibilities. Python's __slots__ is an example of this. But it's generally not an essential prerequisite.

The question of language longevity for new languages

Every so often I feel a temptation to rewrite DWiki (the engine behind this blog) in Go. While there are all sorts of reasons not to (so many that it's at best a passing whimsy), one concern that immediately surfaces is the question of Go's likely longevity. I'd like the blog to still be here in, say, ten years, and if the engine is written in Go that needs Go to be a viable language in ten years (and on whatever platform I want to host the blog on).

Of course this isn't just a concern for Go; it's a concern for any new language and there's a number of aspects to it. To start with there's the issue of the base language. There are lots of languages that have come and gone, or come and not really caught on very much so that they're still around but not really more than a relatively niche language (even though people often love them very much and are very passionate about them). Even when a language is still reasonably popular there's the question of whether it's popular enough to be well supported on anything besides the leading few OS platforms.

(Of course the leading few OS platforms are exactly the ones that I'm most likely to be using. But that's not always the case; this blog is currently hosted on FreeBSD, for example, not Linux, and until recently it was on a relatively old FreeBSD.)

But you'd really like more than just the base language to still be around, because these days the base language is an increasingly small part of the big picture of packages and libraries and modules that you can use. We also want a healthy ecology of addons for the language, so that if you need support for, say, a new protocol or a new database binding or whatever you probably don't have to write it yourself. The less you have to do to evolve your program the more likely it is to evolve.

Finally there's a personal question: will the language catch on with you so that you'll still be working with it in ten years? Speaking from my own experience I can say that it's no fun to be stuck with a program in a language that you've basically walked away from, even if the language and its ecology is perfectly healthy.

Of course, all of this is much easier if you're writing things that you know will be superseded and replaced before they get anywhere near ten years old. Alternately you could be writing an implementation of a standard so that you could easily swap it out for something written in another language. In this sense a dynamically rendered blog with a custom wikitext dialect is kind of a worst case.

(For Go specifically I think it's pretty likely to be around and fully viable in ten years, although I have less of a sense of my own interest in programming in it. Of course ten years can be long time in computing and some other language could take over from it. I suspect that Rust would like to, for example.)

Thinking about how to split logging up in multiple categories et al

I've used programs that do logging (both well and badly) and I've also written programs that did logging (also both reasonably well and badly) and the whole experience has given me some views on how I like logging split up to make it more controllable.

It's tempting to say that controlling logging is only for exceptional cases, like debugging programs. This is not quite true. Certainly this is the dominant case, but there are times when people have different interests about what to log even in routine circumstances. For example, on this blog I log detailed information about conditional GETs for syndication feeds because I like tracking down why (or why not) feed fetchers succeed at this. However this information isn't necessarily of interest to someone else running a DWiki instance so it shouldn't be part of the always-on mandatory logging; you should be able to control it.

The basic breakdown of full featured logging in a large system is to give all messages both a category and a level. The category is generally going to be the subsystem that they involve, while the level is the general sort of information that they have (informational, warnings, progress information, debugging details, whatever). You should be able to control the two together and separately, to say that you want only progress reports from all systems or almost everything from only one system and all the way through.

My personal view is that this breakdown is not quite sufficient by itself and there are a bunch of cases where you'll also want a verbosity level. Even if verbosity could in theory be represented by adding more categories and levels, in practice it's much easier for people to crank up the verbosity (or crank it down) rather than try to do more complex manipulations of categories and levels. As part of making life easier on people, I'd also have a single option that means 'turn on all logging options and log all debugging information (and possibly everything)'; this gives people a simple big stick to hit a problem with when they're desperate.

If your language and programming environment doesn't already have a log system that makes at least the category plus level breakdown easy to do, I wouldn't worry about this for relatively small programs. It's only fairly large and complex programs with a lot of subsystems where you start to really want this sort of control.

Sidebar: the two purposes of specific control

There are two reasons to offer people specific control over logging. The first is what I mentioned: sometimes not all information is interesting to a particular setup. I may want information on syndication feed conditional GETs while you may want 'time taken' information for all requests. Control over logging allows the program to support both of us (and the person who doesn't care about either) without cluttering up logs with stuff that we don't want. This is log control for routine logs, stuff that you're going to use during normal program operation.

The second reason is that a big system can produce too much information at full logging flow when you're trying to troubleshoot it, so much that useful problem indicators are lost in the overall noise. Here categorization and levels are a way of cutting down on the log volume so that people can see the important things. This is log control for debugging messages.

(There is an overlap between these two categories. You might log all SQL queries that a system does and the time they take for routine metrics, even though this was originally designed for debugging purposes.)

Why I sometimes reject patches for my own software

I recently read Drew Crawford's Conduct unbecoming of a hacker (via), which argues that you should basically always accept other people's patches for your software unless they are clearly broken. Lest we bristle at this, he gives the example of Firefox and illustrates how many patches it accepts. On the whole I sympathize with this view, and I've even had some pragmatic experience with it; a patch to mxiostat that I wasn't very enthusiastic about initially has actually become something I use routinely. But despite this there are certain sorts of patches I will reject basically out of hand. Put simply they're patches that I think will make the program worse for me, no matter how much they might help the author of the patch (or other people).

This is selfish behavior on my part, but so far all of my public software is things that I'm ultimately developing for myself first. It's nice if other people use my programs too but I don't expect any of them to get popular enough that other people's usage is going to be my major motivation for maintaining and developing them. So my priorities come first and the furthest I'm willing to go is that I'll accept patches that don't get in the way of my usage.

(Drew Crawford's article has sort of convinced me that I should be more liberal about accepting patches in general; he makes a convincing case for 'accept now, bikeshed later'. So far this is mostly a theoretical issue for my stuff.)

By the way, this would obviously be different if I was developing things with the explicit goal of having them used by other people. In that case I should (and hopefully would) suck it up and put the patch in unless I had strong indications that it would make the program worse for a bunch of people instead of just me. Maybe someday I'll write something like that, but so far it's not the case.

A bit more about the various levels of IPC: whether or not they're necessary

A question you could ask about the levels of IPC that I outlined is if anything past basic process to process IPC is actually necessary (this is the level of TCP connections or Unix domain sockets). One answer to this is basically 'of course not'. All you really need is for programs to be able to talk to each other and then you can build whatever each particular system needs from there, possibly using common patterns like simple ASCII request/response protocols. A lot of software has gotten very far on this basis.

The other answer is that yes, in the end you do need all of those additional levels. To put it one way, very few people design introspectable message bus systems with standardized protocol encodings for fun. These systems get designed and built and adopted because they solve real problems; in the jargon, they are a design pattern. If you don't create them what you really get is a whole collection of ad-hoc and often partial versions that all of the various systems have reinvented on their own. For example, not having a standard protocol encoding does not free programs from needing to define a wire protocol; it just means that every program does it separately and differently and some number of them will do it badly.

(And in the modern world some number of them will make security mistakes or have buffer overruns and other flaws. A single standard system that everyone uses has the potential advantage of being carefully designed and built based on a lot of research and maybe even experience. Of course it can also be badly done, in which case everyone gets a badly done version.)

In this sense the additional levels of IPC really do wind up being necessary. It's just not the mathematical minimization sense of 'necessary' that people sometimes like to judge systems on.

The multiple levels of interprocess communication

In some quarters it's popular to say that things in computer programming go in cycles and history repeats itself. I'm a bit more optimistic so I like to view it as people repeatedly facing the same problems and solving them with today's tools. One of those things that keeps coming around is IPC aka interprocess communication. One way to look at IPC is to split it up into multiple levels that seem to reoccur again and again.

The first layer of IPC is simply connecting processes to each other. Since you can't get very far without this it tends to be well solved, although people keep inventing new ways of doing it with slightly different properties. Good local IPC mechanisms provide reliable authentication information about who is connecting to you and provide some way to transport credentials, access rights, or the equivalent.

(On modern Unix systems Unix domain sockets provide at least UID information and can be used to pass file descriptors around.)

The next IPC level is a common protocol format, which gets created when people get tired of making up data formats and (re)implementing encoders and decoders. Generally the high level protocol format will specify some sort of destination (in this sense, a method) plus a structured payload (which may or may not be self documenting to some degree). Sometimes you see things like sequencing and multiplexing added at this level.

(Modern examples include Google protobufs and JSON, but there are a very large number of protocol formats that people have used over the years. One early Unix example is the Sun RPC wire format they used for NFS and a number of related things.)

The next IPC level is some kind of a 'message bus', which generally exists to solve two related registration problems. In the addressing problem, programs want to talk to whatever is implementing service X (and to have a way of registering as handling X, and stopping handling X, and so on). In the broadcast problem, a program wants to broadcast information about X to any interested parties, whoever they may be. Once you have an active message bus it can also do additional things like starting a program when a request for service X comes in.

(Message bus services on Unix date at least as far back as the venerable rpcbind, which was originally invented by Sun for NFS-related addressing issues. There is a theme here.)

The final IPC level is introspection, where you can use the IPC mechanism to find out what is registered, what operations they support, and hopefully monitor traffic and other activity. Technically IPC introspection can appear without a message bus but I think this is uncommon. The power of the introspection varies tremendously from IPC system to IPC system.

You can invent ad-hoc mechanisms to solve all of these levels of problems, and people do. What drives people to 'standardize' them is a desire to avoid reinventing the wheel for each new program and each new level of the problem. Common protocol formats almost invariably come with libraries or codec generators (or both), message busses come with standard programs and libraries (instead of ad-hoc solutions like 'Unix domain sockets with specific names in directory Z'), and so on. Introspection and tools related to it solve a bunch of common problems and can often be used to enable ad-hoc interaction with services, eg through shell scripts.

(Once you have a reasonably sophisticated message bus with a lot of things going on through it, you really want some way of finding out what messages are flowing where so you can debug the total system.)