Wandering Thoughts

2024-03-08

A realization about shell pipeline steps on multi-core machines

Over on the Fediverse, I had a realization:

This is my face when I realize that on a big multi-core machine, I want to do 'sed ... | sed ... | sed ...' instead of the nominally more efficient 'sed -e ... -e ... -e ...' because sed is single-threaded and if I have several costly patterns, multiple seds will parallelize them across those multiple cores.

Even when doing on the fly shell pipelines, I've tended to reflexively use 'sed -e ... -e ...' when I had multiple separate sed transformations to do, instead of putting each transformation in its own 'sed' command. Similarly I sometimes try to cleverly merge multi-command things into one command, although usually I don't try too hard. In a world where you have enough cores (well, CPUs), this isn't necessarily the right thing to do. Most commands are single threaded and will use only one CPU, but every command in a pipeline can run on a different CPU. So splitting up a single giant 'sed' into several may reduce a single-core bottleneck and speed things up.

(Giving sed multiple expressions is especially single threaded because sed specifically promises that they're processed in order, and sometimes this matters.)

Whether this actually matters may vary a lot. In my case, it only made a trivial difference in the end, partly because only one of my sed patterns was CPU-intensive (but that pattern alone made sed use all the CPU it could get and made it the bottleneck in the entire pipeline). In some cases adding more commands may add more in overhead than it saves from parallelism. There are no universal answers.

One of my lessons learned from this is that if I'm on a machine with plenty of cores and doing a one-time thing, it probably isn't worth my while to carefully optimize how many processes are being run as I evolve the pipeline. I might as well jam more pipeline steps whenever and wherever they're convenient. If it's easy to move one step closer to the goal with one more pipeline step, do it. Even if it doesn't help, it probably won't hurt very much.

Another lesson learned is that I might want to look for single threaded choke points if I've got a long-running shell pipeline. These are generally relatively easy to spot; just run 'top' and look for what's using up all of one CPU (on Linux, this is 100% CPU time). Sometimes this will be as easy to split as 'sed' was, and other times I may need to be more creative (for example, if zcat is hitting CPU limits, maybe pigz can help a bit.

(If I have the fast disk space, possibly un-compressing the files in place in parallel will work. This comes up in system administration work more than you'd think, since we can want to search and process log files and they're often stored compressed.)

ShellPipelineStepsAndCPUs written at 22:27:42; Add Comment

2024-02-26

How to make your GNU Emacs commands 'relevant' for M-X

Today I learned about the M-X command (well, key binding) (via), which "[queries the] user for a command relevant to the current mode, and then execute it". In other words it's like M-x but it restricts what commands it offers to relevant ones. What is 'relevant' here? To quote the docstring:

[...] This includes commands that have been marked as being specially designed for the current major mode (and enabled minor modes), as well as commands bound in the active local key maps.

If you're someone like me who has written some Lisp commands to customize your experience in a major mode like MH-E, you might wonder how you mark your personal Lisp commands as 'specially designed' for the relevant major mode.

In modern Emacs, the answer is that this is an extended part of '(interactive ...)', the normal Lisp form you use to mark your Lisp functions as commands (things which will be offered in M-x and can be run interactively). As mentioned in the Emacs Lisp manual section Using interactive, 'interactive' takes additional arguments to label what modes your command is 'specially designed' for; more discussion is in Specifying Modes For Commands. The basic usage is, say, '(interactive "P" mh-folder-mode)'

If your commands already take arguments, life is simple and you can just put the modes on the end. But not all commands do (especially for quick little things you do for yourself). If you have just '(interactive)', the correct change is to make it '(interactive nil mh-folder-mode)'; a nil first argument is how you tell interactive that there is no argument.

(Don't make my initial mistake and assume that '(interactive "" mh-folder-mode)' will work. That produced a variety of undesirable results.)

Is it useful to do this, assuming you have personal commands that are truly specific to a given mode (as I do for commands that operate on MH messages and the MH folder display)? My views so far are a decided maybe in my environment.

First, you don't need to do this if your commands have keybindings in your major mode, because M-X (execute-extended-command-for-buffer) will already offer any commands that have keybindings. Second, my assortment of packages already gives me quite a lot of selection power to narrow in on likely commands in plain M-x, provided that I've named them sensibly. The combination of vertico, marginalia, and orderless let me search for commands by substrings, easily see a number of my options, and also see part of their descriptions. So if I know I want something to do with MH forwarding I can type 'M-x mh forw' and get, among other things, my function for forwarding in 'literal plaintext' format.

With that said, adding the mode to '(interactive)' isn't much work and it does sort of add some documentation about your intentions that your future self may find useful. And if you want a more minimal minibuffer completion experience, it may be more useful to have a good way to winnow down the selection. If you use M-X frequently and you have commands you want to be able to select in it in applicable modes without having them bound to keys, you really have no choice.

EmacsMetaXRelevantCommands written at 22:11:26; Add Comment

2024-02-24

The Go 'range over functions' proposal and user-written container types

In Go 1.22, the Go developers have made available a "range over function" experiment, as described in the Go Wiki's "Rangefunc Experiment". Recently I read a criticism of this, Richard Ulmer's Questioning Go's range-over-func Proposal (via). As I read Ulmer's article, it questions the utility of the range over func (proposed) feature based on the grounds that this isn't a significant enough improvement in standard library functions like strings.Split (which is given as an example in the "more motivation" section of the wiki article).

I'm not unsympathetic to this criticism, especially when it concerns standard library functionality. If the Go developers want to extend various parts of the standard library to support streaming their results instead of providing the results all at once, then there may well be better, lower-impact ways of doing so, such as developing a standard API approach or set of approaches for this and then using this to add new APIs. However, I think that extending the standard library into streaming APIs is by far the less important side of the "range over func" proposal (although this is what the "more motivation" section of the wiki article devotes the most space to).

Right from the beginning, one of the criticisms of Go was that it had some privileged, complex builtin types that couldn't be built using normal Go facilities, such as maps. Generics have made it mostly possible to do equivalents of these (generic) types yourself at the language level (although the Go compiler still uniquely privileges maps and other builtin types at the implementation level). However, these complex builtin types still retain some important special privileges in the language, and one of them is that they were the only types that you could write convenient 'range' based for loops.

In Go today you can write, for example, a set type or a key/value type with some complex internal storage implementation and make it work even for user-provided element types (through generics). But people using your new container types cannot write 'for elem := range set' or 'for k, v := range kvstore'. The best you can give them is an explicit push or pull based iterator based on your type (in a push iterator, you provide a callback function that is given each value; in a pull iterator, you repeatedly call some function to obtain the next value). The "range over func" proposal bridges this divide, allowing non-builtin types to be ranged over almost as easily as builtin types. You would be able to write types that let people write 'for elem := range set.Forward()' or 'for k, v := kvstore.Walk()'.

This is an issue that can't really be solved without language support. You could define a standard API for iterators and iteration (and the 'iter' package covered in the wiki article sort of is that), but it would still be more code and somewhat awkward code for people using your types to write. People are significantly attracted to what is easy to program; the more difficult it is to iterate user types compared to builtin types, the less people will do it (and the more they will use builtin types even when they aren't a good fit). If Go wants to put user (generic) types on almost the same level (in the language) as builtin types, then I feel it needs some version of a "range over func" approach.

(Of course, you may feel that Go should not prioritize putting user types on almost the same level as builtin types.)

GoRangefuncAndUserContainers written at 22:30:08; Add Comment

2024-02-14

Understanding a recent optimization to Go's reflect.TypeFor

Go's reflect.TypeFor() is a generic function that returns the reflect.Type for its type argument. It was added in Go 1.22, and its initial implementation was quite simple but still valuable, because it encapsulated a complicated bit of reflect usage. Here is that implementation:

func TypeFor[T any]() Type {
  return TypeOf((*T)(nil)).Elem()
}

How this works is that it constructs a nil pointer value of the type 'pointer to T', gets the reflect.Type of that pointer, and then uses Type.Elem() to go from the pointer's Type to the Type for T itself. This requires constructing and using this 'pointer to T' type (and its reflect.Type) even though we only what the reflect.Type of T itself. All of this is necessary for reasons to do with interface types.

Recently, reflect.TypeFor() was optimized a bit, in CL 555597, "optimize TypeFor for non-interface types". The code for this optimization is a bit tricky and I had to stare at it for a while to understand what it was doing and how it worked. Here is the new version, which starts with the new optimization and ends with the old code:

func TypeFor[T any]() Type {
  var v T
  if t := TypeOf(v); t != nil {
     return t
  }
  return TypeOf((*T)(nil)).Elem()
}

What this does is optimize for the case where you're using TypeFor() on a non-interface type, for example 'reflect.TypeFor[int64]()' (although you're more likely to use this with more complex things like struct types). When T is a non-interface type, we don't need to construct a pointer to a value of the type; we can directly obtain the Type from reflect.TypeOf. But how do we tell whether or not T is an interface type? The answer turns out to be right there in the documentation for reflect.TypeOf:

[...] If [TypeOf's argument] is a nil interface value, TypeOf returns nil.

So what the new code does is construct a zero value of type T, pass it to TypeOf(), and check what it gets back. If type T is an interface type, its zero value is a nil interface and TypeOf() will return nil; otherwise, the return value is the reflect.Type of the non-interface type T.

The reason that reflect.TypeOf returns nil for a nil interface value is because it has to. In Go, nil is only sort of typed, so if a nil interface value is passed to TypeOf(), there is effectively no type information available for it; its old interface type is lost when it was converted to 'any', also known as the empty interface. So all TypeOf() can return for such a value is the nil result of 'this effectively has no useful type information'.

Incidentally, the TypeFor() code is also another illustration of how in Go, interfaces create a difference between two sorts of nils. Consider calling 'reflect.TypeFor[*os.File]()'. Since this is a pointer type, the zero value 'v' in TypeFor() is a nil pointer. But os.File isn't an interface type, so TypeOf() won't be passed a nil interface and can return a Type, even though the underlying value in the interface that TypeOf() receives is a nil pointer.

GoReflectTypeForOptimization written at 23:12:03; Add Comment

2024-02-11

Go 1.22's go/types Alias type shows the challenge of API compatibility

Go famously promises backward compatibility to the first release of Go and pretty much delivers on that (although the tools used to build Go programs have changed). Thus, one may be a bit surprised to read the following about go/types in the Go 1.22 Release Notes:

The new Alias type represents type aliases. Previously, type aliases were not represented explicitly, so a reference to a type alias was equivalent to spelling out the aliased type, and the name of the alias was lost. [...]

Because Alias types may break existing type switches that do not know to check for them, this functionality is controlled by a GODEBUG field named gotypesalias. [...] Clients of go/types are urged to adjust their code as soon as possible to work with gotypesalias=1 to eliminate problems early.

(The bold emphasis is mine, while the italics are from the release notes. The current default is gotypesalias=0.)

A variety of things in go/types return a Type, which is an interface type that 'represents a type of Go'. Well, more specifically these things return values of type Type, and these values have various underlying concrete types. Some code using go/types and dealing with Type values can handle them purely as interfaces, but other code needs to specifically handle all of the particular types (such as Array and so on). Since Type is an interface, such code will use a type switch that is supposed to be exhaustive over all of the concrete types of Type interface values.

Now we can see the problem. When Go introduces a new concrete type that can be returned as a Type value, those previously exhaustive type switches stop being exhaustive; there's a new concrete type that they're not prepared to handle. This could cause various problems in actual code. And Go has no way of requiring type switches to be exhaustive, so such code would still build fine but malfunction at runtime.

Much like the last time we saw something like this, this change is arguably not an API break, at least in theory; Go never explicitly promised that there was a specific and limited list of go/types types that implemented Type, and so in theory Go is free to expand the list. However, as we can see from the release notes (and the current behavior of not generating these new Alias types by default), the Go authors recognize that this is in practice a compatibility break, one that they're explicitly urging people to be prepared for.

What this shows is that true long term backward compatibility is very hard, and it's especially hard in an area that is inherently evolving, like exposing information about an evolving language. Getting complete backward compatibility requires more or less everything about an exposed API to be frozen, and that generally requires the area to be extremely well understood (and often pushes towards exposing very minimal APIs, which has its own problems).

As a side note, I think that Go is handling this change quite well. They've added the type to go/types so that people can add it to their own code (which will make it require Go 1.22 or later), and also provided a way that people can test the code (by building with gotypesalias=1). At the same time no actual 'Alias' types will appear (by default) until some time in the future; I'd guess no earlier than Go 1.24, a year from now.

Go122TypesAliasAndCompatibility written at 21:31:08; Add Comment

2024-02-02

One of my MH-E customizations: 'narrow-to-pending' (refiles and deletes)

I recently switched from reading my email with exmh, a graphical X frontend to (N)MH, to reading it with MH-E in GNU Emacs, which is also a frontend to (N)MH. I had a certain amount of customizations to exmh, and for reasons beyond the scope of this entry, I wound up with more for MH-E. One of those customizations is a new MH-E command (and keybinding for it), mh-narrow-to-pending.

Both exmh and MH-E process deleting messages and refiling them to folders in two phases. In the first phase your read your email and otherwise go over the current folder, marking messages to be deleted and refiled; once you're satisfied, you tell them to actually execute these pending actions. MH-E also a general feature to limit what messages are listed in the current folder. In Emacs jargon this general idea is known as narrowing, and there's various tools to 'narrow' the display of buffers to something of current interest. My customization narrows to show only the messages in the current folder that have pending actions on them; these are the messages that will be affected if you execute your pending actions.

So here's the code:

(defun mh-narrow-to-pending ()
  "Narrow to any message with a pending refile or delete."
  (interactive)
  (if (not (or mh-delete-list mh-refile-list))
      (message "There are no pending deletes or refiles.")
    (when (assoc 'mh-e-pending mh-seq-list) (mh-delete-seq 'mh-e-pending))
    (when mh-delete-list (mh-add-msgs-to-seq mh-delete-list 'mh-e-pending t t))
    (when mh-refile-list
      (mh-add-msgs-to-seq
       (cl-loop for folder-msg-list in mh-refile-list
                append (cdr folder-msg-list))
       'mh-e-pending t t))
    (mh-narrow-to-seq 'mh-e-pending)))

(This code could probably be improved, and reading it I've discovered that I've already forgotten what parts of it do and the details of how it works, although the broad strokes are obvious.)

Writing this code required reading the existing MH-E code to find out how it did narrowing and how it marked messages that were going to be refiled or deleted. In the usual GNU Emacs way, this is not a documented extension API for MH-E, although in practice it's unlikely to change and break my code. To the best of my limited understanding of making your own tweaks for GNU Emacs modes like MH-E, this is basically standard practice; generally you grub around in the mode's ELisp source, figure things out, and then do things on top of it.

There are two reasons that I never tried to write something like this for exmh. The first is that exmh doesn't do anywhere near as much with the idea of 'narrowing' the current folder display. The other is that I wound up using the two differently. In MH-E, it's become quite common for me to pick through my inbox (or sometimes other folders) for messages that I'm now done with, going far enough back in one pass that I wind up with a sufficient patchwork that I want to double check what exactly I'm going to be doing before I commit my changes. Since I can easily narrow to messages in general, narrowing to see these pending changes was a natural idea.

(Picking through the past week or more of email threads in my inbox has become a regular Friday activity for me, especially given that MH-E has a nice threaded view.)

It's easy to fall into the idea that any readily extendable program is kind of the same, because with some work you can write plugins, extensions, or other hacks that make it dance to whatever tune you want. What my experience with extending MH-E has rubbed my nose into is that the surrounding context matters in practice, both in how the system already works and in what features it offers that are readily extended. 'Narrow to pending' is very much an MH-E hack.

MHENarrowToPending written at 22:57:05; Add Comment

2024-01-25

In Go, I'm going to avoid using 'any' as an actual type

As modern Go programmers know, when Go introduced generics it also introduced a new 'any' type. This is officially documented as:

For convenience, the predeclared type any is an alias for the empty interface.

The 'any' type (alias) exists because it's extremely common in code that's specifying generic types to want to be able to say 'any type', and the way this is done in generics is 'interface{}', the empty interface. This makes generic code clearly easier to read and follow. Consider these two versions of the signature of reflect.TypeFor

func TypeFor[T any]() Type
func TypeFor[T interface{}]() Type

These are semantically equivalent but the first is clearer, because you don't have to remember this special case of what 'interface{}' means. Instead, it's right in the name 'any' (and there's less syntactic noise).

But after Go generics became a thing, there's been a trend of using this new 'any' alias outside of generic types, instead of writing out 'interface{}'. I don't think this is a good idea. To show why, consider the following two function signatures, both of which use 'any':

func One[T any](v T) bool
func Two(v any) bool

These two function signatures look almost the same, but they have wildly different meanings, even if (or when) they're invoked with the same argument. The effects of 'One(10)' are rather different from 'Two(10)', since 'One' is a generic function while 'Two' is a regular one. Now consider them written this way:

func One[T any](v T) bool
func Two(v interface{}) bool

Now we see clearly what Two() is doing differently than One(); it's obvious that it isn't taking 'any type' as such, but instead it's taking a generic interface as the argument type. This makes it obvious that a non-interface value will be converted to an interface value (and will tell some people that an interface value will lose its interface type).

This increased immediate clarity without needing to remember what 'any' is why I'm planning to use 'interface{}' in my code in the future, and why I think you should too. Yes, 'any' is shorter and it has a well defined meaning in the specification and we can probably remember the special meaning all of the time. But why give ourselves that extra cognitive burden when we can be explicit?

(In generics, the argument goes the other way; 'any' really does mean 'any type', and the 'any' name is clearer than writing 'interface{}' and then needing to remember that that's how generics do it.)

In a sense the 'any' name is a misnomer when used as a type. It's true that 'interface{}' will accept any type, but used as a type, it doesn't mean 'any type'; it means specifically the type 'an empty interface', which is to say an interface that has no methods, which implies interface type conversion (unless you already have an 'interface{}' value). Since 'any' does mean 'any type' in the context of generics, I think it's better to use a different name for each thing, even if Go formally makes the names equivalent. The names of things are fundamentally for people.

GoAvoidingAnyAsAType written at 23:03:50; Add Comment

2024-01-14

Git branches as a social construct

Over on the Fediverse, I had a half-baked thesis:

A half-baked thesis: branches in Git are a social construct, somewhat enabled by technical features. We talk about things having been done on a branch or existing on a branch, or what branches are what on an intertwined tree of them, even when this is not something you can find in the Git repository.

(This is since commits aren't permanently associated with a branch; they are merely currently reachable from one or more branches. What branch a multi-head-reachable commit is on is up to us.)

The background on this is more or less Julia Evans' git branches: intuition & reality, or more exactly a Fediverse discussion between Julia Evans and Mark Dominus (and Mark Dominus's I wish people would stop insisting that Git branches are nothing but refs).

This ties into my long standing view that modern version control systems are a user interface to their underlying mathematics. Git has an internal, mathematical view of what 'branches' are, but very few people actually use this mathematical view; instead we use a variety of 'user interface' views of what branches are and how we think of them. Git supports these user views of branches with various 'porcelain' features.

Some projects using Git actively work to create branches that have a more concrete and durable existence. For instance, commits on a Go release branch have the branch name in the commit's title, which is something the Go project does for a lot of branches that are used in Go development and release. For development branches specifically, this durably marks commits as having been done on the branch even after the branch is merged to the 'main' development branch.

Certainly, how I normally think of Git branches is different from their technical existence, and it differs from branch to branch. For example, in a typical repository I think of the 'main' branch as running all the way back to the creation of the repository, but other branches as only running back to where they split from 'main', despite this not being technically correct.

(Another sign of Git branches as being a bit socially constructed is how you can rename them (per the comments).)

PS: There are other VCSes where branches have a more durable existence in the VCS history. These VCSes are neither wrong nor right; my view is that they've taken a different view of both the UI and the mathematics of what 'branches' are in their mathematical version of version control.

GitBranchesSocialConstructs written at 21:54:44; Add Comment

2024-01-04

'Unmaintained' (open source) code represents a huge amount of value

I recently read Aaron Ballman's Musings on the C charter (via). As part of musing on backward compatibility in new versions of the C standard, Ballman wrote:

[...] I would love to see this principle updated to set a time limit, along the lines of: existing code that has been maintained to not use features marked deprecated, obsolescent, or removed in the past ten years is important; unmaintained code and existing implementations are not. If you cannot update your code to stop relying on deprecated functionality, your code is not actually economically important — people spend time and money maintaining things that are economically important.

To put it one way, I disagree strongly with the view that 'unmaintained' code is not valuable or important. In the open source world, I routinely use a significant number of programs and a large amount of code that no longer sees meaningful changes and development. This code may be maintained in the sense that there is someone who will fix security issues and important bugs, and maybe make a few changes here and there, but it is not 'maintained' in the sense that I think Ballman means, where it undergoes enough development that changing away from newly deprecated functionality (in C or any other language) would be lost in the noise.

(This code has 'value' in the sense that there's a community of people who are (still) using the software, often happily and by choice. Often these are relatively small communities, although not always. If there's no community still using the code, then it's mostly unimportant.)

Some of this open source code is genuinely more or less finished; its authors don't have any particular features they want to add to it. Other amounts of this open source code has fallen out of favour, with no one left behind that is interested in active development to move it forward, but potentially with plenty of people who derive value from its current working state. You can probably name projects for each camp.

(An extremely relevant example of the second case is X. People are quite aggressively not doing any further development of X, and they're happy to tell you about it. At the same time, a great deal of things still run on X.)

Making a future version of C (as an example) unable to build those code bases is effectively branching the language, in much the same way (although probably to a lesser extent) than the Python 2 versus Python 3 split. If C compilers fully support the old version of C, everything is probably reasonably fine; even though the language has branched, old projects can continue to use the old language forever. If C compilers start deciding that they want to drop the old version of C because it's been a while, we are not so fine.

PS: This has also come up in the case of Go. Old Go code itself still compiles and works fine, but the Go build environment has changed significantly enough that old code only builds through what is basically a hack in the main Go toolchain. Based on personal experience, I can tell you that there are a number of Go programs out there in the world that have not had even the minimal update to build using Go modules, but which are likely still used.

UnmaintainedCodeHugeValue written at 23:33:19; Add Comment

2023-12-18

In Go, constant variables are not used for optimization

Recently I wrote about partially emulating #ifdef with build tags and consts, exploiting Go's support for dead code elimination, and I said that this technique didn't work with variables. That's actually a somewhat interesting result. To see how it is, let's start with a simple Go program, where the following code is the entire program:

package main
import "fmt"

var doThing bool

func main() {
  fmt.Println("We may or may not do the thing.")

  if doThing {
     fmt.Println("We did the thing.")
  }
}

Here, 'doThing' is a boolean variable that is left at a zero value (false), and isn't exported on top of being in the 'main' package. There's nothing in the Go specification that allows the false value of 'doThing' to ever change. Despite this, if you inspect the resulting code the if and its call to 'fmt.Println()' is still present. If you go in with a debugger and manually set doThing to true, this code will run.

If you feed a modern C compiler a similar program, with 'doThing' declared as a static int, what you get back is code that has optimized out the code guarded by 'doThing'. The C compiler knows that the rules of the C abstract machine don't permit 'doThing' to change, so it has optimized accordingly. Functionally your 'static int doThing;' is now a constant, so the C compiler has then proceeded to do dead code elimination. The C compiler doesn't care that you could, for example, go in with a debugger and want to change the value of 'doThing', because the existence of debuggers is not included in the C abstract machine.

(This focus of C optimization on the C abstract machine and nothing beyond it is somewhat controversial, to put it one way.)

Go could have chosen to optimize this case in the same way as C compilers do, but for whatever reasons the Go developers didn't choose to do so. One possible motivation to not do this is the case of debuggers, where you can manually switch 'doThing' on at runtime. Another possible motivation is simply to speed up compiling Go code and to keep the compiler simpler. A C compiler needs a certain amount of infrastructure so that it knows that the static int 'doThing' never has its value changed, and then to propagate that knowledge through code generation; Go doesn't.

Well actually that's a bit of a white lie. The normal Go toolchain doesn't do all of this with these constant variables, but there's also gccgo, a Go implementation that's a frontend for GCC (along side C, C++, and some others). Since gccgo is built on top of GCC, it can inherit all of GCC's C focused optimizations, such as recognizing constant variables, and if you invoke gccgo with the optimization level high enough, it will optimize the 'doThing' guarded expression out just like C (this omits the first call to fmt.Println to make the generated code slightly clearer).

(There have been some efforts to build a Go toolchain based on LLVM, and I'd expect such a toolchain to also optimize this Go code the way gccgo does.)

GoKeepsConstantVariables written at 21:09:55; Add Comment

(Previous 10 or go back to December 2023 at 2023/12/14)

Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.