Wandering Thoughts

2015-09-02

Thinking about the different models of supplying computing

It's the time of year when new graduate students show up here, so one of the things on my mind has been the various ways that computers can be supplied to people in an environment like ours. There are at least three that come to mind.

First is the 'bring your own device' model where every incoming graduate student (or professor) is expected to bring their own computer (probably a laptop) and, as a corollary, to look after it. Perhaps we'd supply some niceties like external screens to hook up to them. The BYOD approach is popular partly because any number of people are going to do this anyways.

Then there is the 'hardware only' model, where we hand a computer to every new graduate student but make no attempt to manage or control it beyond that; the graduate student can run whatever they want in whatever configuration they want. Probably we'd preinstall some OS in a recommended configuration just for convenience (and many grad students would leave it as-is). Lots of people like this model for its freedom and similarity to the BYOD experience (at least until the OS install blows up in their face).

The final model is managed desktops, where we both supply hardware and maintain the OS installed on it. On the one hand, we guarantee that it works right; on the other hand, people lose the freedom to run whatever they want and have to generally live with our choices. 'We don't support that' will probably get said a lot.

(Note that these are not necessarily a good set of options for any environment other than our peculiar one.)

As you might suspect, in practice right now we have a mix of all three options. The historical evolution of our environment is that we started out providing fully managed computing because computing was too expensive for any other answer, but over time the decrease in computing costs (especially compared to staff costs) has caused more and more people to shift towards BYOD and 'here, have a box'.

(I will skip a discussion of trying to do managed Windows installs and just say that we were and are primarily a Unix shop without much expertise in that area. This leads to non-technical issues beyond the scope of this entry.)

I'm mulling this over partly because how computing get supplied to people has a big impact on what services they're interested in consuming from us (and how). For one obvious example, in the days when we provided serial terminals on everyone's desk, having Unix servers for people to log in to was a big deal and they were very important. Today an increasing number of people here have probably only used our login servers to change their password.

(Since we're a Computer Science department, you could actually argue that we should actively push people to interact with Unix because Unix is an important part of effective, practical computing and so something they should be learning. But that's another debate entirely.)

sysadmin/ComputingSupplyModels written at 01:47:00; Add Comment

2015-08-31

CGo's Go string functions explained

As plenty of its documentation will tell you, cgo provides four functions to convert between Go and C types by making copies of the data. They are tersely explained in the CGo documentation; too tersely, in my opinion, because the documentation only covers certain things by implication and omits two very important glaring cautions. Because I made some mistakes here I'm going to write out a longer explanation.

The four functions are:

func C.CString(string) *C.char
func C.GoString(*C.char) string
func C.GoStringN(*C.char, C.int) string
func C.GoBytes(unsafe.Pointer, C.int) []byte

C.CString() is the equivalent of C's strdup() and copies your Go string to a C char * that you can pass to C functions, just as documented. The one annoying thing is that because of how Go and CGo types are defined, calling C.free will require a cast:

cs := C.CString("a string")
C.free(unsafe.Pointer(cs))

Note that Go strings may contain embedded 0 bytes and C strings may not. If your Go string contains one and you call C.CString(), C code will see your string truncated at that 0 byte. This is often not a concern, but sometimes text isn't guaranteed to not have null bytes.

C.GoString() is also the equivalent of strdup(), but for going the other way, from C strings to Go strings. You use it on struct fields and other things that are declared as C char *'s, aka *C.char in Go, and (as we'll see) pretty much nothing else.

C.GoStringN() is the equivalent of C's memmove(), not to any normal C string function. It copies the entire length of the C buffer into a Go string, and it pays no attention to null bytes. More exactly, it copies them too. If you have a struct field that is declared as, say, 'char field[64]' and you call C.GoStringN(&field, 64), the Go string you get will always be 64 characters long and will probably have a bunch of 0 bytes at the end.

(In my opinion this is a bug in cgo's documentation. It claims that GoStringN takes a C string as the argument, but it manifestly does not, as C strings are null-terminated and GoStringN does not stop at null bytes.)

C.GoBytes() is a version of C.GoStringN() that returns a []byte instead of a string. Since it doesn't claim to be taking a C string as the argument, it's clearer that it is simply a memory copy of the entire buffer.

If you are copying something that is not actually a null terminated C string but is instead a memory buffer with a size, C.GoStringN() is exactly what you want; it avoids the traditional C problem of dealing with 'strings' that aren't actually C strings. However, none of these functions are what you want if you are dealing with size-limited C strings in the form of struct fields declared as 'char field[N]'.

The traditional semantics of a fixed size string field in structs, fields that are declared as 'char field[N]' and described as holding a string, is that the string is null terminated if and only if there is room, ie if the string is at most N-1 characters long. If the string is exactly N characters long, it is not null terminated. This is a fruitful source of bugs even in C code and is not a good API, but it is an API that we are generally stuck with. Any time you see such a field and the documentation does not expressly tell you that the field contents are always null terminated, you have to assume that you have this sort of API.

Neither C.GoString() nor C.GoStringN() deal correctly with these fields. Using GoStringN() is the less wrong option; it will merely leave you with N-byte Go strings with plenty of trailing 0 bytes (which you may not notice for some time if you usually just print those fields out; yes, I've done this). Using the tempting GoString() is actively dangerous, because it internally does a strlen() on the argument; if the field lacks a terminating null byte, the strlen() will run away into memory beyond it. If you're lucky you will just wind up with some amount of trailing garbage in your Go string. If you're unlucky, your Go program will take a segmentation fault as strlen() hits unmapped memory.

(In general, trailing garbage in strings is the traditional sign that you have an unterminated C string somewhere.)

What you actually want is the Go equivalent of C's strndup(), which guarantees to copy no more than N bytes of memory but will stop before then if it finds a null byte. Here is my version of it, with no guarantees:

func strndup(cs *C.char, len int) string {
   s := C.GoStringN(cs, C.int(len))
   i := strings.IndexByte(s, 0)
   if i == -1 {
      return s
   }
   return C.GoString(cs)
}

This code does some extra work in order to minimize extra memory usage due to how Go strings can hold memory. You may want to take the alternate approach of returning a slice of the GoStringN() string. Really sophisticated code might decide which of the two options to use based on the difference between i and len.

Update: Ian Lance Taylor showed me the better version:

func strndup(cs *C.char, len int) string {
   return C.GoStringN(cs, C.int(C.strnlen(cs, C.size_t(len))))
}

Yes, that's a lot of casts. That's the combination of Go and CGo typing for you.

programming/GoCGoStringFunctions written at 23:49:44; Add Comment

Turning, well copying blobs of memory into Go structures

As before, suppose (not entirely hypothetically) that you're writing a package to connect Go up to something that will provide it with blobs of memory that are actually C structs; these might be mmap()'d files, information from a library, or whatever. Once you have a compatible Go struct, you still have to get the data from a C struct (or raw memory) to the Go struct.

One way to do this is to manually write your own struct copy function that does it field by field (eg 'io.Field = ks_io.field' for each field). As with defining the Go structs by hand, this is tedious and potentially error prone. You can do it and you'll probably have to if the C struct contains unions or other hard to deal with things, but we'd like an easier approach. Fortunately there are two good ones for two different cases. In both cases we will wind up copying the C struct or the raw memory to a Go struct variable that is an exact equivalent of the C struct (or at least we hope it is).

The easy case is when we're dealing with a fixed struct that we have a known Go type for. Assuming that we have a C void * pointer to the original memory area called ks.ks_data, we can adopt the C programmer approach and write:

var io IO
io = *((*IO)(ks.ks_data))
return &io

This casts ks.ks_data to a pointer to an IO struct and then dereferences it to copy the struct itself into the Go variable we made for this. Depending on the C type of ks_data, you may need to use the hammer of unsafe.Pointer() here:

io = *((*IO)(unsafe.Pointer(ks.ks_data)))

At this point, some people will be tempted to skip the copying and just return the 'casted-to-*IO' ks.ks_data pointer. You don't want to do this, because if you return a Go pointer to C data, you're coupling Go and C memory management lifetimes. The C memory must not be freed or reused for something else for as long as Go retains at least one pointer to it, and there is no way for you to find out when the last Go reference goes away so that you can free the C memory. It's much simpler to treat 'C memory' as completely disjoint from 'Go memory'; any time you want to move some information across the boundary, you must copy it. With copying we know we can free ks.ks_data safely the moment the copy is done and the Go runtime will handle the lifetime of the io variable for us.

The more difficult case is when we don't know what structs we're dealing with; we're providing the access package, but it's the callers who actually know the structs are. This situation might come up in a package for accessing kernel stats, where drivers or other kernel systems can export custom stats structs. Our access package can provide specific support for known structs, but we need an escape hatch for when the callers knows that some specific kernel system is providing a 'struct whatever' and it wants to retrieve that (probably into an identical Go struct created through cgo).

The C programmer approach to this problem is memmove(). You can write memmove() in Go with sufficiently perverse use of the unsafe package, but you don't want to. Instead we can use the reflect package to create a generic version of the specific 'cast and copy' code we used above. How to do this wasn't obvious to me until I did a significant amount of flailing around with the package, so I'm going to go through the logic of what we're doing in detail.

We'll start with our call signature:

func (k *KStat) CopyTo(ptri interface{}) error { ... }

CopyTo takes a pointer to a Go struct and copies our C memory in ks.ks_data into the struct. I'm going to omit the reflect-based code to check ptri to make sure it's actually a pointer to a suitable struct in the interests of space, but you shouldn't in real code. Also, there are a whole raft of qualifications you're going to want to impose on what types of fields that struct can contain if you want to at least pretend that your package is somewhat memory safe.

To actually do the copy, we first need to turn this ptri interface value into a reflect.Value that is the destination struct itself:

ptr := reflect.ValueOf(ptri)
dst := ptr.Elem()

We now need to cast ks.ks_data to a Value with the type 'pointer to dst's type'. This is most easily done by creating a new pointer of the right type with the address taken from ks.ks_data:

src := reflect.NewAt(dst.Type(), unsafe.Pointer(ks.ks_data))

This is the equivalent of 'src := ((*IO)(ks.ks_data))' in the type-specific version. Reflect.NewAt is there for doing just this; its purpose is to create pointers for 'type X at address Y', which is exactly the operation we need.

Having created this pointer, we then dereference it to copy the data into dst:

dst.Set(reflect.Indirect(src))

This is the equivalent of 'io = *src' in the type-specific version. We're done.

In my testing, this approach is surprisingly robust; it will deal with even structs that I didn't expect it to (such as ones with unexported fields). But you probably don't want to count on that; it's safest to give CopyTo() straightforward structs with only exported fields.

On the whole I'm both happy and pleasantly surprised by how easy it turned out to be to use the reflect package here; I expected it to require a much more involved and bureaucratic process. Getting to this final form involved a lot of missteps and unnecessarily complicated approaches, but the final form itself is about as minimal as I could expect. A lot of this is due to the existence of reflect.NewAt(), but there's also that Value.Set() works fine even on complex and nested types.

(Note that while you could use the reflect-based version even for the first, fixed struct type case, my understanding is that the reflect package has not insignificant overheads. By contrast the hard coded fixed struct type code is about as minimal and low overhead as you can get; it should normally compile down to basically a memory copy.)

Sidebar: preserving Go memory safety here

I'm not fully confident that I have this right, but I think that to preserve memory safety in the face of this memory copying you must insure that the target struct type does not contain any embedded pointers, either explicit ones or ones implicitly embedded into types like maps, chans, interfaces, strings, slices, and so on. Fixed-size arrays are safe because in Go those are just fixed size blocks of memory.

If you copy a C struct containing pointers into a Go struct containing pointers, what you're doing is the equivalent of directly returning the 'casted-to-*IO' ks.ks_data pointer. You've allowed the creation of a Go object that points to C memory and you now have the same C and Go memory lifetime issues. And if some of the pointers are invalid or point to garbage memory, not only is normal Go code at risk of bad things but it's possible that the Go garbage collector will wind up trying to dereference them and take a fault.

(This makes it impossible to easily copy certain sorts of C structures into Go structures. Fortunately such structures rarely appear in this sort of C API because they often raise awkward memory lifetime issues even in C.)

programming/GoMemoryToStructures written at 02:50:28; Add Comment

2015-08-30

Getting C-compatible structs in Go with and for cgo

Suppose, not entirely hypothetically, that you're writing a package to connect Go up to something that will provide it blobs of memory that are C structs. These structs might be the results of making system calls or they might be just informational things that a library provides you. In either case you'd like to pass these structs on to users of your package so they can do things with them. Within your package you can use the cgo provided C.<whatever> types directly. But this is a bit annoying (they don't have native Go types for things like integers, which makes interacting with regular Go code a mess of casts) and it doesn't help other code that imports your package. So you need native Go structs, somehow.

One way is to manually define your own Go version of the C struct. This has two drawbacks; it's tedious (and potentially error-prone), and it doesn't guarantee that you'll wind up with exactly the same memory layout that C has (the latter is often but not always important). Fortunately there is a better approach, and that is to use cgo's -godefs functionality to more or less automatically generate struct declarations for you. The result isn't always perfect but it will probably get you most of the way.

The starting point for -godefs is a cgo Go source file that declares some Go types as being some C types. For example:

// +build ignore
package kstat
// #include <kstat.h>
import "C"

type IO C.kstat_io_t
type Sysinfo C.sysinfo_t

const Sizeof_IO = C.sizeof_kstat_io_t
const Sizeof_SI = C.sizeof_sysinfo_t

(The consts are useful for paranoid people so you can later cross-check the unsafe.Sizeof() of your Go types against the size of the C types.)

If you run 'go tool cgo -godefs <file>.go', it will print out to standard output a bunch of standard Go type definitions with exported fields and everything. You can then save this into a file and use it. If you think the C types may change, you should leave the generated file alone so you won't have a bunch of pain if you have to regenerate it; if the C types are basically fixed, you can annotate the generated output with eg godoc comments. Cgo worries about matching types and it will also insert padding where it existed in the original C struct.

(I don't know what it does if the original C struct is impossible to reconstruct in Go, for instance if Go requires padding where C doesn't. Hopefully it complains. This hope is one reason you may want to check those sizeofs afterwards.)

The big -godefs limitation is the same limitation as cgo has in general: it has no real support for C unions, since Go doesn't have them. If your C struct has unions, you're on your own to figure out how to deal with them; I believe cgo translates them as appropriate sized uint8 arrays, which is not too useful to actually access the contents.

There are two wrinkles here. Suppose you have one struct type that embeds another struct type:

struct cpu_stat {
   struct cpu_sysinfo  cpu_sysinfo;
   struct cpu_syswait  cpu_syswait;
   struct vminfo       cpu_vminfo;
}

Here you have to give cgo some help, by creating Go level versions of the embedded struct types before the main struct type:

type Sysinfo C.struct_cpu_sysinfo
type Syswait C.struct_cpu_syswait
type Vminfo  C.struct_cpu_vminfo

type CpuStat C.struct_cpu_stat

Cgo will then be able to generate a proper Go struct with embedded Go structs in CpuStat. If you don't do this, you get a CpuStat struct type that has incomplete type information; the 'Sysinfo' et al fields in it will refer to types called _Ctype_... that aren't defined anywhere.

(By the way, I do mean 'Sysinfo' here, not 'Cpu_sysinfo'. Cgo is smart enough to take that sort of commonly seen prefix off of struct field names. I don't know what its algorithm is for doing this, but it's at least useful.)

The second wrinkle is embedded anonymous structs:

struct mntinfo_kstat {
   ....
   struct {
      uint32_t srtt;
      uint32_t deviate;
   } m_timers[4];
   ....
}

Unfortunately cgo can't deal with these at all. This is issue 5253, and you have two options. The first is that at the moment, the proposed CL fix still applies to src/cmd/cgo/gcc.go and works (for me). If you don't want to build your own Go toolchain (or if the CL no longer applies and works), the other solution is to create a new C header file that has a variant of the overall struct that de-anonymizes the embedded struct by creating a named type for it:

struct m_timer {
   uint32_t srtt;
   uint32_t deviate;
}

struct mntinfo_kstat_cgo {
   ....
   struct m_timer m_timers[4];
   ....
}

Then in your Go file:

...
// #include "myhacked.h"
...

type MTimer C.struct_m_timer
type Mntinfo C.struct_mntinfo_kstat_cgo

Unless you made a mistake, the two C structs should have the same sizes and layouts and thus be totally compatible with each other. Now you can use -godefs on your version, remembering to make an explicit Go type for m_timer due to the first wrinkle. If you feel bold (and you don't think you'll need to regenerate things), you can then reverse this process in the generated Go file, re-anonymizing the MTimer type into the overall struct (since Go supports that perfectly well). Since you're not changing the actual contents, just where types are declared, the result should be layout-identical to the original.

PS: the file that's input to -godefs is set to not be built by the normal 'go build' process because it is only used for this godefs generation. If it gets included in the build, you'll get complaints about multiple definitions of your (Go) types. The corollary to this is that you don't need to have this file and any supporting .h files in the same directory as your regular .go files for the package. You can put them in a subdirectory, or keep them somewhere entirely separate.

(I think the only thing the package line does in the godefs .go file is set the package name that cgo will print in the output.)

programming/GoCGoCompatibleStructs written at 03:37:54; Add Comment

2015-08-29

The mailing list thread to bug tracking system problem

I will start with the thesis: open source projects would benefit from a canonical and easy way to take a mailing list message or thread and turn it into an issue or bug report in your bug tracking system.

It's entirely natural and rather normal for a but report to start with someone uncertainly asking 'is this supposed to happen?' or 'am I understanding this right?' questions on one of your mailing lists. They're not necessarily doing this because they don't know where to report bugs; often they may be doing it because they're not sure that what they're seeing is a bug (or at least a new bug), or they don't know how to file what your project considers a good bug report, and they don't want to take the hit of a bad bug report. It's usually easier to ask questions on a mailing list where some degree of ignorance is expected and accepted than to venture into what can be a sharp-edged swamp of a bug tracker.

If the requester is energetic, they'll jump through a lot of extra hoops to actually file a bug report once they've built up their confidence (or just been pointed to the right place). But in general, the more difficult the re-filing process is the fewer bug reports you're going to get, while the easier it is the more you'll get.

This leads me to my view that most open source projects make this too hard today, usually by having no explicit way to do it because their mailing list systems and bug tracking systems are completely separate things. Maybe this separation can be overcome through cultural changes so that brief pointers to mailing list messages or cut and paste copies from mailing list threads become acceptable as bug reports.

(My impression, perhaps erroneous, is that most open source projects want you to rewrite your bug reports from more or less scratch when you make them. The mailing list is the informal version of your report, the bug tracker gets the 'formal' one. Of course the danger here is that people just don't bother to write the formal version for various reasons.)

PS: I admit that one reason I've wound up feeling this way is that I'm currently sitting on a number of issues that I first raised on mailing lists and still haven't gotten around to filing bug reports for. And by now some of them are old enough that I'd have to reread a bunch of stuff just to recover the context I had at the time and understand the whole problem once again.

tech/MailingListToBugReport written at 02:27:12; Add Comment

2015-08-28

The somewhat surprising history of chroot()

The chroot() system call is one of those that I think of as being really old. Up until a while back, if you'd pressed me I'd have guessed that it originated in V7. Then I dug into it and in that tweet claimed it was from 4.2 BSD. This is what you'd kind of expect and what it sort of looks like, but it turns out to be wrong and the history of chroot() seems to be much more interesting than I expected.

The classic use of chroot() is for safely confining network daemons, and indeed this is what ftpd uses it for in 4.2 BSD. But there's another use of it in the 4.2 BSD tree lurking in a file called /usr/src/games/compat/runcompat.c. As a manpage and a Readme there make clear, this is a system to let you run PDP-11 V6 and V7 programs on your VAX. It uses chroot() to optionally put those programs inside a familiar V6 or V7 directory hierarchy. This appears to go more or less all the way back to 32V, the first Unix version to run on a VAX and thus the first one where this desire would come up. But it's not where chroot(2) was added.

Much to my surprise, it turns out that chroot(2) is in V7 after all. It's not obvious because there's no separate manpage for it; instead it's part of the chdir(2) manpage. I don't know why it's there, apart from 'someone thought it was an obvious feature'; tuhs.org has more or less complete V7 source, and there's no programs in the source tree that use it that I can find. Apparently it was added first and then people found various uses for it later. What we now think of as its most prominent uses (for things like confining anonymous ftp) are relatively latecomers to the game, since they started years after chroot() itself came into existence.

(I don't believe that the V7 chroot() was a security measure.)

Sidebar: why chroot is in the chdir manpage

V7 implements chdir(2) and chroot(2) with basically the same code; you can see it in /usr/sys/sys/sys4.c. It's pretty clever:

chdir()
{
    chdirec(&u.u_cdir);
}

chroot()
{
    if (suser())
        chdirec(&u.u_rdir);
}

chdirec(ipp)
register struct inode **ipp;
{
[...]

In other words, the only real difference between chdir and chroot is what field in the u user structure they act on. chdir acts on the 'current directory' field, chroot acts on the 'root directory' field.

The actual implementation of chroot requires a bit more kernel code than this, because of course the u_rdir field didn't exist before chroot was added. But there turn out to be remarkably few places that deal with u_rdir in the V7 kernel. By the way, reading this code makes me suspect that a V7 chroot was remarkably permeable; I don't see any reason why you can't just 'cd ..' your way right out of one (although I may be missing something subtle in the code). That obviously changed later.

(Note that the V7 manpage doesn't claim that chroot() confines processes; it simply says that it changes what '/' means to them. Which is literally what the kernel code does.)

unix/ChrootHistory written at 02:42:31; Add Comment

2015-08-27

Some notes on using Solaris kstat(s) in a program

Solaris (and Illumos, OmniOS, etc) has for a long time had a 'kstat' system for systematically providing and exporting kernel statistics to user programs. Like many such systems in many OSes, kstat doesn't need root permissions; all or almost all of the kstats are public and can be read by anyone. If you're a normal Solaris sysadmin, you've mostly interacted with this system via kstat(1) (as I have) or perhaps Perl, for which there is the Sun::Solaris::Kstat module. Due to me not wanting to write Perl, I opted to do it the hard way; I wrote a program that talks more or less directly to the C kstat library. When you do this, you are directly exposed to some kstat concepts that kstat(1) and the Perl bindings normally hide from you.

The stats that kstat shows you are normally given a four element name of the form module:instance:name:statistic. This is actually kind of a lie. A 'kstat' itself is the module:instance:name triplet, and is a handle for a bundle of related statistics (for example, all of the per-link network statistics exposed by a particular network interface). When you work at the C library level, getting a statistic is a three level process; you get a handle to the kstat, you make sure the kstat has loaded the data for its statistics, and then you can read out the actual statistic (how you do this depends on what type of kstat you have).

This arrangement makes sense to a system programmer, because if we peek behind the scenes we can see a two stage interaction with the kernel. When you call kstat_open() to start talking with the library, the library loads the index of all of the available kstats from the kernel into your process but it doesn't actually retrieve any data for them from the kernel. You only take the relatively expensive operation of copying some kstat data from the kernel to user space when the user asks for it. Since there are a huge number of kstats, this cost saving is quite important.

(For example, a random OmniOS machine here has 3,627 kstats right now. How many you have will vary depending on how many network interfaces, disks, CPUs, ZFS pools, and so on there are in your system.)

A kstat can have its statistics data in several different forms. The most common form is a 'named' kstat, where the statistics are in a list of name=value structs. If you're dealing with this sort of kstat, you can look up a specific named statistic in the data with kstat_data_lookup() or just go through the whole list manually (it's fully introspectable). The next most common form is I/O statistics, where the data is simply a C struct (a kstat_io_t). There are also a few kstats that return other structs as 'raw data', but to find and understand them you get to read the kstat(1) source code. Only named-data kstats really have statistics with names; everyone else really just has struct fields.

(kstat(1) and the Perl module hide this complexity from you by basically pretending that everything is a named-data kstat.)

Once read from the kernel, kstat statistics data does not automatically update. Instead it's up to you to update it whenever you want to, by calling kstat_read() on the relevant kstat again. What happens to the kstat's old data is indeterminate, but I think that you should assume it's been freed and is no longer something you should try to look at.

This brings us to the issue of how the kstat library manages its memory and what bits of memory may change out from underneath you when, which is especially relevant if you're doing something complicated while talking to it (as I am). I believe the answer is that kstat_read() changes the statistics data for a kstat and may reallocate it, kstat_chain_update() may cause random kstat structs and their data to be freed out from underneath you, and kstat_close() obviously frees and destroys everything.

(The Perl Kstat module has relatively complicated handling of its shadow references to kstat structs after it updates the kstats chain. My overall reaction is 'there are dragons here' and I would not try to hold references to any old kstats after a kstat chain update. Restarting all your lookups from scratch is perhaps a bit less efficient, but it's sure to be safe.)

In general, once I slowly wrapped my mind around what the kstat library was doing I found it reasonably pleasant to use. As with the nvpair library, the hard part was understanding the fundamental ideas in operation. Part of this was (and is) a terminology issue; 'kstat' and 'kstats' are in common use as labels for what I'm calling a kstat statistic here, which makes it easy to get confused about what is what.

(I personally think that this is an unfortunate name choice, since 'kstat' is an extremely attractive name for the actual statistics. Life would be easier if kstats were called 'kstat bundles' or something.)

solaris/KStatProgrammingNotes written at 02:02:26; Add Comment

2015-08-26

Why I wind up writing my own (sysadmin) tools

I have a tendency to write my own versions of tools, like bulk IO measurement tools, versions of netcat, and network bandwidth reporters. Historically there have been two official reasons for this and a third unofficial one.

First, when I write a tool myself I know exactly what it's doing and what its output means. This has historically been a problem with other people's tools, including system supplied ones (eg). If I'm going to wind up reading the source of a program just to be sure I really understand what it's telling me, I may not be saving much time over writing my own.

(Also, if I write it myself the tool can generally wind up doing exactly what I want in the way that I want it. Other people's tools may have various sorts of rough edges and little annoyances for me.)

Second, because finding and testing and investigating the existing potential options is generally a pain in the rear. People have written a very many tools for doing disk IO benchmarking, for example, and as an outsider it is all a big mess. I'm honest enough to admit that for almost anything I want, there probably are tools out there that would meet my needs and make me happy; the problem is finding them and sorting out the wheat from the chaff. It's unfortunately easy for it to be less work and less frustration to write my own, or at least to feel like it is or will be.

(The frustration primarily comes from investing time into things that turn out to not be useful despite all of the effort, especially if they have active irritations. And most programs have active irritations somewhere.)

The third, unofficial reason is that programming is fun. When it's going well, it's a pure shot of the 'solving problems' drug that I get only in moderation when I'm doing less glamorous activities like building and testing new systems (and never mind the humdrum daily routine; the problems I solve there are all very small). It's especially fun when I contrast it with the slogging work of searching the Internet for a pile of programs that might perhaps do what I want, getting copies of all of them, testing each out, and throwing most of them away. That's not solving problems, that's shuffling files around.

sysadmin/WhyIWriteOwnTools written at 01:55:15; Add Comment

2015-08-25

One view on practical blockers for IPv6 adoption

I recently wound up reading Russ White's Engineering Lessons, IPv6 Edition (via), which is yet another meditation by a network engineer about why people haven't exactly been adopting IPv6 at a rapid pace. Near the end, I ran across the following:

For those who weren't in the industry those many years ago, there were several drivers behind IPv6 beyond just the need for more address space. [...]

Part of the reason it's taken so long to deploy IPv6, I think, is because it's not just about expanding the address space. IPv6, for various reasons, has tried to address every potential failing ever found in IPv4.

As a sysadmin, my reaction to this is roughly 'oh god yes'. One of the major pain points in adding IPv6 (never mind moving to it) is that so much has to be changed and modified and (re)learned. IPv6 is not just another network address for our servers (and another set of routes); it comes with a whole new collection of services and operational issues and new ways of operating our networks. There are a whole host of uncertainties, from address assignment (both static and dynamic) on upwards. Given that right now IPv6 is merely nice to have, you can guess what this does to IPv6's priority around here.

Many of these new things exist primarily because the IPv6 people decided to solve all of their problems with IPv4 at once. I think there's an argument that this was always likely to be a mistake, but beyond that it's certainly made everyone's life more complicated. I don't know for sure that IPv6 adoption would be further along if IPv6 was mostly some enlarged address fields, but I rather suspect that it would be. Certainly I would be happier to be experimenting with it if that was the case.

What I can boil this down to is the unsurprisingly news that large scale, large scope changes are hard. They require a lot of work and time, they are difficult for many people to test, and they are unusually risky if something goes wrong. And in a world of fragile complexity, their complexity and complex interactions with your existing environment are not exactly confidence boosters. There are a lot of dark and surprising corners where nasty things may be waiting for you. Why go there until you absolutely have to?

(All of this applies to existing IPv4 environments. If you're building something up from scratch, well, going dual stack from the start strikes me as a reasonably good idea even if you're probably going to wind up moving slower than you might otherwise. But green field network development is not the environment I live in; it's rather the reverse.)

sysadmin/IPv6BigChangeProblem written at 00:50:07; Add Comment

2015-08-24

PS/2 to USB converters are complex things with interesting faults

My favorite keyboard and mice are PS/2 ones, and of course fewer and fewer PCs come with PS/2 ports (especially two of them). The obvious solution is PS/2 to USB converters, so I recently got one at work; half as an experiment, half as stockpiling against future needs. Unfortunately it turned out to have a flaw, but it's an interesting flaw.

The flaw was that if I held down CapsLock (which I remap to Control) and then hit some letter keys, the converter injected a nonexistent CapsLock key-up event into the event stream. The effect was that I got a sequence like '^Cccc'. This didn't happen with the real Control keys on my keyboard, only with CapsLock, and it doesn't happen with CapsLock when the keyboard is directly connected to my machine as a PS/2 keyboard. Unfortunately this is behavior that I reflexively count on working, so this PS/2 to USB converter is unsuitable for me.

(Someone else tested the same brand of converter on another PS/2 keyboard and saw the same thing, so this is not specific to my particular make of keyboards. For the curious, this converter was a ByteCC BT-2000.)

What this really says to me is two things. The first is that PS/2 to USB converters are actually complex items, no matter how small and innocuous they seem. Going from PS/2 to USB requires protocol conversion and when you do protocol conversion you can have bugs and issues. Clearly PS/2 to USB converters are not generic items; I'm probably going to have to search for one that not just 'works' according to most reports but that actually behaves correctly, and such a thing may not be easy to find.

(I suspect that such converters are actually little CPUs with firmware, rather than completely fixed ASICs. Little CPUs are everywhere these days.)

The second is the depressing idea that there are probably PS/2 keyboards out there that actively require this handling of CapsLock. Since it doesn't happen with the Control keys, it's not a generic bug with handling held modifier keys; instead it's specific behavior for CapsLock. People generally don't put in special oddball behavior for something unless they think they need to, and usually they've got reasons to believe this.

(For obvious reasons, if you have a PS/2 to USB converter that works and doesn't do this, I'd love to hear about it. I suspect that the ByteCC will not be the only one that behaves this way.)

tech/PS2ToUSBInterestingIssue written at 01:22:59; Add Comment

(Previous 10 or go back to August 2015 at 2015/08/23)

Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.