Wandering Thoughts archives

2019-07-03

Converting a variable to a single-element slice in Go via unsafe

I was recently reading Chris Wellons' Go Slices are Fat Pointers. At the end of the article, Wellons says:

Slices aren’t as universal as pointers, at least at the moment. You can take the address of any variable using &, but you can’t take a slice of any variable, even if it would be logically sound.

[...] However, if you really wanted to do this, the unsafe package can accomplish it. I believe the resulting slice would be perfectly safe to use:

// Convert to one-element array, then slice
fooslice = (*[1]int)(unsafe.Pointer(&foo))[:]

I had to read this carefully before I understood what it was doing, but then after I read the documentation for unsafe.Pointer() carefully, I believe that this is fully safe. So let's start with what it's doing. The important thing is this portion of the expression:

(*[1]int)(unsafe.Pointer(&foo))

This is essentially reinterpreting foo from an integer to a one-element array of integers, by taking a pointer to it and then converting that to a pointer to a one-element array. I believe that this use of unsafe.Pointer() is probably valid, because it seems like it falls under the first valid use in the documentation:

(1) Conversion of a *T1 to Pointer to *T2.

Provided that T2 is no larger than T1 and that the two share an equivalent memory layout, this conversion allows reinterpreting data of one type as data of another type. [...]

In Go today, an integer and a one-element array of integers are the same size, making the first clause true and pretty much requiring that the second one is true as well. I don't think that Go requires this in the language specification, but in practice it's very likely to be the case in any implementation that wants to adhere to Go's ethos of efficiency and minimalism. Once we have a valid pointer to a (valid) one-element array of int, it's perfectly legal to create a slice from it, which is what the '[:]' does. So if this use of unsafe is valid, the resulting slice is fully safe and valid.

Now we get to the interesting question of why Go doesn't allow this without the use of unsafe.Pointer(). One possible answer is that this is not allowed simply because it would require extra work in the language specification and the compiler. This may well be the case (and it's certainly a very Go style reason), but another possibly reason is that Go doesn't want to require that all implementations make a one-element array have exactly the same memory layout and implementation as a single variable. By confining this to the limited assurances of unsafe and not making it part of the guaranteed language specification, Go keeps people's options open.

(Of course this is only theoretical, because in practice a new implementation will likely want to reuse as much of the standard library as possible and the current standard library uses unsafe in various places. If you don't match what works with unsafe today in mainline Go, you're going to have to rewrite some of that code. Also, see how unsafe type conversions are still garbage collection safe for some more discussion of this area.)

programming/GoVariableToArrayConversion written at 22:00:33; Add Comment

Finding out what 32-bit x86 Linux programs your users are running

Canonical has recently created a certain amount of uncertainty over how many 32-bit x86 programs are still going to run on Ubuntu 20.04 LTS and future versions. Statically linked 32-bit programs seem likely to keep working, if you have any of those, but Canonical is making noises about omitting some or many of the 32-bit shared libraries that dynamically linked 32-bit programs need to run (see my entry on what 32-bit programs need on a 64-bit system). This is quite relevant for us, as we definitely have a certain number of users with a certain amount of old 32-bit programs.

(Canonical's latest statement is not entirely clear, and anyway it only applies to 19.10 and 20.04 LTS, not versions past them.)

Given all of this, it would be nice to find out (in advance) what 32-bit programs our users are still running. There are probably a number of different ways to do this, but the approach I've explored is using the Linux kernel's audit system. The audit framework can log both the use of 32-bit system calls and the use of various paths, so there are several specific approaches you can take. I will start with the one that doesn't work.

The obvious thing to attempt is to record 32-bit execve()s. This works in one sense, in that it does what it says on the can, but it doesn't in another sense; it records when a 32-bit program uses an execve(), not when an execve() results in a 32-bit program being run. As far as I can see, there is no direct way of doing the latter; instead we have to do it indirectly.

If you're content to log a great deal of audit records, you can record many or basically all use of 32-bit system calls. This will definitely catch 32-bit programs, but it's noisy; you'll log a lot of records for each program. To do better we need to find a system call, perhaps with a specific set of arguments, that is only used relatively infrequently in 32-bit programs. Fortunately there is one, at least for programs written in C or using the standard C library entry point, and it is 'brk(NULL)', which is run very early on in the program's life to find the break address. In audit syntax, this is:

-a always,exit -F arch=b32 -F a0=0 -S brk -k 32bit-program

If it matters to you, this audit rule will not catch statically linked 32-bit Go programs (although it will catch statically linked 32-bit C programs). Of course as statically linked programs, they don't care if Canonical drops 32-bit shared libraries.

If we specifically want to catch 32-bit dynamically linked programs, we can use an audit path watch on /lib/ld-linux.so.2. However, there is a complication; ld-linux.so.2 is a symlink, so we must watch the specific thing it points to, and that depends on the version of GNU libc you have installed and on the system you're on. So, first you use 'readlink -f /lib/ld-linux.so.2' to find the expanded path, then put the result into an audit rule like so:

-w /lib/i386-linux-gnu/ld-2.27.so -p x -k 32bit-program

(This is for Ubuntu 18.04, with GNU libc 2.27.)

You can also set an audit path watch on the main C shared library, libc.so.6, or for that matter on any shared library of interest or that you think is especially in danger of not being available in future Ubuntu versions. Just remember to always expand it out to the real path or you'll get audit rules that mysteriously don't work (ask me how I know this). If you want, you can be specific by watching openat() system calls for the path, since GNU libc uses that when mmap()'ing shared libraries.

If you want to get audit records from statically linked non-C programs, you'll need to use strace to investigate what system calls they make. A cross-built statically linked 32-bit Go program appears to call set_thread_area and sched_getaffinity early on, for example. Your mileage will vary with other languages and other C libraries.

linux/Finding32BitProgramsWithAudit written at 00:53:42; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.