2025-07-01
How you can wind up trying to allocate zero bytes in C
One of the reactions I saw to my entry on malloc(0) being allowed
to return NULL is to wonder why you'd
ever do a zero-size allocation in the first place. Unfortunately,
it's relatively easy to stumble into this situation with simple
code in certain sorts of not particularly uncommon situations. The
most obvious one is if you're allocating memory for a variable size
object, such as a Python tuple or a JSON array. In a simple C
implementation these will typically have a fixed struct that contains
a pointer to a C memory block with either the actual elements or
an array of pointers to them. The natural way to set this up is to
write code that winds up calling 'malloc(nelems * sizeof(...))
'
or something like that, like this:
array_header * alloc_array(unsigned int nelems) { array_header *h; h = malloc(sizeof(array_header)) if (h == NULL) return NULL; /* get space for the element pointers except oops */ h->data = malloc(nelems * sizeof(void *)); if (h->data == NULL) { free(h); return NULL; } h->nelems = nelems; /* maybe some other initialization */ return h; }
(As a disclaimer, I haven't tried to compile this C code because I'm lazy, so it may contain mistakes.)
Then someone asks your code to create an empty tuple or JSON array and on some systems, things will explode because nelems will be 0 and you will wind up doing 'malloc(0)' and that malloc() will return NULL, as it's allowed to, and your code will think it's out of memory. You can obviously prevent this from happening, but it requires more code and thus requires you to have thought of the possibility.
(Allocating C strings doesn't have this problem because you always need one byte for the terminating 0 byte, but it can come up with other forms of strings where you track the length explicitly.)
One tricky bit about this code is that it will only ever go wrong in an obvious way on some uncommon systems. On most systems today, 'malloc(0)' returns a non-NULL result, usually because the allocator rounds up the amount of memory you asked for to some minimum size. So you can write this code and have it pass all of your tests on common platforms and then some day someone reports that it fails on, for example, AIX.
(It's possible that modern C linters and checkers will catch this; I'm out of touch with the state of the art there.)
As a side note, if malloc(0) returns anything other than NULL, I believe that each call is required to return a unique pointer (see eg the POSIX description of malloc()). I believe that these unique pointers don't have to point to actual allocated memory; they could point to some large reserved arena and be simply allocated in sequence, with a free() of them effectively doing nothing. But it's probably simpler to have your allocator round the size up and return real allocated memory, since then you don't have handle things like the reserved arena running out of space.
2025-06-24
Some bits on malloc(0) in C being allowed to return NULL
One of the little traps in standard C and POSIX is that malloc(0) is allowed to return NULL instead of a pointer. This makes people unhappy for various reasons. Today I wound up reading 017. malloc(0) & realloc(…, 0) ≠ 0, which runs through a whole collection of Unix malloc() versions and finds almost none of them which return NULL on malloc(0) except for some Unix System V releases that ship with an optional 'fast' malloc library that does return NULL on zero-sized allocations. Then AT&T wrote the System V Interface Definition and requires this 'fast malloc' behavior, except that actual System V releases (probably) didn't behave this way unless you explicitly used the fast malloc instead of the standard one.
(Apparently AIX may behave this way, eg, and it's old enough to have influenced POSIX and C. But I suspect that AIX got this behavior by making the System V fast malloc their only malloc, possibly when the SVID nominally required this behavior. AIX may have wound up weird but IBM didn't write it from scratch.)
When I read all of this today and considered what POSIX had done, one of my thoughts was about non-Unix C compilers (partly because I'd recently heard about the historical Whitesmiths C compiler source code being released). C was standardized at a time when C was being increasingly heavily used on various personal computers, including in environments that were somewhat hostile to it, and also other non-Unix environments. These C implementations used their own standard libraries, including malloc(), so maybe they had adopted the NULL return behavior.
As far as I can tell, Whitesmiths' malloc() doesn't have this behavior (also). However, I did find this in the MS-DOS version of Manx Aztec C, or at least it's in version 5.2a; the two earlier versions also available have a simpler malloc() that always rounds up, like the Whitesmiths malloc(). My memory is that you could get the Manx Aztec C compiler for the Amiga with library source, but I'm not particularly good at poking around the Amiga image available so I was unable to spot it if it's included in that version, and I haven't looked at the other Aztec C versions.
(I wouldn't be surprised if a number of 1980s non-Unix C compilers had this behavior, but I don't know where to find good information on this. If someone has written a comprehensive history page on malloc(0) that covers non-Unix C compilers, I haven't found it.)
On systems with small amounts of memory, one reason to specifically make your malloc() return NULL for 0-sized allocations is to reduce memory usage if someone makes a number of such allocations through some general code path that deals with variable-sized objects. Otherwise you'd have to consume some minimum amount of memory even for these useless allocations.
2025-06-05
You could automate (some) boilerplate Go error handling with a formatter
Recently I read [ On ¦ No ] syntactic support for error handling, where the Go team have decided that they're not going to do anything with error handling (contrary to my fears). One part of it sparked a thought about making it less annoying to write basic, boilerplate error handling, and I briefly said something on the Fediverse:
A thought on Go error handling: you could reduce the annoyance of writing the boiler plate by creating a code formatter (along the lines of goimports) that added the standard 'if (err != nil) {...}' stuff to any unhandled 'err' return in your code. Then you'd write code with just 'a, err := Thing(); b, err := Thing2(a)' and so on, and the code formatter would fill in things for you.
This thought was inspired by the model of goimports; my impression
is that almost everyone uses goimports to automatically update import
statements as part of (auto-)formatting their code (certainly I do). If
goimports can update imports as part of 'formatting', then in theory we
can extend it to add boilerplate 'check and return' error handling. This
would take a set of code like this:
a, err := Thing() b, err := Thing2(a) return Thing3(b)
This code would be turned into:
a, err := Thing() if (err != nil) { return <zero value>, err } b, err := Thing2(a) if (err != nil) { return <zero value>, err } return Thing3(b)
(Here by '<zero value>' I mean some suitable zero value, like 'nil', not the literal text '<zero value>'.)
This would only happen if the error return was assigned to a variable
and then the variable was unused (before potentially being reassigned).
If you wrote 'a, _ := Thing()
', there would be no error check
added, and if you re-ran the formatter on the post-formatter code
it wouldn't do anything because all of the error variables are used.
Determining if the error variable was used would need some control
flow analysis, and determining what to return and the syntax of
creating appropriate zero values for non-error return values would
take some function signature and type analysis.
(As an extra trick, if you had an 'if (err != nil) {return ...}' block already, the formatter could copy the 'return ...' into the blocks it added.)
A Go code formatter isn't the only place you could implement this feature. These days many people would be able to use it if it was a 'code action' in a LSP server such as gopls, which already supports other code actions. As a LSP server code action, you could easily apply it selectively only to a function (or a section of a function), rather than having to trust the formatter to run over your entire file. You could also have a LSP server code action that added the boilerplate error check immediately after a call, so you'd write the function call line and then pick 'add boilerplate error check' at the end.
I don't know if this is a good idea, either as a standalone formatter or as a LSP code feature. I'm not a big user of gopls code actions (I mostly ignore them) and I'm not sure I'd be happy writing code that looked like the starting point. But perhaps some people would be, especially if it was an 'expand this function call to have boilerplate error checks' in gopls.
(I don't write enough Go code to have strong feelings about the boilerplate error handling. At my scale, hand-writing the standard 'if (err != nil)' stuff is fine, and I not infrequently want to do something with the error.)
2025-05-26
My GNU Emacs settings for the vertico package (as of mid 2025)
As covered in my Emacs packages, vertico is one of the third party Emacs packages that I have installed to modify how minibuffer completion works for me, or at least how it looks. In my experience, vertico took a significant amount of customization before I really liked it (eventually including some custom code), so I'm going to write down some notes about why I made various settings.
Vertico itself is there to always show me a number of the completion targets, as a help to narrowing in on what I want; I'm willing to trade vertical space during completion for a better view of what I'm navigating around. It's not the only way to do this (there's fido-vertical-mode in standard GNU Emacs, for example), but it's what I started with and it has a number of settings that let me control both how densely the completions are presented (and so how many of them I get to see at once) and how they're presented.
The first thing I do with vertico is override its key binding for TAB, because I want standard Emacs minibuffer tab completion, not vertico's default behavior of inserting the current thing completion is currently on. Specifically, my key bindings are:
:bind (:map vertico-map ("TAB" . minibuffer-complete) ;; M-v is taken by vertico ("M-g M-c" . switch-to-completions) ;; Original tab binding, which we want sometimes when ;; using orderless completion. ("M-TAB" . vertico-insert))
I normally work by using regular tab completion and orderless's completion until I'm happy, then hitting M-TAB if necessary and then RET. I use M-g M-c so rarely that I'd forgotten it until writing this entry. Using M-TAB is especially likely for a long filename completion, where I might use the cursor keys (or theoretically the mouse) to move vertico's selection to a directory and then hit M-TAB to fill it in so I can then tab-complete within it.
Normally, vertico displays a single column of completion candidates, which potentially leaves a lot of wasted space on the right; I use marginalia to add information some sorts of completion targets (such as Emacs Lisp function names) in this space. For other sorts of completions where there's no particular additional information, such as MH-E mail folder names, I use vertico's vertico-multiform-mode to switch to a vertico-grid so I fill the space with several columns of completion candidates and reduce the number of vertical lines that vertico uses (both are part of vertico's extensions).
(I also have vertico-mouse enabled when I'm using Emacs under X, but in practice I mostly don't use it.)
Another important change (for me) is to turn off vertico's default behavior of remembering the history of your completions and putting recently used entries first in the list. This sounds like a fine idea, but in practice I want my completion order to be completely predictable and I'm rarely completing the same thing over and over again. The one exception is my custom MH-E folder completion, where I do enable history because I may be, for example, refiling messages into one of a few folders. This is done through another extension, vertico-sort, or at least I think it is.
(When vertico is installed as an ELPA or MELPA package and then use-package'd, you apparently get all of the extensions without necessarily having to specifically enable them and can just use bits from them.)
My feeling is that effective use of vertico probably requires this sort of customization if you regularly use minibuffer completion for anything beyond standard things where vertico (and possibly marginalia) can make good use of all of your horizontal space. Beyond what key bindings and other vertico behavior you can stand and what behavior you have to change, you want to figure out how to tune vertico so that it's significantly useful for each thing you regularly complete, instead of mostly showing you a lot of empty space and useless results. This is intrinsically a relatively personal thing.
PS: One area where vertico's completion history is not as useful as it looks is filename completion or anything that looks like it (such as standard MH-E folder completion). This is because Emacs filename completion and thus vertico's history happens component by component, while you probably want your history to give you the full path that you wound up completing.
PPS: I experimented with setting vertico-resize
, but found that
the resulting jumping around was too visually distracting.
2025-05-13
Using awk to check your script's configuration file
Suppose, not hypothetically, that you have a shell script with a relatively simple configuration file format that people can still accidentally get wrong. You'd like to check the configuration file for problems before you use it in the rest of your script, for example by using it with 'join' (where things like the wrong number or type of fields will be a problem). Recently on the Fediverse I shared how I was doing this with awk, so here's a slightly more elaborate and filled out version:
errs=$(awk ' $1 ~ "^#" { next } NF != 3 { printf " line %d: wrong number of fields\n", NR; next } [...] ' "$cfg_file" ) if [ -n "$errs" ]; then echo "$prog: Errors found in '$cfg_file'. Stopping." 1>&2 echo "$errs" 1>&2 exit 1 fi
(Here I've chosen to have awk's diagnostic messages indented by one space when the script prints them out, hence the space before 'line %d: ...'.)
The advantage of having awk simply print out the errors it detects and letting the script deal with them later is that you don't need to mess around with awk's exit status; your awk program can simply print what it finds and be done. Using awk for the syntax checks is handy because it lets you express a fair amount of logic and checks relatively simply (you can even check for duplicate entries and so on), and it also gives you line numbers for free.
One trick with using awk in this way is to progressively filter
things in your checks (by skipping further processing of the current
line with 'next
'). We start out by skipping all comments, then
we report and otherwise skip every line with the wrong number of
fields, and then every check after this can assume that at least
we have the right number of fields so it can confidently check what
should be in each one. If the number of fields in a line is wrong
there's no point in complaining about how one of them has the wrong
sort of value, and the early check and 'next
' to skip the rest
of this line's processing is the simple way.
If you're also having awk process the configuration file later you might be tempted to have it check for errors at the same time, in an all-in-one awk program, but my view is that it's simpler to split the error checking from the processing. That way you don't have to worry about stopping the processing if you detect errors or intermingle processing logic with checking logic. You do have to make sure the two versions have the same handling of comments and so on, but in simple configuration file formats this is usually easy.
(Speaking from personal experience, you don't want to use '$1 == "#"' as your comment definition, because then you can't just stick a '#' in front of an existing configuration file line to comment it out. Instead you have to remember to make it '# ', and someday you'll forget.)
PS: If your awk program is big and complex enough, it might make more sense to use a here document to create a shell variable containing it, which will let you avoid certain sorts of annoying quoting problems.
2025-05-05
I moved my local Firefox changes between Git trees the easy way
Firefox recently officially switched to Git, in a completely different Git tree than their old mirror. This presented me a little bit of a problem because I have a collection of local changes I make to my own Firefox builds, which I carry as constantly-rebased commits on top of the upstream Firefox tree. The change in upstream trees meant that I was going to have to move my commits to the new tree. When I wrote my first entry I thought I might try to do this in some clever way similar to rebasing my own changes on top of something that was rebased, but in the end I decided to do it the simple and brute force way that I was confident would either work or would leave me in a situation I could back out from easily.
This simple and brute force way was to get both my old tree and my
new 'firefox' tree
up to date, then export my changes with 'git format-patch
' from the old tree and
import them into the new tree with 'git am
'. There were a few irritations
along the way, of course. First I (re)discovered that 'git am
'
can't directly consume the directory of patches you create with
'git format-patch
'.
Git-am will consume a Maildir of patches, but git-format-patch will
only give you a directory full of files with names like
'00NN-<author>-<title>.patch', which is not a proper Maildir. The
solution is to cat all of the .patch files together in order to
some other file, which is now a mailbox that git-am will handle.
The other minor thing is that git-am unsurprisingly has no 'dry-run'
option (which would probably be hard to implement). Of course in
my situation, I can always reset 'main' back to 'origin/main', which
was one reason I was willing to try this.
(Looking at the 'git format-patch' manual page suggests that what I might have wanted was the '--stdout' option, which would have automatically created the mbox format version for me. On the other hand it was sort of nice to be able to look at the list of patches and see that they were exactly what I expected.)
On the one hand, moving my changes in this brute force way (and to a completely separate new tree) feels like giving in to my unfamiliarity with git. There are probably clever git ways to do this move in a single tree without having to turn everything into patches and then apply them (even if most of that is automated). On the other hand, this got the job done with minimal hassles and time consumed, and sometimes I need to put a stop to my programmer's urge to be clever.
2025-04-30
Being reminded that Git commits are separate from Git trees
Firefox's official source repository has moved to Git, but to a completely new Git repository, not the Git mirror that I've used for the past few years. This led me to a lament on the Fediverse:
This is my sad face that Firefox's switch to using git of course has completely different commit IDs than the old not-official gecko-dev git repository, meaning that I get to re-clone everything from scratch (all ~8 GB of it). Oh well, so it goes in the land of commit hashes.
Then Tim Chase pointed out something that I should have thought of:
If you add the new repo as a secondary remote in your existing one and pull from it, would it mitigate pulling all the blobs (which likely remain the same), limiting your transfer to just the commit-objects (and possibly some treeish items and tags)?
Git is famously a form of content-addressed storage, or more specifically a tree of content addressed storage, where as much as possible is kept the same over time. This includes all the portions of the actual source tree. A Git commit doesn't directly include a source tree; instead it just has the hash of the source tree (well, its top level, cf).
What this means is that if you completely change the commits so that all of them have new hashes, for example by rebuilding your history from scratch in a new version of the repository, but you keep the actual tree contents the same in most or all of the commits, the only thing that actually changes is the commits. If you add this new repository (with its new commit history) as a Git remote to your existing repository and pull from it, most or all of the tree contents are the same across the two sets of commits and won't have to be fetched. So you don't fetch gigabytes of tree contents, you only fetch megabytes (one hopes) of commits.
As I mentioned on the Fediverse, I was told this too late to save me from re-fetching the entire new Firefox repository from scratch on my office desktop (which has lots of bandwidth). I may yet try this on my home desktop, or alternately use it on my office desktop to easily move my local changes on top of the new official Git history.
(I think this is effectively rebasing my own changes on top of
something that's been rebased, which I've done
before, although not recently. I'll also want to refresh my
understanding of what 'git rebase
' does.)
2025-04-27
The glass box/opaque box unit testing argument in light of standards
One of the traditional divides in unit testing is whether you should write 'glass box' or 'opaque box' tests (like GeePawHill I think I prefer those terms to the traditional ones), which is to say whether you should write tests exploiting your knowledge of the module's code or without it. Since I prefer testing inside my modules, I'm implicitly on the side of glass box tests; even if I'm testing public APIs, I write tests with knowledge of potential corner cases. Recently, another reason for this occurred to me, by analogy to standards.
I've read about standards (and read the actual standards) enough by now to have absorbed the lesson that it is very hard to write a (computer) standard that can't be implemented perversely. Our standards need good faith implementations and there's only so much you can do to make it hard for people implementing them in bad faith. After that, you have to let the 'market' sort it out (including the market of whether or not people want to use perverse implementations, which generally they don't).
(Of course some time the market doesn't give you a real choice. Optimizing C compilers are an example, where your only two real options (GCC and LLVM) have aggressively exploited arguably perverse readings of 'undefined behavior' as part of their code optimization passes. There's some recent evidence that this might not always be worth it [PDF], via.)
If you look at them in the right way, unit tests are also a sort of standard. And like standards, opaque box unit tests have a very hard time of completely preventing perverse implementations. While people usually don't deliberately create perverse implementations, they can happen by accident or by misunderstandings, and there can be areas of perverse problems due to bugs. Your cheapest assurance that you don't have a perverse implementation is to peer inside and then write glass box tests that in part target the areas where perverse problems could arise. If you write opaque box tests, you're basically hoping that you can imagine all of the perverse mistakes that you'll make.
(Some things are amenable to exhaustive testing, but usually not very many.)
PS: One way to get perverse implementations is 'write code until all of the tests pass, then stop'. This doesn't guarantee a perverse implementation but it certainly puts the onus on the tests to force the implementation to do things, much like with standards (cf).
2025-04-05
My pessimism about changes to error handling in Go (but they'll happen)
I've said in the past that Go is not our language, and I still stand by that. At the same time, the Go developers do eventually respond to the clamour from the community, which I maintain that we've seen with both Go's eventual addition of generics and the change to Go modules and Go dependency handling (where Go started with one story until it clearly didn't work and they had to change). This leads me to two related views.
First, I think that changes to Go's error handling are inevitably coming sooner or later. Error handling is something the community keeps being unhappy about (even though some people are fine with the current situation), and we know that some people in the core team have written up ideas (via, also). This issue is on the radar, and because it's such a popular issue, I think that change is inevitable.
At the same time, I'm not optimistic about that change, because I don't think error handling is a solved problem. We have a relatively good understanding of things like generics and dependency management, but we don't have a similar understanding of 'good error handling' that can drive a good new implementation in Go. It's possible that the Go developers will find something great, but I think it's more likely that what we'll get is a change that comes with its own set of drawbacks (although it'll be better overall than the current approach).
Go is slowly but steadily becoming a more and more complicated language, and a new method of error handling will inevitably add to that complexity. Also, as I once wrote about an earlier error handling proposal (and another one), a change in error handling will inevitably change how Go is written. People will be pushed to write code that works well with the new error handling mechanism and some number of people will use it for nominally clever tricks, because that's what happens with any language feature.
All of this leaves me feeling somewhat pessimistic about any error handling changes to Go. The current situation isn't ideal, but at least the language is kept simple. Given that error handling isn't a solved problem, I'm not sure any error handling change will improve things enough to make up for its other effects.
2025-04-01
Getting a (vague) understanding of error handling in Rust
When I wrote about how error handling isn't a solved problem, I said some things about Rust's error handling that were flat out wrong, which I had in my mind through superstition. Today is a brief correction on that, since I looked it up.
Rust's usual way of signalling (recoverable) errors
is to use the Result
type,
which is an enum
with one option for errors and one option for success (so it is the
Go 'result, err := call(...)' pattern where only one of result
and err
can be valid at once, and you have to check before using
either). The verbose way of handling this is to explicitly match
and handle each option.
However, often you're only going to propagate an error,
and Rust has special syntax for that in the '?
' operator, which
immediately propagates an error return and otherwise continues:
fn read_username_from_file() -> Result<String, io::Error> { let mut username_file = File::open("hello.txt")?; [...]
Rust's '?' operator can be used in any function with a compatible
return type.
This includes main()
if you declare it appropriately, so you can
use error propagation with '?' as your only way of handling errors
all through your program if you want. The result will probably be
a little bit mysterious since people won't get any specific message
on error, just a non-zero exit status, but for quick programs I can
see the appeal of doing this all the way up through main().
(This makes the '?' operator a far less verbose equivalent of the common Go idiom of 'r, err := ...; if err != nil {return ..., err}'. The '...' will vary depending on the function's return type.)
The other approach is to panic on error with .unwrap(), if you're okay with a basic panic, or .expect(), if you want to provide some sort of diagnostic message to explain a bit about the problem. Although Rust people will probably twitch at this comparison, it feels to me like using .unwrap() is the equivalent of a Python program that does nothing to catch any exceptions (and so winds up with the default stack backtrace), while .expect() is the equivalent of a Python try/except block that prints some sort of a message before exiting.
Since both of these approaches are using Result, you can combine them in quick utility programs. You can propagate errors upward through most of your code, then .expect() on high level operations in main() or functions directly below it to provide some information if things go wrong.
(As a sysadmin, I'm used to the idea of writing quick and rough programs that are run by hand, used only rarely, and operate in environments where they almost never expect to fail. These programs often can get away with minimal error handling, but if things do go wrong it's handy to have some idea of roughly what and where.)
Obviously, what I'd vaguely remembered was a common usage of .unwrap(). I think the wires got crossed in my mind because more recent Rust code I've seen uses '?' a lot, so I sort of vaguely crossed the two in my mind.