2018-08-31
Link: A deep dive into the Go memory allocator
Allocator Wrestling is a summary of Eben Freeman's talk from GopherCon 2018 on the Go memory allocator (via, and see also) and its garbage collection system. The slides are here (via) and have more details and elaborations on various things than the livebloged summary, although you probably want to read both (good talks are rarely entirely captured by their slides).
I love seeing under the hood of a complex system this way, and it's probably helped me move towards understanding some things about how much memory Go programs use (or appear to use).
Configurations can quietly drift away from working over time, illustrated
At this point, we've been running various versions of Ubuntu LTS
for over ten years. While we reinstall individual systems when we
move from LTS version to LTS version, we almost never rebuild our
local customizations from scratch unless we're forced to; instead
we carry forward the customizations from the last LTS version, only
changing what seems to need it. This is true both for the configuration
of our systems and also for the configuration of things we build
on top of Ubuntu, such as our user-run web servers. However, one of the hazards of carrying
forward configurations for long enough is that they can silently
drift away from actually working or making sense. For example, you
can set (or try to set) Linux sysctls that don't exist any more
and often nothing will complain loudly enough for you to notice.
Today, I had an interesting illustration of how far this can go
without anything obvious breaking or anyone saying anything.
For our user-run web servers, we supply a set of configurations for Apache, PHP, and MySQL that works out of the box, so users with ordinary needs don't have to muck around with that stuff themselves. Although some people customize their setups (or run web servers other than Apache), most people just use the defaults. In order to make Ubuntu version to Ubuntu version upgrades relatively transparent, most of this configuration is central and maintained by us, instead of being copied to each user's Apache configuration area and so on. This has basically worked over all of the years and all of the Ubuntu LTS versions; generally the only version to version change people have had to do in their user-run web server is to run a magic MySQL database update process. Everything else is handled by us changing the our central configurations.
(I'm quite thankful that both Apache and MySQL have 'include' directives in their configuration file formats. You may also detect that we know very little about operating MySQL.)
One of the things that we customize for user-run web server is the
MySQL settings in PHP, because the stock settings are set up to try
to talk to the system MySQL and we don't run a system MySQL (especially
not one that people can interact with). We do this with a custom
php.ini, and that php.ini is configured in the Apache configuration
in a little .conf snippet. Here is the current one, faithfully
carried forward from no more recently than 2009 and currently running
on our Ubuntu 16.04 web server since the fall of 2016 or so:
<IfModule mod_php5.c> PHPIniDir conf/php.ini </IfModule>
Perhaps you can see the problem.
Ubuntu 16.04 doesn't ship with PHP 5 any more; it only ships with
PHP 7. That makes the IfModule directive here false, which means
that PHP is using its standard system Apache php.ini. For that
matter, I'm not certain this directive was actually working for
Ubuntu 14.04's PHP 5 either.
This means that for at least the past two years or so, people have been operating their user-run web servers without our PHP customizations that are supposed to let their PHP code automatically talk to their MySQL instances. I'm not sure that no one noticed anything but at the very least no one said anything to us about the situation, and I know that plenty of people have user-run web servers with database-driven stuff installed, such as WordPress. Apparently everyone who needed to was able to set various parameters so that they could talk to their MySQL anyway.
(This is probably not surprising, since 'configure your database settings' is likely a standard part of the install process for a lot of software. It does seem to be part of WordPress's setup, for example.)
On the one hand, that this slipped past us is a bit awkward (although understandable; it's not as if this makes PHP not load at all). On the other hand, it doesn't seem to have done any real harm and it means that we can apparently discard our entire php.ini customization scheme and make our lives simpler, since clearly it's not actually necessary in practice.
(I stumbled over this in the process of preparing our user-run webserver system for an upgrade to 18.04. How I noticed it actually involve another bit of quiet configuration drift, although that's story for another entry.)
Making Ubuntu bug reports seems to be useless (or pointless)
I've mentioned this in passing in a few places, so I might as well say it here: I've mostly given up on making Ubuntu bug reports for the simple reason that doing so seems to be useless. Every so often I'll file one for no particularly strong reason (eg, which I'd actually forgotten I'd filed), but even when I do file a bug I usually don't expect anything. When I say that I don't expect anything, I mean more than I don't expect a response to my bug; I mean that I expect that my bug report will have basically no effect not only on the bug in this Ubuntu LTS version but also on whether or not it's in future ones.
(I'm also much less likely to file bugs that might require me to argue with someone, such as over 18.04's packaging of libreadline.)
It's possible that Ubuntu makes use of bug reports for some internal purposes and so filing them is not technically pointless. But from my perspective as an outsider, filing Ubuntu bugs is certainly useless and to me that makes them pointless as well. There really isn't much more to say about the situation than that. Ubuntu can run its bug tracker however it wants to, and it's not being actively hostile to people submitting bug reports in the way that some environments are. It's just that Ubuntu has created a situation where there's no point in submitting bug reports, so I'm mostly not going to bother.
(Of course, Ubuntu has never been a distribution that did very many bugfix updates. A long time ago I wrote a grumpy entry about this lack of such updates, and nothing has changed since then.)
Many Ubuntu packages are inherited more or less untouched from Debian and Debian is generally reasonably responsive to bug reports. It's potentially worth keeping a Debian system around so you can reproduce bugs and submit them to Debian in the hopes that an update will trickle through to at least the next Ubuntu release (or LTS release, if that's the only Ubuntu version you use). You likely need to use Debian 'testing' for this, since it's generally what Ubuntu draws packages from.
(Our Amanda packaging bug was fixed in Debian, for example.)
PS: Occasionally useful discussions do break out in Ubuntu bug reports between a group of people with the problem who are working together to diagnose it and perhaps come up with fixes; I think I've seen one or two. But the odds are that no bug report that I make will spark such a discussion.
(I might feel more motivated to file bug reports so that other people with the problem could find them if Launchpad's search wasn't basically terrible as far as I've seen. If I want other people to be able to find my reports, I'm probably better off writing up things here on Wandering Thoughts and hoping that search engines index them. That's one reason I've taken to putting exact error messages in entries.)