Wandering Thoughts


Link: Linux Load Averages: Solving the Mystery

Brendan Gregg's Linux Load Averages: Solving the Mystery (via, and) is about both the definition and the history of load average calculations in Linux. Specifically:

Load averages are an industry-critical metric – my company spends millions auto-scaling cloud instances based on them and other metrics – but on Linux there's some mystery around them. Linux load averages track not just runnable tasks, but also tasks in the uninterruptible sleep state. Why? I've never seen an explanation. In this post I'll solve this mystery, and summarize load averages as a reference for everyone trying to interpret them.

In the process of doing this, Brendan Gregg goes back to TENEX (including its source code) for the more or less original load average. Then he chases down the kernel patch from October 1993 that changed Linux's load averages from purely based on the size of the run queue to including processes in disk wait. It goes on from there, including some great examples of how to break down a load average to see what's contributing what (using modern Linux tracing tools, which Gregg is an expert on). The whole thing is really impressive and worth reading.

(Gregg's discussion is focused on Linux alone. For a cross-Unix view, I've written entries on when the load average was added to Unix and the many load averages of different Unix strains. In the latter entry I confidently asserted that Linux's load average included 'disk wait' processes from the start, which Gregg's research has revealed to be wrong.)

LinuxLoadAveragesMystery written at 00:01:14; Add Comment


Link: How does "the" X11 clipboard work?

X11: How does β€œthe” clipboard work? (via) is a technical walk through the modern X11 selection system, one that winds up discussing things at the level of the X protocol and Xlib, with helpful code examples. I learned some quite useful things in the process, for example how to use xclip to query things to find out what formats a selection is available in.

(Technical details about X selections are relevant to me because I use a program that deals with them and which I'd like to see do so more conveniently.)

X11ClipboardDetails written at 12:32:26; Add Comment


Link: NASA DASHlink - Real System Failures

The Observed Failures slide deck from NASA DASHlink (via many places) is an interesting and even alarming collection of observed failures in hardware and software, mostly avionics related. I find it both entertaining and a useful reminder that all of this digital stuff is really analog underneath and that leads to interesting failure modes. Lest you think that all of these were hardware faults and us software people can be smug, well, not really. There are more; read the whole thing, as they say.

NASAObservedFailures written at 22:15:39; Add Comment


Link: ZFS Storage Overhead

ZFS Storage Overhead (via) is not quite about what you might think. It's not about, say, the overhead added by ZFS's RAIDZ storage (where there are surprises); instead it's about some interesting low level issues of where space disappears to even in very simple pools. The bit about metaslabs was especially interesting to me. It goes well with Matthew Ahrens' classic ZFS RAIDZ stripe width, or: How I Learned to Stop Worrying and Love RAIDZ, which is endlessly linked and cited for very good reasons.

ZFSStorageOverhead written at 12:25:07; Add Comment


Link: Survey of [floating point] Rounding Implementations in Go

Matt Jibson's Survey of Rounding Implementations in Go (via) is sort of what it says, namely a survey of a bunch of floating point rounding implementations:

Rounding in Go is hard to do correctly. That is, given a float64, truncate the fractional part (anything right of the decimal point), and add one to the truncated value if the fractional part was >= 0.5. [...]

But it's a lot more than that, because the article then proceeds to demonstrate just how complicated floating point rounding really is (in any language), how non-obvious this complexity is, and how easy it is to get it wrong. It ends with a walk-through of what will be math.Round in Go 1.10, which is implemented not through floating point operations but through direct knowledge of how floating point values are implemented at the bit level.

Floating point is one of those things in programming that looks like it's reasonably simple but instead is a deep pit of potential complexity once you move beyond doing very simple things. I enjoy articles like this because they are both a good reminder of this and a peek behind the curtain.

GoRoundingImplementations written at 20:31:19; Add Comment


Links: Git remote branches and Git's missing terminology (and more)

Mark Jason Dominus's Git remote branches and Git's missing terminology (via) is about what it sounds like. This is an issue of some interest to me, since I flailed around in the same general terminology swamp in a couple of recent entries. Dominus explains all of this nicely, with diagrams, and it helped reinforce things in my mind (and reassure me that I more or less understood what was going on).

He followed this up with Git's rejected push error, which covers a frequent 'git push' issue with the same thoroughness as his first article. I more or less knew this stuff, but again I found it useful to read through his explanation to make sure I actually knew as much as I thought I did.

GitRemoteTermsAndMore written at 22:29:23; Add Comment


Link: The evolution of Unix's overall architecture

Diomidis Spinellis has created a set of great resources for looking at Unix's history. He started with the Unix history rep and has then used that to create block diagrams of Unix's structure in V1 and modern FreeBSD. His article Unix Architecture Evolution Diagrams (via) explains the interesting story of how he put these diagrams together. He also has a site on where manpages appear over Unix versions.

EvolutionUnixArchitecture written at 19:33:14; Add Comment


Link: Rob Landley's Linux Memory Management Frequently Asked Questions

Rob Landley's Linux Memory Management Frequently Asked Questions is what it says in the title (via). As of writing this pointer to it, the current version of the FAQ appears to be from 2014, but much of its information is relatively timeless.

Landley's FAQ is good for a lot more than Linux specific information. It's a good general overview of a number of hardware MMU concepts and general Unix MMU concepts. Things like page tables, TLBs, and even memory mappings are basically universal, and the FAQ has a nice solid explanation of all of them and much more.

(See also Rob Landley's other writing, and his website in general.)

LandleyLinuxMMFAQ written at 20:35:13; Add Comment


Link: Introduction to Certificate Transparency for Server Operators

Alex Gaynor's Introduction to Certificate Transparency for Server Operators (via) is what it says in the title, and taught me some things about certificate transparency in practice. Sadly, one of the things it taught me is that once again Lighttpd seems to be coming up short as far as modern TLS goes. I really should switch over my personal site to using Apache, even if it will kind of be a pain because Fedora fumbled good Apache configuration.

(I also hadn't known about Cert Spotter, which has the advantage that you don't need a Facebook login to use it and thus don't have to helpfully (for Facebook) tie one or more domains to your Facebook login. All you need is an email address and on the modern Internet, you already need a source of individualized disposable ones.)

CertTransForServerOps written at 13:50:48; Add Comment


Link: The Unix Heritage Society now has the 8th, 9th, and 10th editions of Research Unix

Today in an email message with the subject of [TUHS] Release of 8th, 9th and 10th Editions Unix, Warren Toomey announced that the Unix Heritage Society has now gained permission to make the source code of Research Unix's 8th, 9th, and 10th editions available for the usual non-commercial purposes. This apparently is the result of a significant lobbying campaign from a variety of Unix luminaries. The actual source trees can be found in TUHS' archive area for Research distributions.

Most people are familiar with Research Unix versions through V7 (the 7th Edition), which was the famous one that really got out into the outside world and started the Unix revolution. The 8th through 10th editions were what happened inside Bell Labs after this (with a trip through BSD for the port to Vaxen, among other things; see the history of Unix), and because Unix was starting to be commercialized around when they were being worked on by Bell Labs, they were never released in the way that the 7th Edition was. Despite that they were the foundation of some significant innovations, such as the original Streams and /proc, and for various reasons they acquired a somewhat legendary status as the last versions of the true original strain of Research Unix. Which you couldn't see or run, which just added to the allure.

You probably still can't actually run these editions, unless you want to engage in extensive hardware archaeology and system (re)construction. But at least the Unix community now has these pieces of history.

UnixHeritageSocietyUnixV8V9V10 written at 21:50:32; Add Comment

(Previous 10 or go back to February 2017 at 2017/02/02)

Page tools: See As Blogdir, See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.