Wandering Thoughts

2017-04-27

Understanding Git's model versus understanding its magic

In a comment on my entry on coming to a better understanding of what git rebase does, Ricky suggested I might find Understanding Git Conceptually to be of interest. This provides me with an opportunity to talk about what I think my problem with mastering Git is.

It's worth quoting Charles Duan here:

The conclusion I draw from this is that you can only really use Git if you understand how Git works. Merely memorizing which commands you should run at what times will work in the short run, but it’s only a matter of time before you get stuck or, worse, break something.

I actually feel that I have a relatively good grasp of the technical underpinnings of Git, what many people would call 'how Git works'. To wave my hands a bit, Git is a content addressable store that is used to create snapshots of trees, which are then threaded together in a sequence with commits, and so on and so forth. This lets me nod and go 'of course' about any number of apparently paradoxical things, such as git repositories with multiple initial commits. I don't particularly have this understanding because I worked for it; instead, I mostly have it because I happened to be standing around in the right place at the right time to see Git in its early days.

(There are bits of git that I understand less about the technicalities, like the index. I have probably read a description of the guts of the index at least a few times, but I couldn't tell you off the top of my head how even a simple version of the index works at a mechanical level. It turns out to be covered in this StackOverflow answer; the short version is that the index is a composite of a directory file and a bunch of normal object blobs.)

But in practice Git layers a great deal of magic on top of this technical model of its inner workings. Branches are references to commits (ie, heads) and git advances the reference when you make commits under the right circumstances; simple. Except that some branches have 'upstreams' and are 'remote tracking branches' and so on. All of these pieces of magic are not intrinsic to the technical model (partly because the technical model is a strictly local one), but they are very important for working with Git in many real situations.

It is this magic that I haven't mastered and internalized. For example, I understand what 'git fetch' does to your repository, and I can see why you would want it to update certain branch references so you can find the newly imported commits. But I have to think about why 'git fetch' will update certain branches and not others, and I don't know off the top of my head the settings that control this or how you change them.

It's possible that Git has general patterns in this sort of magic, the way it has general patterns at its technical level. If it does, I have not yet understood enough of the magic to have noticed the general patterns. My personal suspicion is that general patterns do not necessarily exist at this layer, because the commands and operations I think of as part of this layer are actually things that have accreted into Git over time and were written by different people.

(At one point Git had a split between 'porcelain' and 'plumbing', where porcelain was the convenient user interface and was at least partially developed by different people than the core 'plumbing'. And bits of porcelain were developed by different people who had their own mental models for how their particular operation should behave, with git rebase's lack of an option for the branch name of the result being an example.)

In a way my understanding of Git's internals has probably held me back with Git in general, because it's helped to encouraged me to have a lackadaisical attitude about learning Git in general. The result is that I make little surgical strikes on manpages and problems, and once I feel I've solved them well enough I go away again. In this I've been mirroring one of the two ways that I approach new programming languages. I've likely reached the point in Git where I should switch over to thoroughly slogging through some parts of it; one weakness that's become obvious in writing this entry is basically everything to do with remote repositories.

programming/GitCoreVersusMagic written at 01:10:48; Add Comment

2017-04-26

Coming to a better understanding of what git rebase does

Although I've used it reasonably regularly, git rebase has so far been a little bit magical to me, as you may be able to tell from my extensive explanation to myself of using it to rebase changes on top of an upstream rebase. In my grand tradition, I'm going to write down what I hope is a better understanding of what it does and how its arguments interact with that.

What git rebase does is that it takes a series of commits, replays them on top of some new commit, and then gives the resulting top commit a name so that you can use it. When you use the three argument form with --onto, you are fully specifying all of these. Take this command:

git rebase --onto muennich/master old-muennich master

--onto names the new commit everything will be put onto (usually it's a branch, as it is here), the series of commits that will be replayed is old-muennich..master, and the new name is also master. You don't get a choice about the new name; git rebase always makes your new rebase into your branch, discarding the old value of the branch.

(As far as I can tell there's no technical reason why git rebase couldn't let you specify the branch name of the result; it's just not in the conceptual model the authors have of how it should work. If you need this, you need to manually create a new branch beforehand.)

The minimal version has no arguments:

git rebase

This only works on branches with an upstream. It replays your commits from the current branch on top of the current (ie new) upstream, and it determines the range of commits to rebase roughly by finding the closest common ancestor of your commits and the upstream:

A -> B -> C -> D               [origin/master]
      \-> local-1 -> local-2   [master]

In this bad plain text diagram, the upstream added C and D while you have local-1 and local-2. The common point is B, and so B..master describes the commits that will be put on top of origin/master and then your master branch will be switched to them (well, the new version of them).

A rebase is conceptually a push to cherry-pick's pull. In cherry picking, you start on the new clean branch and pull in changes from elsewhere. In rebasing, you start on your 'dirty' local branch and push its changes on top of some other (clean) branch. You then keep the name of your local branch but not its old origin point.

If you use the one or two argument form of git rebase, you're explicitly telling rebase what to consider the 'upstream' for both determining the common ancestor commit and for what to put your changes on top of. If I'm understanding this correctly, the following commands are both equivalent to a plain 'git rebase' on your master branch:

git rebase origin/master
git rebase origin/master master

Based on the diagrams in the git-rebase manpage, it looks like the one and two argument forms are most useful for cases where you have multiple local branches and want to shuffle around the relationship between them.

In general the git-rebase manpage has helpful examples combined with extensive ASCII diagrams. If I periodically read it carefully whenever I'm confused, it will probably all sink in eventually.

(Of course, the git manual page that I actually should read carefully several times until it all sinks in and sticks is the one on specifying revisions and ranges for Git. I sort of know what a number of the different forms mean, but in practice it's one part folklore to one part actual knowledge.)

programming/GitRebaseUnderstanding written at 02:01:28; Add Comment

2017-04-24

What we need from an Illumos-based distribution

It started on Twitter:

@thatcks: Current status: depressing myself by looking at the status of Illumos distributions. There's nothing else like OmniOS out there.

@ptribble: So what are the values of OmniOS that are important to you and you feel aren't present in other illumos distributions?

This is an excellent question from Peter Tribble of Tribblix and it deserves a more elaborate answer than I could really give it on Twitter. So here's my take on what we need from an Illumos distribution.

  • A traditional Unix server environment that will do NFS fileservice with ZFS, because we already have a lot of tooling and expertise in running such an Illumos-based environment and our current environment has a very particular setup due to local needs. If we have to completely change how we design and operate NFS fileservice (for example to move to a more 'storage appliance' environment), the advantages of continuing with Illumos mostly go away. If we're going to have to drastically redesign our environment no matter what, maybe the simpler redesign is 'move to ZFS on FreeBSD doing NFS service as a traditional Unix server'.

  • A distribution that is actively 'developed', by which I mean that it incorporates upstream changes from Illumos and other sources on a regular basis and the installer is updated as necessary and so on. Illumos is not a static thing; there's important evolution in ZFS fixes and improvements, hardware drivers, and various other components.

  • Periodic stable releases that are supported with security updates and important bug fixes for a few years but that do not get wholesale rolling updates from upstream Illumos. We need this because testing, qualifying, and updating to a whole new release (with a wide assortment of changes) is a very large amount of work and risk. We can't possibly do it on a regular basis, such as every six months; even once a year is too much. Two years of support is probably our practical minimum.

    We could probably live with 'security updates only', although it'd be nice to be able to get high priority bug fixes as well (here I'm thinking of things that will crash your kernel or cause ZFS corruption, which are very close to 'must fix somehow' issues for people running ZFS fileservers).

    We don't need the ability to upgrade from stable release to stable release. For various reasons we're probably always going to upgrade by installing from scratch on new system disks and then swapping things around in downtimes.

  • An installer that lets us install systems from USB sticks or DVD images and doesn't require complicated backend infrastructure. We're not interested in big complex environments where we have to PXE-boot initial installs of servers, possibly with various additional systems to auto-configure them and tell them what to do.

In a world with pkgsrc and similar sources of pre-built and generally maintained packages, I don't think we need the distribution itself to have very many packages (I'm using the very specific 'we' here, meaning my group and our fileservers). Sure, it would be convenient for us if the distribution shipped with Postfix and Python and a few other important things, but it's not essential the way the core of Illumos is. While it would be ideal if the distribution owned everything important the way that Debian, FreeBSD, and so on do, it doesn't seem like Illumos distributions are going to have that kind of person-hours available, even for a relatively small package collection.

With that said I don't need for all packages to come from pkgsrc or whatever; I'm perfectly happy to have a mix of maintained packages from several sources, including the distribution in either their main source or an 'additional packages we use' repository. Since there's probably not going to be a plain-server NFS fileserver focused Illumos distribution, I'd expect any traditional Unix style distribution to have additional interests that lead to them packaging and maintaining some extra packages, whether that's for web service or X desktops or whatever.

(I also don't care about the package format that the distribution uses. If sticking with IPS is the easy way, that's fine. Neither IPS nor pkgsrc are among my favorite package management systems, but I can live with them.)

Out of all of our needs, I expect the 'stable releases' part to be the biggest problem. Stable releases are a hassle for distribution maintainers (or really for maintainers of anything); you have to maintain multiple versions and you may have to backport a steadily increasing amount of things over time. The amount of pain involved in them is why we're probably willing to live with only security updates for a relatively minimal set of packages and not demand backported bugfixes.

(In practice we don't expect to hit new bugs once our fileservers have gone into production and been stable, although it does happen every once in a while.)

Although 10G Ethernet support is very important to us in general, I'm not putting it on this list because I consider it a general Illumos issue, not something that's going to be specific to any particular distribution. If Illumos as a whole has viable 10G Ethernet for us, any reasonably up to date distribution should pick it up, and we don't want to use a distribution that's not picking those kind of things up.

Sidebar: My current short views on other Illumos distributions

Peter Tribble also asked what was missing in existing Illumos distributions. Based on an inspection of the Illumos wiki's options, I can split the available distributions into three sets:

  • OpenIndiana and Tribblix are developed and more or less traditional Unix systems, but don't appear to have stable releases that are maintained for a couple of years; instead there are periodic new releases with all changes included.

  • SmartOS, Nexenta, and napp-it are Illumos based but as far as I can tell aren't in the form of a traditional Unix system. (I'm not sure if napp-it is actively updated, but the other two are.)

  • the remaining distributions don't seem to be actively developed and may not have maintained stable releases either (I didn't look deeply).

Hopefully you can see why OmniOS hit a sweet spot for us; it is (or was) actively maintained, it has 'long term stable' releases that are supported for a few years, and you get a traditional Unix OS environment and a straightforward installation system.

solaris/IllumosDistributionNeeds written at 22:18:16; Add Comment

Corebird and coming to a healthier relationship with Twitter

About two months ago I wrote about my then views on the Corebird Twitter client. In that entry I said that Corebird was a great client for checking in on Twitter and skimming through it, but wasn't my preference for actively following Twitter; for that I still wanted Choqok for various reasons. You know what? It turns out that I was wrong. I now feel that Corebird is both a better Linux Twitter client in general and that it's a better Twitter client for me in specific. Unsurprisingly, it's become the dominant Twitter client that I use.

Corebird is mostly a better Twitter client in general because it has much better support for modern Twitter features, even if it's not perfect and there are things from Choqok that I wish it did (even as options). It has niceties like displaying quoted tweets inline and letting me easily and rapidly look at attached media (pictures, animations, etc), and it's just more fluid in general (even if it has some awkward and missing bits, like frankly odd scrolling via the keyboard). Corebird has fast, smooth updates of new tweets more or less any time you want, and it can transparently pull in older tweets as you scroll backwards to a relatively impressive level. Going back to Choqok now actually feels clunky and limited, even though it has features that I theoretically rather want (apart from the bit where I know that several of those features are actually bad for me).

(Corebird's ability to display more things inline makes a surprising difference when skimming Twitter, because I can see more without having to click on links and spawn things in my browser and so on. I also worked out how to make Corebird open up multiple accounts on startup; it's hiding in the per-account settings.)

Corebird is a better Twitter client for me in specific because it clearly encourages me to have a healthier approach to Twitter, the approach I knew I needed a year ago. It's not actually good for me to have a Twitter client open all the time and to try to read everything, and it turns out that Corebird's lack of some features actively encourages me to not try to do this. There's no visible unread count to prod me to pay attention, there is no marker of read versus unread to push me to trying to read all of the unread Tweets one by one, and so on. That Corebird starts fast and lets me skim easily (and doesn't hide itself away in the system tray) also encourages me to close it and not pay attention to Twitter for a while. If I do keep Corebird running and peek in periodically, its combination of features make it easy and natural to skim, rapidly scan, or outright skip the new tweets, so I'm pretty sure I spend less time catching up than I did in Choqok.

(Fast starts matter because I know I can always come back easily if I really want to. As I have it configured, Choqok took quite a while to start up and there were side effects of closing it down with unread messages. In Corebird, startup is basically instant and I know that I can scroll backwards through my timeline to where I was, if I care enough. Mostly I don't, because I'm looking at Twitter to skim it for a bit, not to carefully read everything.)

The net result is that Corebird has turned checking Twitter into what is clearly a diversion, instead of something to actively follow. I call up Corebird when I want to spend some time on Twitter, and then if things get busy there is nothing to push me to get back to it and maybe I can quit out of it in order to make Twitter be even further away (sometimes Corebird helps out here by quietly crashing). This is not quite the 'stop fooling yourself you're not multitasking here' experience that using Twitter on my phone is, but it feels closer to it than Choqok did. Using Corebird has definitely been part of converting Twitter from a 'try to read it all' experience to a 'dip in and see what's going on' one, and the latter is much better for me.

(It turns out that I was right and wrong when I wrote about how UI details mattered for my Twitter experience. Back then I said that a significantly different client from Choqok would mean that my Twitter usage would have to change drastically. As you can see, I was right about that; my Twitter usage has changed drastically. I was just wrong about that necessarily being a bad thing.)

linux/CorebirdViewsII written at 00:29:40; Add Comment

2017-04-23

How I rebased changes on top of other rebased changes in Git

A while ago I wrote an entry on some git repository changes that I didn't know how to do well. One of them was rebasing my own changes on top of a repository that itself had been rebased; in the comments, Aristotle Pagaltzis confirmed that his Stackoverflow answer about this was exactly what I wanted. Since I've now actually gone through this process for the first time, I want to write down the details for myself, with commentary to explain how and why everything works. Much of this commentary will seem obvious to people who use Git a lot, but it reflects some concerns and confusions that I had at the time.

First, the repositories involved. rc is the master upstream repository for Byron Rakitzis's Unix reimplementation of Tom Duff's rc shell. It is not rebased; infrequent changes flow forward as normal for a public Git repo. What I'm going to call muennich-rc is Bert Münnich's collection of interesting modifications on top of rc; it is periodically rebased, either in response to changes in rc or just as Bert Münnich does development on it. Finally I have my own repository with my own local changes on top of muennich-rc. When muennich-rc rebases, I want to rebase my own changes on top of that rebase.

I start in my own repository, before fetching anything from upstream:

  1. git branch old-me

    This creates a branch that captures the initial state of my tree. It's not used in the rebasing process; instead it's a safety measure so that I can reset back to it if necessary without having to consult something like the git reflog. Because I've run git branch without an additional argument, old-me is equivalent to master until I do something to change master.

  2. git branch old-muennich muennich/master.

    muennich/master is the upstream for muennich-rc. Creating a branch captures the (old) top commit for muennich-rc that my changes are on top of.

    Because both old-me and old-muennich have been created as plain ordinary git branches, not upstream tracking branches, their position won't change regardless of fetching and other changes during the rebase. I'm really using them as bookmarks for specific commits instead of actual branches that I will add commits on top of.

    (I'm sure this is second nature to experienced Git people, but when I made old-muennich I had to pause and convince myself that what commit it referred to wasn't going to change later, the way that master changes when you do a 'git pull'. Yes, I know, 'git pull' does more than 'git fetch' does and the difference is important here.)

  3. git fetch

    This pulls in the upstream changes from muennich-rc, updating what muennich/master refers to to be the current top commit of muennich-rc. It's now possible to do things like 'git diff old-muennich muennich/master' to see any differences between the old muennich-rc and the newly updated version.

    (Because I did git fetch instead of git pull or anything else, only muennich/master changed. In particular, master has not changed and is still the same as old-me.)

  4. git rebase --onto muennich/master old-muennich master

    This does all the work (well, I had to resolve and merge some conflicts). What it means is 'take all of the commits that go from old-muennich to master and rebase them on top of muennich/master; afterward, set the end result to be master'.

    (If I omitted the old-muennich argument, I would be trying to rebase both my local changes and the old upstream changes from muennich-rc on top of the current muennich-rc. Depending on the exact changes involved in muennich-rc's rebasing, this could have various conflicts and bad effects (for instance, reintroducing changes that Bert Münnich had decided to discard). There is a common ancestor in the master rc repository, but there could be a lot of changes between there and here.)

    The local changes that I added to the old version of muennich-rc are exactly the commits from old-muennich to master (ie, they're what would be shown by 'git log old-muennich..master', per the git-rebase manpage), so I'm putting my local commits on top of muennich/master. Since the current muennich/master is the top of the just-fetched new version of muennich-rc, I'm putting my local commits on top of the latest upstream rebase. This is exactly what I want to do; I'm rebasing my commits on top of an upstream rebase.

  5. After the dust has settled, I can get rid of the two branches I was using as bookmarks:

    git branch -D old-me
    git branch -D old-muennich
    

    I have to use -D because as far as git is concerned these branches both have unmerged changes. They're unmerged because these branches have both been orphaned by the combination of the muennich-rc rebase and my rebase.

Because I don't care (much) about the old version of my changes that are on top of the old version of muennich-rc, doing a rebase instead of a cherry-pick is the correct option. Following my realization on cherry-picking versus rebasing, there are related scenarios where I might want to cherry-pick instead, for example if I wasn't certain that I liked some of the changes in the rebased muennich-rc and I might want to fall back to the old version. Of course in this situation I could get the same effect by keeping the two branches after the rebase instead of deleting them.

programming/GitRebaseOnRebase written at 01:02:10; Add Comment

2017-04-21

OmniOS's (suddenly) changed future and us

Today, Robert Treat, the CEO of OmniTI, sent a message to the OmniOS user discussion list entitled The Future of OmniOS. The important core of that email is in two sentences:

Therefore, going forward, while some of our staff may continue contributing, OmniTI will be suspending active development of OmniOS. Our next release, currently in beta, will become the final release from OmniTI. [...]

As Treat's message explains, OmniTI has been basically the only major contributor and organizing force behind OmniOS, and no real outside development community has materialized around it. OmniTI is no longer interested in having things continue this way and so they are opting for what Treat adequately calls 'a radical approach'.

(There is much more to Treat's email and if you use OmniOS you should read it in full.)

For various reasons, I don't expect this attempt to materialize a community to be successful. This obviously means that OmniOS will probably wither away as a viable and usable Illumos distribution, especially if you want to more or less freeze a version and then only have important bug fixes or security fixes ported into it (which is what we want). This obviously presents a problem for us, since we use OmniOS on our current generation fileservers. I think that we'll be okay for the remaining lifetime of the current generation, but that's mostly because we're approaching the point where we should start work on the next generation (which I have somewhat optimistically put at some time in 2018 in previous entries).

(We ran our first generation of fileservers for longer than that, but it wasn't entirely a great experience so we'd like to not push things so far this time around.)

One option for our next generation of fileservers is another Illumos distribution. I have no current views here since I haven't looked at what's available since we picked OmniOS several years ago, but investigating the current Illumos landscape is clearly now a priority. However, I have some broad concerns about Illumos in general and 10G Ethernet support on Illumos in specific, because I think 10G is going to be absolutely essential in our next generation of fileservers (living without 10G in our current generation is already less than ideal). Another option is ZFS on a non-Illumos platform, either Linux or FreeBSD, which seems to be a viable option now and should likely be even more viable in 2018.

(Oracle Solaris 11 remains very much not an option for all sorts of reasons, including my complete lack of faith and trust in Oracle.)

Fortunately we probably don't have to make any sort of decision any time soon. We'll probably have at least until the end of 2017 to see how the OmniOS situation develops, watch changes in other Illumos distributions, and so on. And maybe I will be surprised by what happens with OmniOS, as I actually was with Illumos flourishing outside of Sun Oracle.

(The one thing that could accelerate our plans is some need to buy new fileserver hardware sooner than expected because we have a limited time funding source available for it. But given a choice we'd like to defer hardware selection for a relatively long time, partly because perhaps a SSD-based future can come to pass.)

Sidebar: On us (not) contributing to OmniOS development

The short version is that we don't have the budget (in money or in free staff time) to contribute to OmniOS as an official work thing and I'm not at all interested in doing it on a purely personal basis on my own time.

solaris/OmniOSChangedFuture written at 23:08:06; Add Comment

Link: Rob Landley's Linux Memory Management Frequently Asked Questions

Rob Landley's Linux Memory Management Frequently Asked Questions is what it says in the title (via). As of writing this pointer to it, the current version of the FAQ appears to be from 2014, but much of its information is relatively timeless.

Landley's FAQ is good for a lot more than Linux specific information. It's a good general overview of a number of hardware MMU concepts and general Unix MMU concepts. Things like page tables, TLBs, and even memory mappings are basically universal, and the FAQ has a nice solid explanation of all of them and much more.

(See also Rob Landley's other writing, and his website in general.)

links/LandleyLinuxMMFAQ written at 20:35:13; Add Comment

A surprising reason grep may think a file is a binary file

Recently, 'fgrep THING FILE' for me has started to periodically report 'Binary file FILE matches' for files that are not in fact binary files. At first I thought a stray binary character might have snuck into one file this was happening to, because it's a log file that accumulates data partly from the Internet, but then it happened to a file that is only hand-edited and that definitely shouldn't contain any binary data. I spent a chunk of time tonight trying to find the binary characters or mis-encoded UTF-8 or whatever it might be in the file, before I did the system programmer thing and just fetched the Fedora debuginfo package for GNU grep so that I could read the source and set breakpoints.

(I was encouraged into this course of action by this Stackexchange question and answers, which quoted some of grep's source code and in the process gave me a starting point.)

As this answer notes, there are two cases where grep thinks your file is binary: if there's an encoding error detected, or if it detects some NUL bytes. Both of these sound at least conceptually simple, but it turns out that grep tries to be clever about detecting NULs. Not only does it scan the buffers that it reads for NULs, but it also attempts to see if it can determine that a file must have NULs in the remaining data, in a function helpfully called file_must_have_nulls.

You might wonder how grep or anything can tell if a file has NULs in the remaining data. Let me answer that with a comment from the source code:

/* If the file has holes, it must contain a null byte somewhere. */

Reasonably modern versions of Linux (since kernel 3.1) have some special additional lseek() options, per the manpage. One of them is SEEK_HOLE, which seeks to the nearest 'hole' in the file. Holes are unwritten data and Unix mandates that they read as NUL bytes, so if a file has holes, it's got NULs and so grep will call it a binary file.

SEEK_HOLE is not implemented on all filesystems. More to the point, the implementation of SEEK_HOLE may not be error-free on all filesystems all of the time. In my particular case, the files which are being unexpected reported as binary are on ZFS on Linux, and it appears that under some mysterious circumstances the latest development version of ZoL can report that there are holes in a file when there aren't. It appears that there is a timing issue, but strace gave me a clear smoking gun and I managed to reproduce it in a simple test program that gives me a clear trace:

open("testfile", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0600, st_size=33005, ...}) = 0
read(3, "aaaaaaaaa"..., 32768) = 32768
lseek(3, 32768, SEEK_HOLE)              = 32768

The file doesn't have any holes, yet sometimes it's being reported as having one at the exact current offset (and yes, the read() is apparently important to reproduce the issue).

(Interested parties can see more weirdness in the ZFS on Linux issue.)

linux/GrepBinaryFileReason written at 00:57:12; Add Comment

2017-04-20

The big motivation for a separate /boot partition

In a recent comment on my entry on how there's no point in multiple system filesystems any more, I was asked:

What about old(er) computers stuck with BIOS booting? Wasn't the whole seperate /boot/ partially there to appease those systems?

There have been two historical motivations for a separate /boot filesystem in Linux. The lesser motivation is mismatches between what GRUB understands versus what your system does; with a separate /boot you can still have a root filesystem that is, say, the latest BTRFS format, without requiring a bootloader that understands the latest BTRFS. Instead you make a small /boot that uses whatever basic filesystem your bootloader is happy with, possibly all the way down to ext2.

The bigger motivation has been machines where the BIOS couldn't read data from the entire hard disk. All the stages of the bootloader read data using BIOS services, so all of the data they need had to be within a portion of the disk that the BIOS could reach; in fact, they all had to be within the area reachable by whatever (old and basic) BIOS service the bootloader was using. The first stage of the bootloader is at the start of the disk, so that's no problem, and the second stage is usually embedded shortly after it, which is also no problem. The real problem is things that fully live in the filesystem, like the GRUB grub.cfg menu and especially the kernel and initramfs that the bootloader needed to load into memory in order to boot.

(There have been various BIOS limits over the years (see also), and some of the early ones are rather small.)

If your /boot was part of the root filesystem, you had to make sure that your entire root filesystem was inside the area that the BIOS could read. On old machines with limited BIOSes, this could drastically constrain both the size and position of your entire root filesystem. If you had a (small) separate /boot filesystem, only the /boot filesystem had to be within this limited area of BIOS readable disk space; your root filesystem could spill outside of it without problems. You could make / as big as you wanted and put it wherever you wanted.

(If you care about this, it's not enough to have a separate /boot and to make it small; you need to put it as close to the start of the disk as possible, and it's been traditional to make it a primary partition instead of an extended one. Linux installers may or may not do this for you if you tell them to make a separate /boot filesystem.)

Today this concern is mostly obsolete and has been for some time. Even BIOS MBR only machines can generally boot from anywhere on the disk, or at least anywhere on the disk that the MBR partitions can address (which is anything up to 2 TB). In theory you could get into trouble if you had a HD larger than 2 TB, used GPT partitioning, put your root filesystem partly or completely after the 2 TB boundary, and your bootloader and BIOS didn't use LBA48 sector addressing. However I think that even this is relatively unlikely, given that LBA48 is pretty old by now.

(This was once common knowledge in the Linux world, but that was back in the days when it was actually necessary to know this because you might run into such a machine. Those days are probably at least half a decade ago, and probably more than that.)

linux/WhySeparateBootFS written at 00:19:49; Add Comment

2017-04-19

Some things on how PCs boot the old fashioned BIOS way

A while back I wrote on how PCs boot and hard disk partitioning, which gave a quick description of the original BIOS MBR-based method of booting a PC (in contrast to UEFI booting). It's worth talking about BIOS booting in slightly more detail so that we can understand some constraints on it.

In practice, BIOS MBR booting goes through at least two and often three stages. The first stage loader is started by the BIOS when the BIOS loads the first 512-byte sector from an appropriate hard disk (which gets complicated these days) and then transfers control to it. Generally all that the first stage does is load more code and data from disk, in other words load the second stage. Because the first stage has only a few hundred bytes to work with, most first-stage loaders hard code the disk block locations where they find the second stage, and basically all first stage loaders use BIOS services to read data from disk.

The second stage boot code contains at least the core of the bootloader, and is often fairly comprehensive. For GRUB, I believe the second stage image contains the GRUB shell and menu system, various software RAID and filesystem modules (such as the one that understands ext4), and some other things. For GRUB, the second stage can't boot your system by itself; instead it will proceed onwards to read your grub.cfg, possibly load additional GRUB modules that your grub.cfg tells it to, and then for Linux hopefully finish by loading your selected kernel and initramfs into memory and starting them.

I believe that most second stage bootloaders continue to use BIOS services to read any additional data they need from disk (for GRUB, your grub.cfg, any additional GRUB modules, and then finally the kernel and initramfs). This means that BIOS MBR based bootloaders are subject to the whims and limitations of those BIOS services, which in some situations may be significant.

It's common for bootloaders to read data directly out of your filesystem. This obviously means that they need to be able to understand your filesystem and anything it sits on top of (such as software RAID, btrfs, LUKS, and so on). Since bootloader filesystem and so on support can lag behind kernel support, this can leave you wanting (or needing) a separate /boot that your bootloader can understand.

A certain amount of this is a lot simpler with UEFI booting. UEFI has a concept of files in a (simple) filesystem, real NVRAM variables, and a reasonably rich collection of UEFI services. This means that bootloaders don't have to hardcode disk block locations or have complicated ways of hunting for additional files; instead they can do relatively normal IO. Since UEFI will load an entire UEFI executable for you, it also eliminates the first stage bootloading.

Since UEFI booting is better in a number of ways than BIOS booting, you might imagine that an increasing number of machines are set up to boot that way. Perhaps they are for name-brand desktops that are preinstalled with Windows, but at least for Linux servers I don't believe that that's the case. Part of this is sheer pragmatics; the people putting together installers for Linux distributions and so on know that BIOS MBR booting works on basically everything while UEFI may or may not be supported and may or may not work well. Maybe in five years, server installs will default to UEFI unless you tell them to use BIOS MBR.

tech/PCBIOSBootingStages written at 23:19:51; Add Comment

For me Chrome clearly wins over Firefox on Javascript-heavy websites

For my sins, I periodically use a number of Javascript-heavy websites (some of them are sites where the Javascript is core to the experience). I have similar Javascript enabled environments in both Chrome (in incognito mode) and Firefox, with relatively similar sets of extensions (Firefox version and Chrome version). My consistent experience over the past while is that Chrome is clearly faster on these sites than Firefox, and in particular Firefox often feels actively laggy.

(Years ago I sort of had the same experience on Flickr, but I believe that more or less went away later for a while. It returned a couple of years ago, and I quietly switched from using Firefox on Flickr to mostly using Chrome.)

This result sort of surprises and depresses me (partly because using Chrome has its pains). My understanding is that in theory Firefox and Chrome are usually relatively neck and neck as far as performance goes, with Firefox at least competitive, and that especially on common sites Firefox should not be laggy. There are a number of things that could be causing this for me and not for other people, especially general users. For a start I'm on Linux and using Fedora's build of Firefox instead of the Mozilla build, while I think most performance comparisons are made on Windows or MacOS and use the official Mozilla builds.

(I'm also using a relatively odd Linux environment with relatively modest OpenGL and compositing support, which might hurt Firefox more than Chrome.)

Beyond that, possibly my core Firefox extensions are slowing down Firefox more than I expect. But if so, well, they're my core extensions for a good reason (and the obvious suspect of NoScript is not entirely the cause, since some of my Firefox usage is without it). What matters to me is the performance of the browsers I've created from Firefox and Chrome, not the performance of the stock versions in some platonic ideal state that I will never use them in. Given that I have one decently performing browser, that's what I'll wind up using for Javascript-heavy sites even if it's not Firefox.

(And I'm using some extensions in Chrome's incognito mode that I would expect to be sort of heavyweight, like uBlock Origin and a mouse gestures extension.)

PS: I care about this partly because I dislike some things Google does with Chrome and partly because I care about Firefox being competitive and good in general. The overall web ecology benefits when we have a real choice in browsers, and part of having a real choice is good performance.

(I also think that Mozilla fundamentally cares more about Linux for Firefox than Google does for Chrome. As a non-Windows, non-Mac user, I remember the days when I was a second class citizen on the web and I would rather like to not go too far back to them.)

web/LinuxChromeFasterJavascript written at 00:05:29; Add Comment

(Previous 11 or go back to April 2017 at 2017/04/17)

Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.