Wandering Thoughts


How recursively flattening a list raises a Python type question

Today I wound up reading Why it's hard for programmers to write a program to flatten a list? (via), where the quiz challenge put forward is to turn an input like [1,[2,3], [4, [5,6]]] into [1,2,3,4,5,6]. My immediate reaction was that I'd do this in Python rather than in any statically typed language I know, because all of them make the input type here hard to represent. But then I realized that doing this in Python raises another type-related question.

If we stick exactly to the specification (and directly implement it), the result is pretty simple and straightforward:

def flatten(inlst):
    olst = []
    for i in inlst:
        if isinstance(i, int):
        elif isinstance(i, list):
            raise ValueError("invalid element in list")
    return olst

(You can optimize this by having a _flatten internal function that gets passed the output list, so you don't have to keep building lists and then merging them into other lists as you work down and then back up the recursion stack. Also, I'm explicitly opting to return an actual list instead of making this a (recursive) generator.)

However, this code is not very Pythonic because it is so very narrowly typed. We can relax it slightly by checking for isinstance(i, (int, float)), but even then most people would say that flatten() should definitely accept tuples in place of lists and probably even sets.

If we're thinking about being Pythonic and general, the obvious thing to do is check if the object is iterable. So we write some simple and general code:

def flatten2(inlst):
    olst = []
    for i in inlst:
            it = iter(i)
        except TypeError:
            it = None
        if it is None:
    return olst

This should flatten any type (or mixture of types) that contains elements, as denoted by the fact that it's iterable. It looks good and passes initial tests. Then some joker calls our code with flatten2(["abcdef",]) and suddenly we have a problem. Then another joker calls our code with flatten2([somedict,]) and files a bug that our code only flattens the keys of their dictionary, not the keys and values.

(As an exercise, can you predict in advance, without trying it, what our problem is with flatten2(["abcdef",]), and why it happens? I got this wrong when I was writing and testing this code in Python 3 and had to scratch my head for a bit before the penny dropped.)

The problem here is that 'is iterable' is not exactly what we want. Some things, such as strings, are iterable but should probably be treated as indivisible by flatten2(). Other things, such as dicts, are iterable but the default iteration result does not fully represent their contents. Really, not only is Python lacking a simple condition for what we want, it's arguably not clear just what we want to do if we're generalizing flatten() (and what making it 'Pythonic' really means).

One valid answer is that we will explicitly check for container types that are close enough to what we want, and otherwise mostly return things as-is. Here we would write a version of flatten() that looked like this:

def flatten3(inlst):
    olst = []
    for i in inlst:
        if isinstance(i, (list, tuple, set)):
        elif isinstance(i, dict):
            raise ValueError("dict not valid in list")
    return olst

We could treat dicts as single elements and just return them, but that is probably not what the caller intended. Still, this check feels dubious, which is a warning sign.

As a minimum, it would be nice to have a Python abstract type or trait that represented 'this is a container object and iterating it returns a full copy of its contents'; you could call this the property of being list-like. This would be true for lists, tuples, and sets, but false for dicts, which would give us a starting point. It would also be true for strings, but you can't win them all; when dealing with iterable things, we'll probably always have to special-case strings.

(I'd go so far as arguing that making strings iterable by default was a Python mistake. It's one of those neat features that winds up getting in the way in practice.)

I don't have an answer here, by the way. If I was in this situation I might either write and carefully document a version of flatten2() (specifying 'recursively flattens any iterable thing using its default iterator; this will probably not do what you want for dicts'), or go with some version of flatten3() that specifically restricted iteration to things that I felt were sufficiently list-like.

(I'd worry about missing some new popular type over time, though. Ten years ago I might not have put set in the list, and who knows what I'm missing today that's going to be popular in Python in the future. Queues? Trees? Efficient numerical arrays?)

python/FlattenTypeQuestion written at 02:09:00; Add Comment


A single email message with quite a lot of different malware

This is the kind of thing where it's easier to show you the log messages first and discuss them later:

1chbMp-0007UF-Jw attachment application/msword; MIME file ext: .doc; zip exts: .rels .xml[3] none
1chbMp-0007UF-Jw attachment application/msword; MIME file ext: .doc; zip exts: .rels .xml[3] none
1chbMp-0007UF-Jw attachment application/msword; MIME file ext: .doc; zip exts: .bin .png .rels .xml[10] none
1chbMp-0007UF-Jw attachment application/msword; MIME file ext: .doc; zip exts: .eps .gif .rels .xml[10] none
1chbMp-0007UF-Jw attachment application/msword; MIME file ext: .doc
rejected 1chbMp-0007UF-Jw from to <redacted>: identified virus: CXmail/OleDl-L2, Troj/20152545-E, Troj/DocDrop-RK
detail 1chbMp-0007UF-Jw Subject: [PMX:SPAM] [PMX:VIRUS] Urgent Order..

That one incoming email message had five different attachments and between them they had at least three different forms of malware. It's possible that all five attachments were bad but with some duplication of malware types, so the report we got only identified the unique malware, especially since the first two attachments have the exact same file extensions.

The origin IP address is in HINET (AS3462, hinet.net), which was a big source of issues back in the days when I actively tracked who was the source of issues. It's not currently listed in the Spamhaus ZEN, but it is on Barracuda's blocklist and psky.me (at their 'defer but don't reject' blocking level). Our logs say it HELO'd as 'mail.synclink.com.tw' and to be relaying the email from (which is on the CBL, as well as psky.me at the 'reject during SMTP' level).

Troj/20152545-E is apparently normally a PostScript file, so I suspect that it was found in the .eps file in the fourth attachment. CXmail/OleDl-L2 is claimed to show up in 'OpenDocument' and Microsoft Office files (see also). Troj/DocDrop-RK is apparently normally seen in RTF files, so who knows where it lurks in this set of MIME attachments.

spam/SingleEmailMuchMalware written at 18:26:47; Add Comment


What an actual assessment of Ubuntu kernel security updates looks like

Ubuntu recently released some of their usual not particularly helpful kernel security update announcements and I tweeted:

Another day, another tedious grind through Ubuntu kernel security announcements to do the assessment that Ubuntu should be doing already.

I have written about the general sorts of things we want to know about kernel security updates, but there's nothing like a specific example (and @YoloPerdiem asked). So here is essentially the assessment email that I sent to my co-workers.

First, the background. We currently have Ubuntu 16.04 LTS, 14.04 LTS, and 12.04 LTS systems, so we care about security updates for the mainline kernels for all of those (we aren't using any of the special ones). The specific security notices I was assessing are USN-3206-1 (12.04), USN-3207-1 (14.04), and USN-3208-1 (16.04). I didn't bother looking at CVEs that require hardware or subsytems that we don't have or use, such as serial-to-USB hardware (CVE-2017-5549) or KVM (several CVEs here). We also don't update kernels just for pure denial of service issues (eg CVE-2016-9191, which turns out to require containers anyway), because our users already have plenty of ways to make our systems crash if they want to.

So here is a slightly edited and cleaned up version of my assessment email:

Subject: Linux kernel CVEs and my assessment of them

16.04 is only affected by CVE-2017-6074, which we've mitigated, and CVE-2016-10088, which doesn't apply to us because we don't have people who can access /dev/sg* devices.

12.04 and 14.04 are both affected by additional CVEs that are use-after-frees. They are not explicitly exploitable so far, but CVE-2017-6074 is also a use-after-free and is said to be exploitable with an exploit released soon, so I think they are probably equally dangerous.

[Local what-to-do discussion elided.]



Andrey Konovalov discovered a use-after-free vulnerability in the DCCP implementation in the Linux kernel. A local attacker could use this to cause a denial of service (system crash) or possibly gain administrative privileges.

This is bad if not mitigated, with an exploit to be released soon (per here), but we should have totally mitigated it by blocking the DCCP modules. See my worklog on that.


Dmitry Vyukov discovered a use-after-free vulnerability in the sys_ioprio_get() function in the Linux kernel. A local attacker could use this to cause a denial of service (system crash) or possibly gain administrative privileges.

Links: 1, 2, 3.

The latter URL has a program that reproduces it, but it's not clear if this can be exploited to do more than crash. But CVE-2017-6074's use-after-free is apparently exploitable, so...


It was discovered that a use-after-free vulnerability existed in the block device layer of the Linux kernel. A local attacker could use this to cause a denial of service (system crash) or possibly gain administrative privileges.

Link: 1

Oh look, another use-after-free issue. Ubuntu's own link for the issue says 'allows local users to gain privileges by leveraging the execution of [...]' although their official release text is less alarming.


It was discovered that the generic SCSI block layer in the Linux kernel did not properly restrict write operations in certain situations. A local attacker could use this to cause a denial of service (system crash) or possibly gain administrative privileges.

Finally some good news! As far as I can tell from Ubuntu's actual CVE-2016-10088 page, this is only exploitable if you have access to a /dev/sg* device, and on our machines people don't.

(The actual email was plain text, so the various links were just URLs dumped into the text.)

As you can maybe see from this, doing a proper assessment requires reading at least the detailed Ubuntu CVE information in order to work out under what circumstances the issue can be triggered, for instance to know that CVE-2016-10088 requires access to a /dev/sg* device. Not infrequently you have to go chasing further; for example, only Andrey Konovalov's initial notice mentions that he will release an exploit in a few days. In this case we could mitigate the issue anyways by blacklisting the DCCP modules, but in other cases 'an exploit will soon be released' drastically raises the importance of a security exposure (at least for us).

The online USN pages usually link to Ubuntu's pages on the CVEs they include, but the email announcements that Ubuntu sends out don't. Ubuntu's CVE pages usually have additional links, but not a full set; often I wind up finding Debian's page on a CVE because they generally have a full set of search links for elsewhere (eg Debian's CVE-2016-9191 page). I find that sometimes the Red Hat or SuSE bug pages will have the most technical detail and thus help me most in understanding the impact of a bug and how exposed we are.

The amount of text that I wind up writing in these emails is generally way out of proportion to the amount of reading and searching I have to do to figure out what to write. Everything here is a sentence or two, but getting to the point where I could write those is the slog. And with CVE-2017-6074, I had to jump in to set up and test an entire mitigation of blacklisting all the DCCP modules via a new /etc/modprobe.d file and then propagating that file around to all of our Ubuntu machines.

linux/UbuntuKernelUpdateAssessment written at 23:26:07; Add Comment

How ZFS bookmarks can work their magic with reasonable efficiency

My description of ZFS bookmarks covered what they're good for, but it didn't talk about what they are at a mechanical level. It's all very well to say 'bookmarks mark the point in time when [a] snapshot was created', but how does that actually work, and how does it allow you to use them for incremental ZFS send streams?

The succinct version is that a bookmark is basically a transaction group (txg) number. In ZFS, everything is created as part of a transaction group and gets tagged with the TXG of when it was created. Since things in ZFS are also immutable once written, we know that an object created in a given TXG can't have anything under it that was created in a more recent TXG (although it may well point to things created in older transaction groups). If you have an old directory with an old file and you change a block in the old file, the immutability of ZFS means that you need to write a new version of the data block, a new version of the file metadata that points to the new data block, a new version of the directory metadata that points to the new file metadata, and so on all the way up the tree, and all of those new versions will get a new birth TXG.

This means that given a TXG, it's reasonably efficient to walk down an entire ZFS filesystem (or snapshot) to find everything that was changed since that TXG. When you hit an object with a birth TXG before (or at) your target TXG, you know that you don't have to visit the object's children because they can't have been changed more recently than the object itself. If you bundle up all of the changed objects that you find in a suitable order, you have an incremental send stream. Many of the changed objects you're sending will contain references to older unchanged objects that you're not sending, but if your target has your starting TXG, you know it has all of those unchanged objects already.

To put it succinctly, I'll quote a code comment from libzfs_core.c (via):

If "from" is a bookmark, the indirect blocks in the destination snapshot are traversed, looking for blocks with a birth time since the creation TXG of the snapshot this bookmark was created from. This will result in significantly more I/O and be less efficient than a send space estimation on an equivalent snapshot.

(This is a comment about getting a space estimate for incremental sends, not about doing the send itself, but it's a good summary and it describes the actual process of generating the send as far as I can see.)

Yesterday I said that ZFS bookmarks could in theory be used for an imprecise version of 'zfs diff'. What makes this necessarily imprecise is that while scanning forward from a TXG this way can tell you all of the new objects and it can tell you what is the same, it can't explicitly tell you what has disappeared. Suppose we delete a file. This will necessarily create a new version of the directory the file was in and this new version will have a recent TXG, so we'll find the new version of the directory in our tree scan. But without the original version of the directory to compare against we can't tell what changed, just that something did.

(Similarly, we can't entirely tell the difference between 'a new file was added to this directory' and 'an existing file had all its contents changed or rewritten'. Both will create new file metadata that will have a new TXG. We can tell the case of a file being partially updated, because then some of the file's data blocks will have old TXGs.)

Bookmarks specifically don't preserve the original versions of things; that's why they take no space. Snapshots do preserve the original versions, but they take up space to do that. We can't get something for nothing here.

(More useful sources on the details of bookmarks are this reddit ZFS entry and a slide deck by Matthew Ahrens. Illumos issue 4369 is the original ZFS bookmarks issue.)

Sidebar: Space estimates versus actually creating the incremental send

Creating the actual incremental send stream works exactly the same for sends based on snapshots and sends based on bookmarks. If you look at dmu_send in dmu_send.c, you can see that in the case of a snapshot it basically creates a synthetic bookmark from snapshot's creation information; with a real bookmark, it retrieves the data through dsl_bookmark_lookup. In both cases, the important piece of data is zmb_creation_txg, the TXG to start from.

This means that contrary to what I said yesterday, using bookmarks as the origin for an incremental send stream is just as fast as using snapshots.

What is different is if you ask for something that requires estimating the size of the incremental sends. Space estimates for snapshots are pretty efficient because they can be made using information about space usage in each snapshot. For details, see the comment before dsl_dataset_space_written in dsl_dataset.c. Estimating the space of a bookmark based incremental send requires basically doing the same walk over the ZFS object tree that will be done to generate the send data.

(The walk over the tree will be somewhat faster than the actual send, because in the actual send you have to read the data blocks too; in the tree walk, you only need to read metadata.)

So, you might wonder how you ask for something that requires a space estimate. If you're sending from a snapshot, you use 'zfs send -v ...'. If you're sending from a bookmark or a resume token, well, apparently you just don't; sending from a bookmark doesn't accept -v and -v on resume tokens means something different from what it does on snapshots. So this performance difference is kind of a shaggy dog story right now, since it seems that you can never actually use the slow path of space estimates on bookmarks.

solaris/ZFSBookmarksMechanism written at 00:26:44; Add Comment


ZFS bookmarks and what they're good for

Regular old fashioned ZFS has filesystems and snapshots. Recent versions of ZFS add a third object, called bookmarks. Bookmarks are described like this in the zfs manpage (for the 'zfs bookmark' command):

Creates a bookmark of the given snapshot. Bookmarks mark the point in time when the snapshot was created, and can be used as the incremental source for a zfs send command.

ZFS on Linux has an additional explanation here:

A bookmark is like a snapshot, a read-only copy of a file system or volume. Bookmarks can be created extremely quickly, compared to snapshots, and they consume no additional space within the pool. Bookmarks can also have arbitrary names, much like snapshots.

Unlike snapshots, bookmarks can not be accessed through the filesystem in any way. From a storage standpoint a bookmark just provides a way to reference when a snapshot was created as a distinct object. [...]

The first question is why you would want bookmarks at all. Right now bookmarks have one use, which is saving space on the source of a stream of incremental backups. Suppose that you want to use zfs send and zfs receive to periodically update a backup. At one level, this is no problem:

zfs snapshot pool/fs@current
zfs send -Ri previous pool/fs@current | ...

The problem with this is that you have to keep the previous snapshot around on the source filesystem, pool/fs. If space is tight and there is enough data changing on pool/fs, this can be annoying; it means, for example, that if people delete some files to free up space for other people, they actually haven't done so because the space is being held down by that snapshot.

The purpose of bookmarks is to allow you to do these incremental sends without consuming extra space on the source filesystem. Instead of having to keep the previous snapshot around, you instead make a bookmark based on it, delete the snapshot, and then do the incremental zfs send using the bookmark:

zfs snapshot pool/fs@current
zfs send -i #previous pool/fs@current | ...

This is apparently not quite as fast as using a snapshot, but if you're using bookmarks here it's because the space saving is worth it, possibly in combination with not having to worry about unpredictable fluctuations in how much space a snapshot is holding down as the amount of churn in the filesystem varies.

(We have a few filesystems that get frequent snapshots for fast recovery of user-deleted files, and we live in a certain amount of concern that someday, someone will dump a bunch of data on the filesystem, wait just long enough for a scheduled snapshot to happen, and then either move the data elsewhere or delete it. Sorting that one out to actually get the space back would require deleting at least some snapshots.)

Using bookmarks does require you to keep the previous snapshot on the destination (aka backup) filesystem, although the manpage only tells you this by implication. I believe that this implies that while you're receiving a new incremental, you may need extra space over and above what the current snapshot requires for space, since you won't be able to delete previous and recover its space until the incremental receive finishes. The relevant bit from the manpage is:

If an incremental stream is received, then the destination file system must already exist, and its most recent snapshot must match the incremental stream's source. [...]

This means that the destination filesystem must have a snapshot. This snapshot will and must match a bookmark made from it, since otherwise incremental send streams from bookmarks wouldn't work.

(In theory bookmarks could also be used to generate an imprecise 'zfs diff' without having to keep the origin snapshot around. In practice I doubt anyone is going to implement this, and why it's necessarily imprecise requires an explanation of why and how bookmarks work.)

solaris/ZFSBookmarksWhatFor written at 23:58:39; Add Comment

Sometimes it can be hard to tell one cause of failure from another

I mentioned recently how a firmware update fixed a 3ware controller so that it worked. As it happens, my experiences with this machine nicely illustrates the idea that sometimes it can be hard to tell one failure from another, or to put it another way, when you have a failure it can be hard to tell what the actual cause is. So let me tell the story of trying to install this machine.

Like many places within universities, we don't have a lot of money, but we do have a large collection of old, used hardware. Rather than throw eg five year old hardware away because it's beyond its nominal service life, we instead keep around anything that's not actively broken (or at least that doesn't seem broken) and press it into use again in sufficiently low-priority situations. One of the things that we have as a result of this is an assorted collection of various sizes of SATA HDs. We've switched over to SSDs for most servers, but we don't really have enough money to use SSDs for everything, especially when we're reconditioning an inherited machine under unusual circumstances.

Or in other words, we have a big box of 250 GB Seagate SATA HDs that have been previously used somewhere (probably as SunFire X2x00 system disks), all of which had passed basic tests when they were put into the box some time ago. When I wanted a pair of system disks for this machine I turned to that box. Things did not go well from there.

One of the disks from the first pair had really slow IO problems, which of course manifested as a far too slow Ubuntu 16.04 install. After replacing the slow drive, the second install attempt ended with the original 'good' drive dropping off the controller entirely, apparently dead. The replacement for that drive turned out to also be excessively slow, which took me up to four 250 GB SATA drives, of which one might be good (and three slow failed attempts to bring up one of our Ubuntu 16.04 installs). At that point I gave up and used some SSDs that we had relatively strong confidence in, because I wasn't sure if our 250 GB SATA drives were terrible or if the machine was eating disks. The SSDs worked.

Before we did the 3ware firmware upgrade and it made other things work great, I would have confidently told you that our 250 GB SATA disks had started rotting and could no longer be trusted. Now, well, I'm not so sure. I'm perfectly willing to believe bad things about those old drives, but were my problems because of the drives, the 3ware controller's issues, or some combination of both? My guess now is on a combination of both, but I don't really know and that shows the problem nicely.

(It's not really worth finding out, either, since testing disks for slow performance is kind of a pain and we've already spent enough time on this issue. I did try the 'dead' disk in a USB disk docking station and it worked in light testing.)

sysadmin/HardToTellFailureCausesApart written at 01:41:50; Add Comment


Some notes on moving a software RAID-1 root filesystem around (to SSDs)

A while ago I got some SSDs for my kind of old home machine but didn't put them to immediate use for various reasons. Spurred on first by the feeling that I should get around to it sometime, before my delay got too embarrassing, and then by one of my system drives apparently going into slow IO mode for a while, I've now switched my root filesystem over to my new SSDs. I've done this process before, but this time around I want to write down notes for my future reference rather than having to re-derive all the steps each time. All of this is primarily for Fedora, currently Fedora 25; some steps will differ on other distributions such as Ubuntu.

I partitioned using GPT partitions, not particularly because I needed to with 750 GB SSDs but because it seemed like a good idea. I broadly copied the partitioning I have on my SSDs at work for no particularly strong reason, which means that I set it up this way:

Number Size Code Name
1 256 MB EF00 EFI System
2 1 MB EF02 BIOS boot partition
3 100 GB FD00 Linux RAID
4 1 GB FD00 Linux RAID (swap)
5 <rest> BF01 ZFS

Some of this is likely superstition by now, such as the BIOS boot partition.

With the pair of SSDs partitioned, I set up the software RAID-1 arrays for the new / and swap. Following my guide to RAID superblock formats I used version 1.0 format for the / array, since I'm going to end up with /boot on it. Having created them as /dev/md10 and /dev/md11 it was time to put them in /etc/mdadm.conf. The most convenient way is to use 'mdadm --examine --scan' and then copy the relevant output into mdadm.conf by hand. Once you have updated mdadm.conf, you also need to update the initramfs version of it by rebuilding the initramfs. Although you can do this for all kernel versions, I prefer to do it only for the latest one so that I have a fallback path if something explodes. So:

dracut --kver $(uname -r) --force

(This complained about a broken pipe for cat but everything seems to have worked.)

When I created the new RAID arrays, I took advantage of an mdadm feature to give them a name with -N; in particular I named them 'ssd root' and 'ssd swap'. It turns out that mdadm --examine --scan tries to use this name as the /dev/ name of the array and the initramfs doesn't like this, so on boot my new arrays became md126 and md127, instead of the names I wanted. To fix that I edited mdadm.conf to give them the proper names, and while I was there I added all of the fields that my other (much older) entries had:

ARRAY /dev/md10  metadata=1.0 level=raid1 num-devices=2 UUID=35d6ec50:bd4d1f53:7401540f:6f971527
ARRAY /dev/md11  metadata=1.2 level=raid1 num-devices=2 UUID=bdb83b04:bbdb4b1b:3c137215:14fb6d4e

(Note that specifying the number of devices may have dangerous consequences if you don't immediately rebuild your initramfs. It's quite possible that Fedora 25 would have been happy without it, but I don't feel like testing. There are only a finite number of times I'm interested in rebooting my home machine.)

After copying my root filesystem from its old home on SATA HDs to the new SSD filesystem, there were a number of changes I need to make to actually use it (and the SSD-based swap area). First, we modify /etc/fstab to use the UUIDs of the new filesystem and swap area for / and, well, swap. The easiest way to get these UUIDs is to use blkid, as in 'blkid /dev/md10' and 'blkid /dev/md11'.

(For now I'm mounting the old HD-based root filesystem on /oldroot, but in the long run I'm going to be taking out those HDs entirely.)

But we're not done, because we need to make some GRUB2 changes in order to actually boot up with the new root filesystem. A normal kernel boot line in grub.cfg looks like this:

linux   /vmlinuz-4.9.9-200.fc25.x86_64 root=UUID=5c0fd462-a9d7-4085-85a5-643555299886 ro acpi_enforce_resources=lax audit=0 SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=us rd.md.uuid=d0ceb4ac:31ebeb12:975f015f:1f9b1c91 rd.md.uuid=c1d99f17:89552eec:ab090382:401d4214 rd.md.uuid=4e1c2ce1:92d5fa1d:6ab0b0e3:37a115b5 rootflags=rw,relatime,data=ordered rootfstype=ext4

This specifies two important things, the UUID of the root filesystem in 'root=...' and the (software RAID) UUIDs of the software RAID arrays that the initramfs should assemble in early boot in the 'rd.md.uid=...' bits (per the dracut.cmdline manpage, and also). We need to change the root filesystem UUID to the one we've already put into /etc/fstab and then add rd.md.uuid= settings for our new arrays. Fortunately mdadm has already reported these UUIDs for us and we can just take them from our mdadm.conf additions. Note that these two UUIDs are not the same; the UUID of a filesystem is different than the UUID of the RAID array that contains it, and one will (probably) not work in the place of the other.

(In the long run I will need to take out the rd.md.uuid settings for the old HD-based root and swap partitions, since they don't need to be assembled in early boot and will actively go away someday.)

The one piece of the transition that's incomplete is that /boot is still on the HDs. Migrating /boot is somewhat more involved than migrating the root filesystem, especially as I'm going to merge it into the root partition when I do move it. In the past I've written up two aspects of that move to cover the necessary grub.cfg changes and a BIOS configuration change I'll need to make to really make my new SSDs into the BIOS boot drives, but I've never merged /boot into / in the process of such a move and I'm sure there will be new surprises.

(This is where I cough in quiet embarrassment and admit that even on my work machine, which moved its / filesystem to SSDs some time ago, my /boot still comes from HDs. I really should fix that by merging /boot into the SSD / at some point. Probably I'll use doing it at work as a trial run for doing it at home, because I have a lot more options for recovery if something goes wrong at work.)

PS: The obvious thing to do for merging /boot into / is to build a Fedora 25 virtual machine with a separate /boot and then use it to test just such a merger. There's no reason to blow up my office workstation when I can work out most of the procedure beforehand. This does require a new custom-built Fedora 25 VM image, but it's still probably faster and less hassle than hacking up my office machine.

PPS: It's possible that grub2-mkconfig will do a lot of this work for me (even things like the rd.md.uuid and root= changes). But I have an old grub.cfg that I like and grub2-mkconfig would totally change it around. It's easier to hand modify grub.cfg than write the new config to a new file and then copy bits of it, and in the process I wind up with a better understanding of what's going on.

linux/RootFilesystemSSDMigrationNotes written at 23:20:05; Add Comment

Some views on the Corebird Twitter client

I mentioned recently that my Fedora 25 version of choqok doesn't support some of the latest Twitter features, like quoted tweets (and this causes me to wind up with a bit of a Rube Goldberg environment to deal with it). In a comment, Georg Sauthoff suggested taking a look at Corebird, which is a (or the) native Gtk+ Twitter client. I've now done so and I have some views as a result, both good and bad.

The good first. Corebird is the best Linux client I've run into for quickly checking in on Twitter and skimming my feed; it comes quite close to the Tweetbot experience, which is my gold standard here. A lot of this is that Corebird understands and supports modern Twitter and does a lot directly in itself; you can see quoted tweets, you can see all of the images attached to a tweet and view them full sized with a click, and Corebird will even play at least some animations and videos. All of this is good for quickly skimming over things because you don't have to go outside the client.

Corebird doesn't quite have all of the aspects of the experience nailed in the way that Tweetbot does, especially in the handling of chains of tweets. Tweetbot shows you the current tweet in the middle, past tweets (tweets it was a reply to) above it, and future tweets (tweets that replied to it) below, and you can jump around to other tweets. Corebird shows only past tweets and shows them below, in reverse chronological order, which kind of irritates me; it should be above with the oldest tweet at the top. And you can't jump around.

However, for me Corebird is not what I want to use to actively follow Twitter on an ongoing basis, and I say this for two reasons. The first is that I tried to do it and it seems to have given me a headache (I'm not sure why, but I suspect something about font rendering and UI design). The second is that it's missing a number of features that I want for this, partly because I've found that the user interface for this matters a lot to me. Things that Corebird is missing for me include:

  • no unread versus read marker.
  • you can't have multiple accounts in a single tabbed window; you need either separate windows, one for each account, or to switch back and forth.
  • it doesn't minimize to (my) system tray the way Choqok does; instead you have to keep it running, which means keeping multiple windows iconified and eating up screen space with their icons.
  • it doesn't unobtrusively show a new message count, so I basically have to check periodically to see if there's more stuff to look at.

(With multiple accounts you don't want to quit out of Corebird on a regular basis, because when it starts up only one of those accounts will be open (in one window), and you'll get to open up windows for all of the other ones.)

Corebird will put up notifications if you want it to, but they're big obtrusive things. I don't want big obtrusive notifications about new unread Twitter messages; I just want to know if there are any and if so, roughly how many. Choqok's little count in its tray icon is ideal for this; I can glance over to get an idea if I want to check in yet or not. I also wish Corebird would scroll the timeline with keys, not just the mouse scrollwheel.

I'm probably going to keep Corebird around because it's good for checking in quickly and skimming things, and there's plenty of time when it's good for me to not actively follow Twitter (to put it one way, following Twitter is a great time sink). I'm definitely glad that I checked it out and that Georg Sauthoff mentioned it to me. But I'm going to keep using Choqok as my primary client because for my particular tastes, it works better.

PS: It turns out that Choqok 1.6 will support at least some of these new Twitter features, and it's on the way some time for Fedora. Probably not before Fedora 26, though, because of dependency issues (unless I want to build a number of packages myself, which I may decide to).

linux/CorebirdViews written at 00:44:53; Add Comment


Using pup to deal with Twitter's increasing demand for Javascript

I tweeted:

.@erchiang 's pup tool just turned a gnarly HTML parsing hassle into a trivial shell one liner. Recommended. https://github.com/ericchiang/pup

I like pup so much right now that I want to explain this and show you what pup let me do easily.

I read Twitter through a moderately Rube Goldberg environment (to the extent that I read it at all these days). Choqok, my Linux client, doesn't currently support new Twitter features like long tweets and quoted tweets; the best it can do is give me a link to read the tweet on Twitter's website. Twitter itself is increasingly demanding that you have Javascript on in order to make their site work, which I refuse to turn on for them. The latest irritation is a feature that Twitter calls 'cards'. Cards basically embed a preview of the contents of a link in the tweet; naturally they don't work without JavaScript, and naturally Twitter is turning an increasing number of completely ordinary links into cards, which means that I don't see them.

(This includes the Github link in my tweet about pup. Good work, Twitter.)

If you look at the raw HTML of a tweet, the actual link URL shows up in a number of places (well, the t.co shortened version of it, at least). In a surprise to me, one of them is in an actual <a> link in the Tweet text itself; unfortunately, that link is deliberately hidden with CSS and I don't currently have a viable CSS modification tool in my browser that could take that out. If we want to extract this link out of the HTML, the easiest place is in a <div> that has the link mentioned as a data-card-url property:

<div class="js-macaw-cards-iframe-container initial-card-height card-type-summary"

All we have to do is go through the HTML, find that property, and extract the property value. There are many ways to do this, some better than others; you might use curl, grep, and sed, or you might write a program in the language of your choice to fetch the URL and parse through the HTML with your language's HTML parsing tools.

This is where Eric Chiang's pup tool comes in. Pup is essentially jq for HTML, which means that it can be inadequately described as a structured, HTML-parsing version of grep and sed (see also). With pup, this problem turns into a shell one-liner:

wcat "$URL" | pup 'div[data-card-url] attr{data-card-url}'

The real script that uses this is somewhat more than one line, because it actually gets the URL from my current X selection and then invokes Firefox on it through remote control.

I've had pup sitting around for a while, but this is the first time I've used it. Now that I've experienced how easy pup makes it to grab things out of HTML, I suspect it's not going to be the last time. In fact I have a hand-written HTML parsing program for a similar job that I could replace with a similar pup one-liner.

(I'm not going to do so right now because the program works fine now. But the next time I have to change it, I'll probably just switch over to using pup. It's a lot less annoying to evolve and modify a shell script than it is to keep fiddling with and rebuilding a program.)

PS: via this response to my tweet, I found out about jid, which is basically an interactive version of jq. I suspect that this is going to be handy in the future.

PPS: That the URL is actually in a real <a> link in the HTML does mean that I can turn off CSS entirely (via 'view page in no style', which I have as a gesture in FireGestures because I use it frequently. This isn't all that great, though, because a de-CSS'd Tweet page has a lot of additional cruft on it that you have to scroll through to get to the actual tweet text. But at least it's an option.

Sidebar: Why I don't have CSS mangling in my Firefox

The short version is that both GreaseMonkey and Stylish leak memory on me. I would love to find an addon that doesn't leak memory and enables this kind of modification (here I'd like to strip a 'u-hidden' class from an <a href=...> link), but I haven't yet.

web/PupFixingTwitterMess written at 01:37:09; Add Comment


robots.txt is a hint and a social contract between sites and web spiders

I recently read the Archive Team's Robots.txt is a suicide note (via), which strongly advocates removing your robots.txt. As it happens, I have a somewhat different view (including about how sites don't crash under load any more; we have students who beg to differ).

The simple way to put it is that the things I add to robots.txt are hints to web spiders. Some of the time they are a hint that crawling the particular URL hierarchy will not be successful anyways, for example because the hierarchy requires authentication that the robot doesn't have. We have inward facing websites with sections that provide web-based services to local users, and for that matter we have a webmail system. You can try to crawl those URLs all day, but you're not getting anywhere and you never will.

Some of the time my robots.txt entries are a hint that if you crawl this anyways and I notice, I will use server settings to block your robot from the entire site, including content that I was letting you crawl before then. Presumably you would like to crawl some of the content instead of none of it, but if you feel otherwise, well, crawl away. The same is true of signals like Crawl-Delay; you can decide to ignore these, but if you do our next line of defense is blocking you entirely. And we will.

(There are other sorts of hints, and for complex URL structures some of the hints of all sorts are delivered through nofollow. Beyond not irritating me, there are good operational reasons to pay attention to this.)

This points to the larger scale view of what robots.txt is, which is a social contract between sites and web spiders. Sites say 'respect these limits and we will (probably) not block you further'. As a direct consequence of this, robots.txt is also one method to see whether a web spider is polite and well behaved or whether it is rude and nasty. A well behaved web spider respects robots.txt; a nasty one does not. Any web spider that is crawling URLs that are blocked in a long-standing robots.txt is not a nice spider, and you can immediately proceed to whatever stronger measures you feel like using against such things (up to and including firewall IP address range bans, if you want).

By the way, it is a feature that robots self-identify themselves when matching robots.txt. A honest and polite web spider is in a better position to know what it is than a site that has to look at the User-Agent and other indicators, especially because people do dangerous things with their user-agent strings. If I ban a bad robot via server settings and you claim to be sort of like that bad robot for some reason, I'm probably banning you too as a side effect, and I'm unlikely to care if that's a misfire; by and large it's your problem.

(With all of this said, the Archive Team has a completely sensible reason for ignoring robots.txt and I broadly support them doing so. They will run into various sorts of problems from time to time as a result of this, but they know what they're doing so I'm sure they can sort the problems out.)

web/RobotsTxtHintAndSocialContract written at 23:16:33; Add Comment

Sometimes, firmware updates can be a good thing to do

There are probably places that routinely apply firmware updates to every piece of hardware they have. Oh, sure, with a delay and in stages (rushing into new firmware is foolish), but it's always in the schedule. We are not such a place. We have a long history of trying to do as few firmware updates as possible, for the usual reason; usually we don't even consider it unless we can identify a specific issue we're having that new firmware (theoretically) fixes. And if we're having hardware problems, 'update the firmware in the hope that it will fix things' is usually last on our list of troubleshooting steps; we tacitly consider it down around the level of 'maybe rebooting will fix things'.

I mentioned the other day that we've inherited a 16-drive machine with a 3ware controller care. As far as we know, this machine worked fine for the previous owners in a hardware (controller) RAID-6 configuration across all the drives, but we've had real problems getting it stable for us in a JBOD configuration (we much prefer to use software RAID; among other things, we already know how to monitor and manage that with Ubuntu tools). We had system lockups, problems installing Ubuntu, and under load such as trying to scan a 14-disk RAID-6 array, the system would periodically report errors such as:

sd 2:0:0:0: WARNING: (0x06:0x002C): Command (0x2a) timed out, resetting card.

(This isn't even for a disk in the RAID-6 array; sd 2:0:0:0 is one of the mirrored system disks.)

Some Internet searches turned up people saying 'upgrade the firmware'. That felt like a stab in the dark to me, especially if the system had been working okay for the previous owners, but I was getting annoyed with the hardware and the latest firmware release notes did talk about some other things we might want (like support for disks over 2 TB). So I figured out how to do a firmware update and applied the 'latest' firmware (which for our controller dates from 2012).

(Unsurprisingly the controller's original firmware was significantly out of date.)

I can't say that the firmware update has definitely fixed our problems with the controller, but the omens are good so far. I've been hammering on the system for more than 12 hours without a single problem report or hiccup, which is far better than it ever managed before, and some things that had been problems before seem to work fine now.

All of this goes to show that sometimes my reflexive caution about firmware updates is misplaced. I don't think I'm ready to apply all available firmware updates before something goes into production, not even long-standing ones, but I'm certainly now more ready to consider them than I was before (in cases where there's no clear reason to do so). Perhaps I should be willing to consider firmware updates as a reasonably early troubleshooting step if I'm dealing with otherwise mysterious failures.

sysadmin/FirmwareUpdatesCanBeGood written at 01:27:00; Add Comment

(Previous 11 or go back to February 2017 at 2017/02/16)

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.