Wandering Thoughts archives


The (current) state of Firefox Nightly and old extensions

Back in January in my entry on how ready my Firefox extensions are for Firefox Electrolysis, I said that Firefox's release calendar suggested that Firefox's development version (aka 'Nightly') would stop supporting old non-Electrolysis extensions some time around June or July. It's now mid June and some things have happened, but I'm not sure where Mozilla's timeline is on this. So here is what I know.

At the start of May, Firefox Nightly landed bug 1352204, which is about disabling a lot of older extensions on Nightly. Mozilla has an information page about this in their wiki, and various news outlets noticed and reported on this change shortly after it went live, which means I'm late to the party here. As the Mozilla page covers, you can fix this by setting the about:config option extensions.allow-non-mpc-extensions to true. I've done this ever since I found the option and everything appears to still work fine in the current Nightly.

(I had some weird things happen with Youtube that caused me to not update my Firefox build for a month or so because I didn't want to deal with tracking the issue down, but when I started to test more extensively they went away. Problems that vanish on their own can be the best problems.)

This change itself doesn't seem to be how Mozilla intends to turn off old extensions, theoretically in Firefox 57. That seems to be bug 1336576, expanded in a Mozilla wiki entry. Based on the Mozilla wiki entry, it appears that Firefox's development code base (and thus Nightly) will continue to allow you to load old extensions even after Firefox 57 is released provided that you flip a magic preference. Firefox 57 itself will not allow you to do so; the preference will apparently do nothing.

As long as Mozilla has legacy extensions that they care about, I believe that the actual code to load and operate such extensions will be present and working in the Firefox code base; this is the 'signed by Mozilla internally' case in their compatibility table. This implies that even if Mozilla disables the preference in the development version, you can force-override this with a code change if you build your own Firefox (which is what I do). You may not be able to turn Electrolysis on if you have such old legacy extensions, but presumably your addons are more important than Electrolysis (this is certainly the case for me).

All of this makes me much happier about the state of my personal Firefox than I used to be, because it looks like the point where many of my current extensions will fall over is much further away than I thought it was. Far from being this summer, it may be next summer, or evn further away than that, and perhaps by then the release of Firefox 57+ will have caused more of the addons that I care about to be updated.

(However, not all of the omens for updated addons are good. For example, Self-Destructing Cookies now explicitly marks itself as incompatible with Electrolysis because apparently addon can't monitor sites' LocalStorage usage in e10s. This suggests that there are important gaps in what addons can now do, gaps that Mozilla may or may not close over time. At least this particular case is a known issue, though; see bugs 1333050, 1329745, and 1340511 (via the addons page for Cookie Autodelete, which I was recently pointed at by a helpful reader of Wandering Thoughts).)

web/FirefoxElectrolysisOldExtensions written at 01:35:49; Add Comment


Why I am not installing your app on my phone

For reasons beyond the scope of this entry, I spent a decent chunk of time today using my phone to amuse myself. Part of that time was reading Twitter, and part of that reading involved following links to interesting articles on various places. Quite a number of those places wanted me to install their iPhone app instead of reading things on their website, and some of them were quite obnoxious about it. For example, Medium sticks a prominent non-dismissable button in the middle of the bottom of the screen, effectively shrinking an already-too-small screen that much further.

Did I install any of the apps that these websites wanted me to? Of course not. This is not because my phone has only limited space, and it's not really because I prefer keeping my phone uncluttered. There's a much more fundamental reason: I don't trust your app. In fact, I assume that almost all apps that websites want me to use instead of reading the site are actually trojan horses.

By this, I don't mean that I expect any of these apps to quietly attack the security of my phone and attempt to compromise it (although I wouldn't take that bet on Android). I don't even necessarily expect all of these apps to demand intrusive device permissions, like constant access to location services (although I suspect that a lot of them will at least ask for lots of permissions, because maybe I'll be foolish enough to agree). I do definitely expect that all of these apps will put their app nature to 'good' use in order to spy on, track, and monetize my in-app activity to a much larger extent than their websites can. Any potential improvement in my user experience over just reading the website is incidental to their actual reason for existing, which is why they're trojan horses.

So no, I will not install your app, because I trust Apple and Safari to do a lot more to preserve my privacy against your website and your spying on me than I trust you. Safari allows various sorts of (ad)blockers, it allows me to totally discard cookies and other tracking identification that websites have foisted off on me, and in general I trust Apple to be pretty carefully restrictive of what it allows websites to do with JavaScript and what information they can get. Your app? I have no such trust; in fact I expect the exact reverse of trust.

There is nothing particularly surprising here, of course. This is simply the inevitable result of the business model of these websites. I'm not their customer, although they may pretend otherwise; instead, I am part of the product, to be packed up and sold off to advertisers. Trying to get me to accept the app is part of fattening me up for their actual customers.

(This is a bit of a grumpy rant, because I got sick and tired of all the 'install our app, really' badgering from various places, especially when it makes their websites less usable. Some of the time these nags encouraged me to close the page as not sufficiently fascinating, which may or may not have been a win for the websites in question.)

tech/WhyNotInstallingYourApp written at 01:05:50; Add Comment


The difference between ZFS scrubs and resilvers

At one level, asking what the difference is between scrubs and resilvers sounds silly; resilvering is replacing disks, while scrubbing is checking disks. But if you look at the ZFS code in the kernel things become much less clear, because both scrubs and resilvers use a common set of code to do all their work and it's actually not at all easy to tell what happens differently between them. Since I have actually dug through this section of ZFS code just the other day, I want to write down what the differences are while I remember them.

Both scrubs and resilvers traverse all of the metadata in the pool (in a non-sequential order), and both wind up reading all data. However, scrubs do this more thoroughly for mirrored vdevs; scrubs read all sides of a mirror, while resilvers only read one copy of the data (well, one intact copy). On raidz vdevs there is no difference here, as both scrubs and resilvers read both the data blocks and the parity blocks. This implies that a scrub on mirrored vdevs does more IO and (importantly) more checking than a resilver does. After a resilver of mirrored vdevs, you know that you have at least one intact copy of every piece of the pool, while after an error-free scrub of mirrored vdevs, you know that all ZFS metadata and data on all disks is fully intact.

For resilvers but not scrubs (at least normally), ZFS will sort of attempt to write everything back to the disks again, as I covered in the sidebar of last entry. As far as I can tell from the code, ZFS always skips even trying to write things back to disks that are either known to have good data (for example they were read from) or that are believed to be good because their DTL says that the data is clean on them (I believe that this case only really applies for mirrored vdevs). For scrubs, the only time ZFS submits writes to the disks is if it detects actual errors.

(Although I don't entirely understand how you get into this situation, it appears that a scrub of a pool with a 'replacing' disk behaves a lot like a resilver as far as that disk is concerned. As you would expect and want, the scrub doesn't try to read from the new disk and, like resilvers, it tries to write everything back to the disk.)

Since we only have mirrored vdevs on our fileservers, what really matters to us here is the difference in what gets read between scrubs and resilvers. On the one hand, resilvers put less read load on the disks, which is good for reducing their impact. On the other hand, resilvering isn't quite as thorough a check of the pool's total health as a scrub is.

PS: I'm not sure if either a scrub or a resilver reads the ZIL. Based on some code in zil.c, I suspect that it's checked only during scrubs, which would make sense. Alternately, zil.c is reusing ZIO_FLAG_SCRUB for non-scrub IO for some of its side effects, but that would be a bit weird.

solaris/ZFSResilversVsScrubs written at 00:44:26; Add Comment


Resilvering multiple disks at once in a ZFS pool adds no real extra overhead

Suppose, not entirely hypothetically, that you have a multi-disk, multi-vdev ZFS pool (in our case using mirrored vdevs) and you need to migrate this pool from one set of disks to another set of disks. If you care about doing relatively little damage to pool performance for the duration of the resilvering, are you better off replacing one disk at a time (with repeated resilvers of the pool), or doing a whole bunch of disk replacements at once in one big resilver?

As far as we can tell from both the Illumos ZFS source code and our experience, the answer is that replacing multiple disks at once in a single pool is basically free (apart from the extra IO implied by writing to multiple new disks at once). In particular, a ZFS resilver on mirrored vdevs appears to always read all of the metadata and data in the pool, regardless of how many replacement drives there are and where they are. This means that replacing (or resilvering) multiple drives at once doesn't add any extra read IO; you do the same amount of reads whether you're replacing one drive or ten.

(This is unlike conventional RAID10, where replacing multiple drives at once will probably add additional read IO, which will affect array performance.)

For metadata, this is not particularly surprising. Since metadata is stored all across the pool, metadata located in one vdev can easily point to things located on another one. Given this, you have to read all metadata in the pool in order to even find out what is on a disk that's being resilvered. In theory I think that ZFS could optimize handling leaf data to skip stuff that's known to be entirely on unaffected vdevs; in practice, I can't find any sign in the code that it attempts this optimization for resilvers, and there are rational reasons to skip it anyway.

(As things are now, after a successful resilver you know that the pool has no permanent damage anywhere. If ZFS optimized resilvers by skipping reading and thus checking data on theoretically unaffected vdevs, you wouldn't have this assurance; you'd have to resilver and then scrub to know for sure. Resilvers are somewhat different from scrubs, but they're close.)

This doesn't mean that replacing multiple disks at once won't have any impact on your overall system, because your system may have overall IO capacity limits that are affected by adding more writes. For example, our ZFS fileservers each have a total write bandwidth of 200 Mbytes/sec across all their ZFS pool disks (since we have two 1G iSCSI networks). At least in theory we could saturate this total limit with resilver write traffic alone, and certainly enough resilver write traffic might delay normal user write traffic (especially at times of high write volume). Of course this is what ZFS scrub and resilver tunables are about, so maybe you want to keep an eye on that.

(This also ignores any potential multi-tenancy issues, which definitely affect us at least some of the time.)

ZFS does optimize resilvering disks that were only temporarily offline, using what ZFS calls a dirty time log. The DTL can be used to optimize walking the ZFS metadata tree in much the same way as how ZFS bookmarks work.

Sidebar: How ZFS resilvers (seem to) repair data, especially in RAIDZ

If you look at the resilvering related code in vdev_mirror.c and especially vdev_raidz.c, how things actually get repaired seems pretty mysterious. Especially in the RAIDZ case, ZFS appears to just go off and issue a bunch of ZFS writes to everything without paying any attention to new disks versus old, existing disks. It turns out that the important magic seems to be in zio_vdev_io_start in zio.c, where ZFS quietly discards resilver write IOs to disks unless the target disk is known to require it. The best detailed explanation is in the code's comments:

 * If this is a repair I/O, and there's no self-healing involved --
 * that is, we're just resilvering what we expect to resilver --
 * then don't do the I/O unless zio's txg is actually in vd's DTL.
 * This prevents spurious resilvering with nested replication.
 * For example, given a mirror of mirrors, (A+B)+(C+D), if only
 * A is out of date, we'll read from C+D, then use the data to
 * resilver A+B -- but we don't actually want to resilver B, just A.
 * The top-level mirror has no way to know this, so instead we just
 * discard unnecessary repairs as we work our way down the vdev tree.
 * The same logic applies to any form of nested replication:
 * ditto + mirror, RAID-Z + replacing, etc.  This covers them all.

It appears that preparing and queueing all of these IOs that will then be discarded does involve some amount of CPU, memory allocation for internal data structures, and so on. Presumably this is not a performance issue in practice, especially if you assume that resilvers are uncommon in the first place.

(And certainly the code is better structured this way, especially in the RAIDZ case.)

solaris/ZFSMultidiskResilversFree written at 22:29:35; Add Comment


Why filing away mailing lists for a while has improved my life

I've been on vacation for the past little while. As part of this vacation, I carried out my plans to improve my vacations, part of which was using procmail to divert messages from various mailing lists off to files instead of having them delivered to my inbox as I usually do. I started out only doing this to mailing lists for work software, like Exim and OmniOS, but as my vacation went on I added the mailing lists for other things that I use. As I hoped and expected, this worked out quite well; I soon got over my urge to check in on the mailing lists and mostly ignored them.

Recently I came to a realization about why this feels so good. It's not specifically that it's reduced the volume of email in my inbox; instead, the really important thing it's done is that right now, pretty much everything that shows up in my inbox is actually important to me. It's email from friends and family, notifications that I care about getting, and so on.

(Coming to this realization and writing it up has sharpened my awareness that some of the remaining email going to my inbox doesn't make this bar, and thus should also be filed away on future breaks and vacations.)

There's nothing wrong with the emails from those mailing lists. They're generally perfectly interesting. But right now (and in general) the mailing list email is not important in that way. It's not something that I care about. When it all was going into my inbox, a significant amount of my inbox was stuff that I didn't really care about. That doesn't feel good (and has other effects). Now my inbox is very pared down; it's either silent and empty, or the new email is something that I actively want to read because it matters to me.

(In other words, it's not just that processing my inbox is faster now, it's that the payoff from doing so is much higher. And when there is no payoff, there's no email.)

If I'm being honest about these mailing lists, most of this is going to be true even when I go back to work tomorrow morning. Sure, if I've just asked a question or gotten into a conversion, reading the mailing list immediately usually has a relatively high payoff. But at other times, the payoff is much lower and having the mailing lists go straight to my inbox is just giving me a slow drizzle of low-priority, low-payoff email that I wind up having to pay some attention to.

In fact I think a drizzle is a good analogy here. Like the moment to moment experience of biking in a light drizzle, the individual emails are not particularly onerous or bad. But the cumulative result of staying out in that light drizzle is that you quietly wind up soaked, bit by bit by bit. So I think it's time for me to get out of the email drizzle for a while, at least to see what it's like on an ongoing basis.

(I intend to still read these mailing list emails periodically, but I'm going to do it in big batches and at a time of my choosing. Over a coffee at the end (or start) of a day at work, perhaps. I'll have to see.)

sysadmin/EmailGettingOutOfTheDrizzle written at 23:21:51; Add Comment

How to see raw USB events on Linux via usbmon

Suppose, not entirely hypothetically, that you're wondering if your USB keyboard actually does anything for certain key presses, such as its Fn button plus a normal letter key like 's'. One way to try to find out is just to type the key combination while sitting at your shell prompt or in your editor or the like, and see what happens. A more elaborate method is to fire up xev and see what it says about X events, since there are a number of things that can happen between the level of X events and what your shell sees. Of course this starts to hint at the broad problem, which is that a modern graphical Linux environment has all sorts of layers that may swallow or distort raw events, so not seeing anything in the shell or in xev only means that things didn't make it that far.

(This is useful knowledge, of course, especially if your ultimate goal is getting characters to your shell or to an X program that should recognize them. And hitting that key produces some event, xev will tell you what it's been turned into.)

Usefully, the Linux kernel gives us a way to bypass all of the (potential) layers of input processing and see the raw USB events being generated (or not generated, as the case may be). Looking at the presence or absence of raw events is pretty definite. If the keyboard is not generating any USB events when you press some keys, well, that's it. How you do this is with the kernel's usbmon system, as covered in the kernel's usbmon.txt and Ubuntu's page on debugging USB. For quick checks, the text interface in /sys/kernel/debug/usb/usbmon is your most convenient option, but Linux distributions seem to vary as to whether you need to load a usbmon kernel module or not to get it.

(On my Fedora 25 machines, everything is ready to go out of the box, with no kernel modules needing to be loaded; the Fedora kernels are apparently built with CONFIG_USB_MON=y. On a relatively stock Ubuntu 16.04 server, there's a usbmon module and it's not loaded by default.)

Wireshark can capture and interpret USB bus traffic, per here, and in theory should be a good way to see the details of USB events in a more user-friendly format than the kernel's text dump. In practice I can't seem to persuade the Fedora 25 version to give me useful information here. I find it more helpful to read the output from the text interface, which at least lets me distinguish one sort of event from another (for example, the mouse scrollwheel going in one direction versus the other direction). Possibly I'm missing some magic Wireshark options, especially since I don't use Wireshark very often. Alternately, I'd need to know a non-casual amount about USB message formats and the details of the USB protocol in order to understand what Wireshark is showing me and extract the things I'm interested in.

(There also may be an issue that apparently Wireshark may only do a good job decoding things if it sees you plug in the USB device. This is perhaps sensible behavior for Wireshark, or even necessary, but it's not very useful for checking the details of what my (only) keyboard or mouse generate. I'm not really enthused about unplugging and then replugging them; it has somewhat annoying side effects.)

As a side note, since you can only monitor an entire USB buss (or all busses at once), it's not necessarily very important to be detailed about identifying what USB device is where. For instance, on my home machine, the only USB bus that reports any non-hub devices is bus 2, so even if a plain 'lsusb' doesn't clearly identify the specific USB device that's my keyboard or mouse, it's a pretty good bet that they're on bus 2. 'lsusb -t' can also give strong hints; on my home machine, all 'Class=Human Interface Device' USB devices are on bus 2 even if lsusb doesn't tell me exactly what their (product) names are.

(I started writing this entry just to have some information about the usbmon feature recorded for my later use, and wound up learning useful things about Wireshark's USB stuff.)

PS: Because Wireshark is actually using libpcap to get the USB traffic from the kernel, you can also use tcpdump to capture USB traces, for example 'tcpdump -i usbmon2 -s 0 -w /tmp/usbmon.pcap'. This may be handy if you have a server seeing USB oddities; at least around here, we're far more likely to have basic tcpdump installed on random machines than we are to put Wireshark on them. As usual, you can use tcpdump to capture the packet trace, transfer it to your workstation, and run Wireshark on the capture to decode and analyze it and so on.

(This is also how I avoid running Wireshark as root even if I'm capturing the traffic on my own machine.)

linux/USBMonSeeingUSBEvents written at 00:52:36; Add Comment


One downside of the traditional style of writing Unix manpages

A while back I wrote about waiting for a specific wall-clock time in Unix, which according to POSIX you can do by using clock_nanosleep with the CLOCK_REALTIME clock and the TIMER_ABSTIME flag. This is fully supported on Linux (cf) and not supported on FreeBSD. But here's a question: is it supported on Illumos-derived systems?

So, let us consult the Illumos clock_nanosleep manpage. This manpage is very much written in the traditional (corporate) style of Unix manpages, high on specification and low on extra frills. This style either invites or actively requires a very close reading, paying very careful attention to both what is said and what is not said. The Illumos manpage does not explicitly say that your sleep immediately ends if the system's wall clock time is adjusted forward far enough; instead it says, well:

If the flag TIMER_ABSTIME is set in the flags argument, the clock_nanosleep() function causes the current thread to be suspended from execution until either the time value of the clock specified by clock_id reaches the absolute time specified by the rqtp argument, [or a signal happens]. [...]

The suspension time caused by this function can be longer than requested because the argument value is rounded up to an integer multiple of the sleep resolution, or because of the scheduling of other activity by the system. [...] The suspension for the absolute clock_nanosleep() function (that is, with the TIMER_ABSTIME flag set) will be in effect at least until the value of the corresponding clock reaches the absolute time specified by rqtp, [except for signals].

On the surface, this certainly describes a fully-featured implementation of clock_nanosleep that behaves the way we want. Unfortunately, if you're a neurotic reader of Unix manpages, all is not so clear. The potential weasel words are 'the suspension ... will be in effect at least until ...'. If you don't shorten CLOCK_REALTIME timeouts when the system clock jumps forward, you are technically having them wait 'at least until' the clock reaches their timeout value, because you sort of gave yourself room to have them wait (significantly) longer. At the same time this is a somewhat perverse reading of the manpage, partly because the first sentence of that paragraph alleges that the system will only delay waking you up because of scheduling, which would disallow this particular perversity.

To add to my uncertainty, let's look at the Illumos timer_settime manpage, which contains the following eyebrow-raising wording:

If the flag TIMER_ABSTIME is set in the argument flags, timer_settime() behaves as if the time until next expiration is set to be equal to the difference between the absolute time specified by the it_value member of value and the current value of the clock associated with timerid. That is, the timer expires when the clock reaches the value specified by the it_value member of value. [...]

These two sentences do not appear to be equivalent for the case of CLOCK_REALTIME clocks. The first describes an algorithm that freezes the time to (next) expiration when timer_settime is called, which is not proper CLOCK_REALTIME behavior, and then the second broadly describes correct CLOCK_REALTIME behavior where the timer expires if the real time clock advances past it for any reason.

With all that said, Illumos probably fully implements CLOCK_REALTIME, with proper handling of the system time being adjusted while you're suspended or have a timer set. But its manpages never comes out and say that explicitly, because that's simply not the traditional style of Unix manpages, and the way they're written leaves me with uncertainty. If I cared about this, I would have to write a test program and then run it on a machine where I could set the system time both forward and backward.

This fault is not really with these specific Illumos manpages, although some elements of their wording aren't helping things. This is ultimately a downside to the terse, specification-like traditional style of Unix manpages. Where every word may count and the difference between 'digit' and 'digits' matters, you sooner or later get results like this, situations where you just can't tell.

(Yes, this would be a perverse implementation and a weird way of writing the manpages, but (you might say) perhaps the original Solaris corporate authors really didn't want to admit in plain text that Solaris didn't have a complete implementation of CLOCK_REALTIME.)

Also, I'm sure that different people will read these manpages differently. My reading is unquestionably biased by knowing that clock_nanosleep support is not portable across all Unixes, so I started out wondering if Illumos does support it. If you start reading these manpages with the assumption that of course Illumos supports it, then you get plenty of evidence for that position and all of the wording that I'm jumpy about is obviously me being overly twitchy.

unix/ManpageStyleDownside written at 01:19:53; Add Comment


My .procmailrc has quietly sort of turned into a swamp

As part of trying to not read some mailing lists for a while, I was recently going through my .procmailrc. Doing this was eye-opening. It's not that my .procmailrc was messy as such, because I don't have rules that are sophisticated enough to get messy (just a bunch of 'if mail is like <X>, put it into file Y' filtering rules). Instead, mostly what it had was a whole lot of old, obsolete rules that haven't been relevant for years.

Looking back, it's easy to see how these have quietly accreted over time. Like many configuration files, I almost never look at my .procmailrc globally, scanning through the whole thing. Instead, when I have a new filtering rule I want to add, I jump to what seems to be the right place (often the bottom) and edit the new rule in. If I notice in passing what might be an obsolete filtering rule for a type of email that I don't get any more, usually I ignore it, because investigating is more work and wasn't part of my goal when I did 'vi .procmailrc'.

(The other thing that a strictly local view of changes has done to my .procmailrc is create a somewhat awkward structure for the sequence of rules. This resulted in some semi-duplication of rules and one bit of recent miss-classification, when I got the ordering of two rules wrong because I didn't even realize there was an ordering dependency.)

As a result of stubbing my toe on this, I now have two issues (or problems) I'm considering. The first is what to do about those obsolete rules. Some of them are completely dead and can just be deleted, but others are for things that just might come back to life, even if it hasn't happened for years. There is a part of me that wants to preserve those rules somewhere, just in case I want them again some day. This is probably foolish. Perhaps what I should do is save a backup copy somewhere (or just check my .procmailrc into RCS first).

The second is how much of a revision to do. Having now actively looked at the various things I'm doing and want to do in my .procmailrc, there's a temptation to completely restructure it by splitting my rules into multiple files and then including them in the right spots. This would make where to put new rules clearer to future me, make the overall structure much clearer, and make it simpler to do global things like temporarily divert almost all the mailing lists I get off to files (all those rules would be in one file, so I'd either include it or not include it in my .procmailrc). On the other hand, grand reforms are arguably second system enthusiasm showing. It might be that I'd spend a bunch of time fiddling around with my mail filtering and wind up with a much more complicated, harder to follow setup that basically did the same thing.

(Over-engineering in a fit of enthusiasm is a real hazard.)

PS: applications to other configuration files you might have lying around are left as an exercise for the reader, but I'm certainly suspecting that this is not the only file I have (or that we have) that exhibits this 'maintained locally but not globally' slow, quiet drift into a swamp.

sysadmin/ProcmailrcSwamp written at 01:19:42; Add Comment


In practice, putting SSDs into 3.5" drive bays is a big hassle

When I talked about how we failed at making all our servers have SSD system disks, I briefly talked about how one issue was that SSDs are not necessarily easily compatible with 3.5" drive bays. If you have never encountered this issue, you may be scratching your head, because basic spacers to let you put 2.5" drives (SSDs included) into 3.5" drive bays are widely available and generally dirt cheap. Sure, you have to screw some extra things on your SSDs, but unless you're working at a much bigger scale than we are, this doesn't really matter.

The problem is that this doesn't always work in servers, depending on how their drive bays work. The fundamental issue is that a 3.5" SATA HD has its power and SATA connectors at the bottom left edge of the drive, as does a 2.5" SSD, and a simple set of spacers can't position the SSD so that both the connectors and the screw holes line up where they need to be. In servers where you manually insert the SATA and power cables and the provided cables are long enough, you can stretch things to make simple spacers work. In servers with exact-length cables or with hot-swap bays that you slide drives into (either with or without carriers), simple spacers don't work and you need much more expensive sleds (such as IcyDock's).

(Sleds are not very expensive in small quantities, but if you're looking at giving a bunch of servers dual SSD system disks and you're planning to use inexpensive SSDs, adding a $15 part to each disk adds up fast.)

We sort of knew about this issue when we started, but we thought it wasn't going to be a big deal. We were wrong. It adds cost and just as important, it adds friction; it's an extra part to figure out, to qualify, to stock, and to reorder when you start running low. You can't just grab a SSD or two and stick them in a random server, even if you have the SSDs; you have to figure out what you need to get the SSDs mounted, perhaps see if you have one or two sleds left, and so on and so forth.

The upshot of all of this is that we're now highly motivated to get 2.5" drive bays in our next set of basic 1U servers, at least for servers with only two drive bays. As a pleasant side benefit, this would basically give us no choice but to use SSDs in these servers, since we don't have any random old 2.5" HDs and we're unlikely to buy new 2.5" HDs.

(Sadly, this issue is basically forced by the constraints of 3.5" and 2.5" HDs. The power and SATA connectors are at the edge of each because that's where the circuit board goes, and it goes on the outside of the drive in order to leave as much space as possible for the platters, the motors, and so on.)

sysadmin/SSDIn3.5DriveBayProblem written at 02:44:37; Add Comment


A humbling experience of misreading some simple (Go) code

Every so often, I get to have a humbling experience, sometimes in public and sometimes in private. Recently I was reading Go Range Loop Internals (via) and hit its link to this Damian Gryski (@dgryski) tweet:

Today's #golang gotcha: the two-value range over an array does a copy. Avoid by ranging over the pointer instead.


I ran the code on the playground, followed it along, and hit a 'what?' moment where I felt I had a mystery where I didn't understand why Go was doing something. Here is the code:

func IndexValueArrayPtr() {
  a := [...]int{1, 2, 3, 4, 5, 6, 7, 8}

  for i, v := range &a {
    a[3] = 100
    if i == 3 {
      fmt.Println("IndexValueArrayPtr", i, v)

Usefully, I have notes about my confusion, and I will put them here verbatim:

why is the IndexValueArrayPtr result '3 100'? v should be copied before a[3] is modified, and v is type 'int', not a pointer.

This is a case of me reading the code that I thought was there instead of the code that was actually present, because I thought the code was there to make a somewhat different point. What I had overlooked in IndexValueArrayPtr (and in fact in all three functions) is that a[3] is set on every pass through the loop, not just when i == 3.

Misreading the code this way makes no difference to the other two examples (you can see this yourself with this variant), but it's crucial to how IndexValueArrayPtr behaves. If the a[3] assignment was inside the if, my notes would be completely true; v would have copied the old value of a[3] before the assignment and this would print '3 4'. But since the assignment happens on every pass of the loop, a[3] has already been assigned to be 100 by the time the loop gets to the fourth element and makes v a copy of it.

(I think I misread the code this way partly because setting a[3] only once is more efficient and minimal, and as noted the other two functions still illustrate their particular issues when you do it that way.)

Reading an imaginary, 'idealized' version of the code instead of the real one is not a new thing and it's not unique to me, of course. When you do it on real code in a situation where you're trying to find a bug, it can lead to a completely frustrating time where you literally can't see what's in front of your eyes and then when you can you wonder how you could possibly have missed it for so long.

(I suspect that this is a situation where rubber duck debugging helps. When you have to actually say things out loud, you hopefully get another chance to have your brain notice that what you want to say doesn't actually correspond to reality.)

PS: The reason I have notes on my confusion is that I was planning to turn explaining this 'mystery' into a blog entry. Then, well, I worked out the mystery, so now I've gotten to write a somewhat different sort of blog entry on it.

programming/GoMisreadingSomeCode written at 23:59:29; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.