Wandering Thoughts archives

Why I can't see IPv6 as a smooth or fast transition

Today I got native IPv6 up at home. My home ISP had previously been doing tunneled IPv6 (over IPv4), except that I'd turned my tunnel off back in June for some reason (I think something broke and I just shrugged and punted). I enjoyed the feeling of doing IPv6 right for a few hours, and then, well:

@thatcks: The glorious IPv6 future: with IPv6 up, Google searches sometimes just cut off below the initial banner and search box.
For bonus points, the searches aren't even going over IPv6. Tcpdump says Google appears to RSET my HTTPS TCPv4 connections sometimes.

(Further staring at packet traces makes me less certain of what's going on, although there are definitely surprise RSET packets in there. Also, when I said 'IPv6 up', I was being imprecise; what makes a difference is only whether or not I have an active IPv6 default route so that my IPv6 traffic can get anywhere. Add the default route (out my PPPoE DSL link) and the problems start to happen; delete it and everything is happy.)

Every so often someone says that the networking world should get cracking on the relatively simple job of adopting and adding IPv6 everywhere. Setting aside anything else involved, what happened to me today is why I laugh flatly at anyone who thinks this. IPv6 is simple only if everything works right, but we have plenty of existence proofs that it does not. Enabling IPv6 in a networking environment is a great way to have all sorts of odd problems come crawling out of the woodwork, some of which don't seem like they have anything to do with IPv6 at all.

It would be nice if these problems and stumbling points didn't happen, and certainly in the nice shiny IPv6 story they're not supposed to. But they do, and combined with the fact that IPv6 is often merely nice, not beneficial, I think many networks won't be moving very fast on IPv6. This makes a part of me sad, but it's the same part of me that thinks that problems like mine just shouldn't happen.

(I don't think I'm uniquely gifted in stumbling over IPv6 related problems, although I certainly do seem to have bad luck with it.)

Maybe I should try to find another good mini keyboard

As I've mentioned a few times before, I've been using one particular mini keyboard for a very long time now and I've become very attached to it. It's thoroughly out of production (although I have spares) and worse, it uses a PS/2 interface which presents problems in the modern world. One solution is certainly to go to a lot of work to keep on using it anyways, but I've been considering if perhaps I shouldn't try to find a modern replacement instead.

Some people are very attached to very specific keyboards for hard to replicate reasons; just ask any strong fan of the IBM Model M. But I'm not really one of them. I'm attached to having a mini keyboard that's not too mimimal (the Happy Hacking keyboard is too far) and has a reasonably sensible key layout, and I'd like to not have space eaten up by a Windows key that I have no use for, but I'm not attached to the BTC-5100C itself. It just happened to be the best mini keyboard we found back fifteen or more years ago when we looked around for them, or at least the best one that was reasonably widely available and written about.

The keyboard world has come a long way in the past fifteen years or so. The Internet has really enabled enthusiasts to connect with each other and for specialist manufacturers to serve them and to spread awareness of their products, making niche products much more viable and thus available. And while I like the BTC-5100C, I suspect that it is not the ultimate keyboard in terms of key feel and niceness for typing; even at the time it was new, it was not really a premium keyboard. In specific, plenty of people feel that mechanical keyboards are the best way to go and there are certainly any number of mechanical mini keyboards (as I've seen on the periodic occasions when I do Internet searches about this).

So I've been considering trying USB mechanical mini keyboard, just as I've sometimes toyed with getting a three button mouse with a scroll wheel. So far what's been stopping me has been the same thing in both cases, namely how much these things cost. I think I'm willing to pay $100 for a good keyboard I like that'll probably last the near side of forever, but it's hard to nerve myself up to spending that much money without being certain first.

(Of course, some or many places offer N-day money back guarantees. While shipping things back is likely to be kind of a pain, perhaps I should bite the bullet and just do it. Especially since I have a definite history of hesitating on hardware upgrades that turn out to be significant. One of the possible keyboards is even Canadian.)

(Of course there's a Reddit board for mechanical keyboards. I'll have to read through their pages.)

Sidebar: What I want in a mini keyboard layout

Based on my experiences with trying out a Happy Hacking keyboard once (and a few other mini keyboards), my basic requirements are:

• a separate row of function keys for F1 through F10. I simply use function keys too much to be satisfied with a very-mini layout that only has a four row layout with numbers and then the Q/A/Z letter rows (and gets at function keys via a 'FN' modifier key).

• actual cursor keys; again, I use them too much to be happy having to shift with something to get them.

• Backspace and Delete as separate keys. I can live with shifted Insert.
• Esc as a real (unshifted) key. Vi people know why.

• A SysRq key being available somehow, as I want to keep on being able to use Linux's magic SysRq key combos. This implies that I actually have to be able to use Alt + SysRq + letters and numbers.

  (I may have to give this up.)

(I think this is called a '75%' layout on Reddit.)

A sensible location for Esc would be nice but frankly I've given up on that; people have been moving Esc off to the outer edges of the keyboard for decades. The last keyboard I saw with a good layout there was the NCD Unix keyboard (which I now consider too big).

The good thing about having these basic requirements is that I can actually rule out a lot of keyboards based purely on looking at pictures of them, without having to hunt down reviews or commentary or the like.

One thing I'm hoping for in our third generation fileservers

If all goes according to my vague schedule, we should be at least starting to plan our third generation of fileservers in 2018, when our second generation fileservers are four years old. 2018 is not all that far off now, so every so often I think a bit about what interesting things might come up from the evolution of technology over the next few years.

Some things are obvious. I certainly hope our entire core network is reliable 10G (copper) Ethernet by 2018, for example, and I optimistically hope for at least doubling and ideally quadrupling the memory in fileservers (from 64 GB to 128 GB or 256 GB). And it's possible that we'll be completely blindsided by some technology shift that's currently invisible (eg a large scale switch from x86 to ARM).

(I call a substantial increase in RAM optimistic because RAM prices have been remarkably sticky for several years now.)

One potential change I'm really looking forward to is moving to all-SSD storage. Running entirely on SSDs would likely make a clear difference to how responsive our fileservers are (especially if we go to 10G Ethernet too), and with the current rate of SSD evolution it doesn't seem out of the bounds of reality. Certainly one part of this is that the SSD price per GB of storage keeps falling, but even by 2018 I'll be surprised if it's as cheap as relatively large HDs. Instead one reason I think it might be feasible for us is that the local demand for our storage just hasn't been growing all that fast (or at least people's willingness to pay for more storage seems moderate).

So let me put some relatively concrete numbers on that. Right now we're using 2 TB HDs and we have only one fileserver that's got more than half its space allocated. If space growth stays modest through 2018, we could likely replace the 2 TB HDs with, say, 3 TB SSDs and still have growth margin left over for the next four or five years. And in 2018, continued SSD price drops could easily make such SSDs cost about as much as what we've been paying for good 2TB 7200 RPM HDs. Even if they cost somewhat more, the responsiveness benefits of an all-SSD setup are very attractive.

(At a casual check, decent 2TB SSDs are currently somewhere around 4x to 5x more expensive than what we paid for our 2 TB HDs. Today to the start of 2018 gives them two years and a bit to cover that price ground, which may be a bit aggressive.)

There are two different scenarios for replacing disks in a RAID

One possible reply to my annoyance at btrfs being limited to two-way mirrors is to note that btrfs, like many RAID systems, allows you to explicitly replace disks. While btrfs is in the process of doing this, it maintains three-way redundancy (as do some but not all RAID systems); only at the end, with the new disk fully up and running, does it drop the old disk out of the replacement. If something goes wrong with the new disk during this process you are presumably no worse off than before. This is certainly better than the alternative, but it's not great because it misses one usage case. You see, there are two different scenarios for replacing disks here.

In the first scenario you are replacing a dying disk (or at least what you think is one). You don't trust it and the sooner you get data off it the better. As a result, a new unproven disk is strictly better than the old disk because at least the new disk (probably) isn't about to die. Discarding the old disk the moment all the data is fully copied to the new disk is perfectly fine; you trust the new disk at least as much as the old disk.

In the second scenario you are replacing a (currently) good disk with what you think is a better one; it has more space, it is more modern, it is a SSD instead of a HD, whatever. However, this new disk is unproven. It could have infant mortality, bad performance, or just some sort of compatibility problem in your environment for various reasons. You trust it less than the old proven disk (which is known to work fine) and so you really don't want to just discard the old disk once the data is fully copied to the new disk. You want the new disk to prove itself for a while before you fully trust it and you want to preserve your redundancy while that trust is being built (or lost, if there are problems).

It is generally the second disk replacement scenario where people want persistent N-way redundancy. Certainly it's where I want it. N-way redundancy during the data copy from the old drive to the new drive is not good enough, because the new drive doesn't really get proven enough during just that.

Unfortunately the second scenario probably works best with mirroring. It's my view that good RAID-[56] systems should have some way to have a component device that's actually two devices paired together, but they're unlikely to want to have this in routine operation for long.

(A RAID-[56] system that supports a true 'replace' operation needs the ability to run two disks in parallel for a while as data copies over. Ideally it would be doing reads from the new disk as well as from the old disk just in case the new disk writes fine but has problems on reads.)

Consistency and durability in the context of filesystems

Here's something that I've seen trip people up more than once when they talk about filesystems. When we talk about what guarantees a filesystem provides to programs that write data to it, we can talk about two things and the difference between them can be important.

Durability is when you write something or change the filesystem and it's still there after the system crashes or loses power unexpectedly. Durability is what you need at a high level to say 'your email has been received' or 'your file has been saved'. As everyone hopefully knows, almost no filesystem provides durability by default for data that you write to files and many don't provide it for things like removing or renaming files.

What I'll call consistency is basically that the filesystem preserves the ordering of changes after a crash. If you wrote one thing then wrote a second thing and then had the system crash, you have consistency if the system will never wind up in a state where it still has the second thing but not the first. As everyone also hopefully knows, most filesystems do not provide data consistency by default; if you write data, they normally write bits of it to disk whenever they find it convenient without preserving your order. Some but not all filesystems provide metadata consistency by default.

(Note that metadata consistency without data consistency can give you odd results that make you unhappy. Consider 'create new file A, write data to A, remove old file B'; with metadata consistency and no data consistency or forced durability, you can wind up with an empty new file A and no file B.)

Durability and consistency are connected but one does not necessarily require the other except in the extreme case of total durability (which necessarily implies total consistency). In particular, it's entirely possible to have a filesystem that has total consistency but no durability at all. Such a filesystem may rewind time underneath applications after a crash, but it will never present you with an impossible situation that didn't exist at some pre-crash point; in the 'write A, write B, crash' case, you may wind up with nothing, A only, or A and B, but you will never wind up with just B and no A.

(Because of its performance impact, most filesystems do not make selective durability of portions of the filesystem impose any sort of consistency outside of those portions. In other words, if you force-flush some files in some order, you're guaranteed that your changes to those files will have consistency but there's no consistency between them and other things going on.)

Applications not infrequently use forced flushes to create either or both of durability (the DB committed the data it told you it did) and consistency (the DB's write log reflects all changes in the DB data files because it was flushed first). In some environments, turning off durability but retaining or creating consistency is an acceptable tradeoff for speed.

(And some environments don't care about either, because the fix procedure in the face of an extremely rare system crash is 'delete everything and restart from scratch'.)

Note that journaled filesystems always maintain consistent internal data structures but do not necessarily guarantee that consistency for what you see, even for metadata operations. A journaled filesystem will not explode because of a crash but it may still partially apply your file creations, renames, deletions and so on out of order (or at least out of what you consider order). However it's reasonably common for journaled filesystems to have fully consistent metadata operations, partly because that's usually the easiest approach.

(This has some consequences for developers, along the same lines as the SSD problem but more so since it's generally hard to test against system crashes or spot oversights.)