Wandering Thoughts archives

2014-11-17

Why I need a browser that's willing to accept bad TLS certificates

One of my peculiarities is that I absolutely need a browser that's willing to accept 'bad' TLS certificates, probably for all species of bad that you can imagine: mismatched host names, expired certificates, self-signed or signed by an unknown certificate authority, or some combination of these. There are not so much two reasons for this as two levels of the explanation.

The direct reason is easy to state: lights out management processors. Any decent one supports HTTPS (and you really want to use it), but we absolutely cannot give them real TLS certificates because they all live on internal domain names and we're not going to change that. Even if we could get proper TLS certificates for them somehow, the cost is prohibitive since we have a fair number of LOMs.

(Our ability to get free certificates has gone away for complicated reasons.)

But in theory there's a workaround for that. We could create our own certificate authority, add it as a trust root, and then issue our own properly signed LOM certificates (all our LOMs accept us giving them new certificates). This would reduce the problem to doing an initial certificate load in some hacked up environment that accepted the LOMs out-of-box bad certificate (or using another interface for it, if and where one exists).

The problem with this is that as far as I know, certificate authorities are too powerful. Our new LOM certificate authority should only be trusted for hosts in a very specific internal domain, but I don't believe there's any way to tell browsers to actually enforce that and refuse to accept TLS certificates it signs for any other domain. That makes it a loaded gun that we would have to guard exceedingly carefully, since it could be used to MITM any of our browsers for any or almost any HTTPS site we visit, even ones that have nothing to do with our LOMs. And I'm not willing to take that sort of a risk or try to run an internal CA that securely (partly because it would be a huge pain in practice).

So that's the indirect reason: certificate authorities are too powerful, so powerful that we can't safely use one for a limited purpose in a browser.

(I admit that we might not go to the bother of making our own CA and certificates even if we could, but at least it would be a realistic possibility and people could frown at us for not doing so.)

web/AcceptBadCertNeed written at 23:11:39; Add Comment

2014-11-16

We've started to decommission our Solaris 10 fileservers

Our migration from our old fileservers to our new fileservers has been a slow process that's hit some rough spots. Still, we've hit a distinct point that I want to mark: this past week we reached the point where we did 'zpool destroy' on one old fileserver's old pools and powered down its backend disks.

While we can migrate rapidly when we need to, our decommissionings usually go slowly. Unless we need the rack space or the hardware for something, we mostly leave old servers and so on running until they get annoying for some reason. This makes the whole thing a gradual process, instead of the big bang that I expect some people have. In our case we actually started losing bits of the old fileserver environment almost a year ago, when we started removing bits of our old test environment. More recently we used our hot spare fileserver as a hardware donor during a crisis; we're unlikely to replace it, although we theoretically could.

(In fact we decommissioned this particular fileserver mostly because it was getting annoying. One of the disks in one of its backends failed, causing us to get plaintive email about the situation, so we decided that we wanted to power off all of the remaining disks to preserve them as potential future spares. Destroying the pools instead of just exporting them insures that the disks won't be seen as still in use by a ZFS pool if we reuse them later.)

On the one hand, this is a trivial step and the fileserver had already had its last filesystems migrated a week or so before this (and we've destroyed some migrated filesystems for various reasons, which is far more irreversible than 'zpool destroy'). On the other hand, it simply feels significant. In a way, it makes the whole thing real; we've decisively torn down what used to be a production fileserver. I can now really believe in a relatively near future where we have no Solaris 10 machines any more.

(It won't be an immediate future; given recent issues all remaining filesystem migrations to the new fileservers are probably on hold until January. In general we try not to do too much in late November and early December due to the university's long holiday around Christmas and the end of the year.)

solaris/SolarisDecommissioningStarts written at 22:18:21; Add Comment

States in a state machine aren't your only representation of state

I think in terms of state machines a lot; they're one of my standard approaches to problems and I wind up using them quite a bit. I've recently been planning out a multi-threaded program that has to coordinate back and forth between threads as they manipulate the state of host authentication. At first I had a simple set of states, then I realized that these simple states only covered the main flow of events and needed to be more and more complicated, and then I had a blinding realization:

Not all state needs to be represented as state machine states.

When you have a state machine it is not so much tempting as obvious to represent every variation in the state of your entities as another state machine state. But if you do this, then like me you may wind up with an explosion of states, many of which are extremely similar to each other and more or less handled the same way. This isn't necessary. Instead, it's perfectly sensible to represent certain things as flags, additional or detailed status fields, or the like. If you want to mark something as going to be deleted once it's unused, there's no need to add new states to represent this if you can just add a flag. If you have three or four different ways for something to fail and they all lead to basically the same processing, well, you don't need three or four different states for 'failed X way'; you can have just one 'failed' state and then another field with the details of why.

Off the top of my head now, I think that states are best for things that have a different flow of processing (ideally a significantly different flow). The more both the origin state and the processing of two 'states' resembles each other, the less they need to be separate states and the more the difference can be captured in a different variable or field (and then handled in the code with only some ifs).

(On the other hand, if two different states were handled the same way but came from different origin states and transitioned to different destination states, I think I'd definitely keep them as separate states and just share the common code somehow. This would preserve the clarity of state flow in the system. Although if two separate states needed exactly the same handling in the code, I might think I was overlooking something about the states in general.)

programming/StatesAreNotEverything written at 01:14:43; Add Comment

2014-11-15

Our current problems with 10G Intel networking on OmniOS

In my writeup on our new OmniOS fileservers I mentioned that we had built them out with 10G-T networking for their iSCSI networking (using onboard Intel X540-AT2 based ports) and their public NFS interface (using one port of a dual-port Intel 82599EB TN card). Since then, well, things have not gone so well and in fact we're in the process of moving all production fileservers to 1G networking until we can understand what's going on and we can fix it.

The initial problems involved more or less total server lockups on our most heavily used fileserver. Due to some warning messages on the console and previous weird issues with onboard ports, we added a second dual-port card and moved the iSCSI networks to them. We also had iSCSI networking issues on two other servers, one of which was also switched to use a second dual-port card for iSCSI networking.

(At this point the tally is two fileservers using the onboard ports for 10G iSCSI and two fileservers using second dual-port cards for it.)

The good news is that the fileservers mostly stopped locking up at this point. The bad news is that both actively used dual-port cards wound up getting themselves into a state where the ixgbe driver couldn't talk properly to the second port and this had very bad effects, most specifically an extremely long lock hold time with spinlocks. At first we saw this only with the first card that had been replaced, on our most-used fileserver, so it was possible for me to believe that this was just a hardware fault (after all, the second port was working fine on the less used fileserver). Today we had exactly the same issue appear on the other fileserver, so it seems extremely likely that there is some combination of a driver bug and a hardware bug involved, one that is probably more and more likely to manifest as you pass more traffic through the ports.

(On top of that problem, we also found a consistent once a second 20msec lock hold time and stall in the ixgbe driver when dealing with those onboard X540-AT2 ports. Interested parties are directed to this email to the illumos-developer mailing list for full details about both issues. Note that it was written when I still thought the big stall might be due to faulty hardware on a single card.)

My understanding is that the Illumos (and thus OmniOS) ixgbe driver is derived from an upstream general Intel driver through a process that must be done by hand and apparently has not happened for several years. At this point enough bad stuff has shown up that I don't think we can trust the current OmniOS version of the driver and we probably don't want to try Intel 10G-T again until it's updated. Unfortunately I don't have any idea if or when that will happen.

(It also seems unlikely that we'll find any simple or quick reproduction for any of these problems in a test environment. My suspicion is that the dual-port issue is due to some sort of narrow race window involving hardware access, so it may depend not just on total traffic volume but on the sort of traffic you send.)

solaris/OmniOS10GIntelProblems written at 01:10:10; Add Comment

2014-11-14

Sometimes there are drawbacks to replicating configuration files

This is a war story, but not my war story; this is all my coworkers' work.

Writing a working Samba configuration is a lot of painful work. There are many options, many of them interact with clients in odd and weird ways, and the whole thing often feels like a delicately balanced house of cards. As a result we we have a configuration that we've painstakingly evolved over the many years that we've been using Samba. When we needed a second Samba server dedicated to a particular group but still using our NFS fileservers we of course copied the file, changed the server name, and used it as is. Starting from scratch would have been crazy; our configuration is battle-tested and we know it works.

We recently built out an infrastructure for cheap bulk storage, originally intended for system backups; the core idea is that people buy some number of disks, give them to us, and we make them accessible via Time Machine (for Macs) and Samba (for Windows). Of course we set up this machine's Samba using our master Samba configuration (again with server names changed, and this time around with a lot of things taken out because eg this server doesn't support printing). Recently we discovered that Samba write performance on this server was absolutely and utterly terrible (we're taking in the kilobytes or very small megabytes a second range). My coworkers chased all sorts of worrysome potential causes and wound up finding it in our standard smb.conf, which had the following lines:

# prevent Windows clients copying files to
# full disks without warning. This can lead
# to data loss.

strict sync = yes
sync always = yes

Surprisingly, when you tell your Samba server to fsync() your writes all the time your write performance on local disks turns out to be terrible. Performance was okay on our main Samba servers for complex reasons involving our NFS servers.

The comment explains the situation we ran into fairly well; Windows clients copying files from the local disk to a Samba disk could run out of space on the filesystem backing the Samba disk, have the write fail, not notice, and delete the local file because it 'copied'. That was very bad. Forcing syncs flushed the writes from the Samba server to the NFS fileserver and guaranteed that if the fileserver accepted them there was space in the filesystem (and conversely that if you were out of space the Samba server knew before it replied to the client). All of this is perfectly rational; we ran into a Samba issue, found some configuration options that fixed it, put them in, and even documented them.

(Maybe there are other configuration options that would have fixed this problem and maybe this problem is not an issue any more on current versions of Samba and everything else in our environment, but remember what I said about us not rewriting Samba configuration files because they're a house of cards.)

This whole thing is a nice illustration of the downside of replicating configuration files when you're setting up new services. Not starting from scratch is a lot faster and may well save you a lot of painful bad experiences, but it can let things slip through that have unpleasant side effects in a new environment. And it's not like you can really avoid this problem without starting from scratch; going through to question and re-validate every configuration setting is almost certainly too mind-numbing to work. Plus there's no guarantee that even a thorough inspection would have caught this issue, since the setting looks perfectly rational unless you've got the advantage of hindsight.

sysadmin/CopyingConfigsDrawback written at 00:54:21; Add Comment

2014-11-13

I want opportunistic, identity-less encryption on the Internet

The tech news has recently been full of reports that some ISPs are modifying SMTP conversations passing through them in order to remove encryption, or more specifically to remove a note that the server is offering it (this goes by the name of 'STARTTLS'). Some people are up in arms over it; others are saying that this is no big deal because it's not as if TLS on SMTP conversations means much in the first place. This latter viewpoint may raise some eyebrows. The reason why TLS SMTP means less than you think is that essentially no clients require signed and verified TLS certificates from the SMTP server; instead they'll almost always accept any random TLS certificate. In one set of jargon, this is called 'opportunistic' encryption; you could also call it 'unauthenticated' encryption. Its drawback is that this lack of authentication of the server means that the server could be anyone, including an attacker playing man in the middle to read your 'TLS encrypted' email.

This general issue is a long-standing dispute in Internet cryptography. On one side are people who say that opportunistic encryption is better than nothing; on the other side are people who say that it's effectively nothing because of its vulnerability to MITM attacks (and that it has other bad side effects, like causing people to think that they're more secure than they are). Once upon a time I might have been reasonably sympathetic to the second view, but these days I have come completely around to the first view.

As far as the second view goes: yes, there are many places that are perfectly happy to perform MITM attacks on you at the drop of a hat. Most such places generally don't call them MITM attacks, of course; instead they're 'captive portals' and so on. Those ISPs that are stripping STARTTLS markers are effectively doing a MITM attack. But in practice, in the real world, this is not equivalent to having no encryption at all. The big difference between opportunistic encryption and no encryption is that opportunistic encryption completely defeats passive monitoring. And in the real world, passive monitoring is now pervasive.

That is why I want as much opportunistic encryption as possible; I want that pervasive passive monitoring to get as little as possible. Sure, in theory the opportunistic encryption could be MITMd. In practice no significant amount of it is likely to be, because the resources required to do so go up much faster than with passive monitoring (and in some situations so do the chances of getting noticed and caught). Opportunistic encryption is not theoretically clean but it is practically useful, and it is much, much easier to design in and set up than authenticated encryption is.

(And this is why ISPs stripping STARTTLS matters quite a bit.)

tech/CasualInternetEncryptionWant written at 00:57:03; Add Comment

2014-11-12

A wish: setting Python 3 to do no implicit Unicode conversions

In light of the lurking Unicode conversion issues in my DWiki port to Python 3, one of the things I've realized I would like in Python 3 is some way to turn off all of the implicit conversions to and from Unicode that Python 3 currently does when it talks to the outside world.

The goal here is the obvious one: since any implicit conversion is a place where I need to consider how to handle errors, character encodings, and so on, making them either raise errors or produce bytestrings would allow me to find them all (and to force me to handle things explicitly). Right now many implicit conversions can sail quietly past because they're only having to deal with valid input or simple output, only to blow up in my face later.

(Yes, in a greenfield project you would be paying close attention to all places where you deal with the outside world. Except of course for the ones that you overlook because you don't think about them and they just work. DWiki is not in any way a greenfield project and in Python 2 it arrogantly doesn't use Unicode at all.)

It's possible that you can fake this by setting your (Unix) character encoding to either an existing encoding that is going to blow up on utf-8 input and output (including plain ASCII) or to a new Python encoding that always errors out. However this gets me down into the swamps of default Python encodings and how to change them, which I'm not sure I want to venture into. I'd like either an officially supported feature or an easy hack. I suspect that I'm dreaming on the former.

(I suspect that there are currently places in Python 3 that always both always perform a conversion and don't provide an API to set the character encoding for the conversion. Such places are an obvious problem for an official 'conversion always produces errors' setting.)

python/Python3NoImplictUnicodeOption written at 01:31:01; Add Comment

2014-11-10

Why I don't have a real profile picture anywhere

Recently I decided that I needed a non-default icon aka profile picture for my Twitter account. Although I have pictures of myself, I never considered using one; it's not something that I do. Mostly I don't set profile pictures on websites that ask for them and if I do, it's never actually a picture of me.

Part of this habit is certainly that I don't feel like giving nosy websites that much help (and they're almost all nosy). Sure, there are pictures of me out on the Internet and they can be found through search engines, but they don't actually come helpfully confirmed as me (and in fact one of the top results right now is someone else). Places like Facebook and Twitter and so on are already trying very hard to harvest my information and I don't feel like giving them any more than the very minimum. For a long time that was all that I needed and all of the reason that I had.

These days I have another reason for refusing to provide a real picture, one involving a more abstract principle than just a reflexive habit towards 'none of your business' privacy. Put simply, I don't put up a profile picture because I've become conscious that I could do so safely, without fear of consequences due to people becoming aware of what I look like. Seeing my picture will not make people who interact with me think any less of me and the views I express. It won't lead to dismissals or insults or even threats. It won't expose me to increased risks in real life because people will know what I look like if they want to find me.

All of this sounds very routine, but there are plenty of people on the Internet for whom this is at least not a sure thing (and thus something that they have to consider consciously every time they make this choice) or even very much not true. These people don't have my freedom to casually expose my face and my name if I feel like it, with no greater consideration than a casual dislike of giving out my information. They have much bigger, much more serious worries about the whole thing, worries that I have the privilege of not even thinking about almost all of the time.

By the way, I don't think I'm accomplishing anything in particular by not using a real picture of myself now that I'm conscious of this issue. It's just a privilege that I no longer feel like taking advantage of, for my own quixotic reasons.

(You might reasonably ask 'what about using your real name?'. The honest answer there is that I am terrible with names and that particular ship sailed a very long time ago, back in the days before people were wary about littering their name around every corner of the Internet.)

PS: One obvious catalyst for me becoming more aware of this issue was the Google+ 'real names' policy and the huge controversy over it, with plenty of people giving lots of excellent arguments about why people had excellent reasons not to give out their real names (see eg the Wikipedia entry if you haven't already heard plenty about this).

PPS: Yes, I have plenty of odd habits related to intrusive websites.

web/WhyNotProfilePictures written at 23:59:07; Add Comment

What it took to get DWiki running under Python 3

For quixotic reasons I recently decided to see how far I could get with porting DWiki (the code behind this blog) to Python 3 before I ran out of either patience or enthusiasm. I've gotten much further than I expected; at this point I'm far enough that it can handle this entire site when running under Python's builtin basic HTTP server, rendering the HTML exactly the same as the Python 2 version does.

Getting this far basically took three steps. The largest step was updating the code to modern Python 2, because Python 3 doesn't accept various bits of old syntax. After I'd done that, I ran 2to3 over the codebase to do a bunch of mechanical substitutions, mostly rewriting print statements and standard modules that had gotten reorganized in the transition. The final necessary step was some Unicode conversion and mangling (and with it reading some files in binary mode).

All of this sounds great, but the reality is that DWiki is only limping along under Python 3 and this is exactly because of the Unicode issue. Closely related to this is that I have not revised my WSGI code for any changes in the Python 3 version of WSGI (I'm sure there must be some, just because of character encoding issues). Doing a real Python 3 port of DWiki would require dealing with this, which means going through everywhere that DWiki talks to the outside world (for file IO, for logging, and for reading and replying to HTTP requests), figuring out where the conversion boundary is between Unicode and bytestrings, what character encoding I need to use and how to recognize this, and finally what to do about encoding and decoding errors. Complicating this is that some of these encoding boundaries are further upstream than you might think. Two closely related cases I've run into so far is that DWiki computes the ETag and Content-Length for the HTTP reply itself, and for obvious reasons both of these must be calculated against the encoded bytestring version of the content body instead of its original Unicode version. This happens relatively far inside my code, not right at the boundary between WSGI and me.

(Another interesting case is encoding URLs that have non-ASCII characters in them, for example from a page with a name that has Unicode characters in it. Such URLs can get encoded both in HTML and in the headers of redirects, and need to be decoded at some point on the way in, where I probably need to %-decode to a bytestring and then decode that bytestring to a Unicode string.)

Handling encoding and decoding errors are a real concern of mine for a production quality version of DWiki in Python 3. The problem is that most input these days is well behaved, so you can go quite a while before someone sends you illegal UTF-8 in headers, URLs, or POST bodies (or for that matter sends you something in another character set). This handily disguises failures to handle encoding and decoding problems, since things work almost all the time. And Python 3 has a lot of places with implicit conversions.

That these Unicode issues exist doesn't surprise me. Rather the reverse; dealing with Unicode has always been the thing that I thought would be hardest about any DWiki port to Python 3. I am pleasantly surprised by how few code changes were required to get to this point, as I was expecting much more code changes (and for them to be much more difficult to make, I think because at some point I'd got the impression that 2to3 wasn't very well regarded).

Given the depths of the Unicode swamps here, I'm not sure that I'll go much further with a Python 3 version of DWiki than I already have. But, as mentioned, it is both nice and surprising to me that I could get this far with this little effort. The basics of porting to Python 3 are clearly a lot less work than I was afraid of.

python/DWikiAndPython3 written at 01:02:16; Add Comment

2014-11-09

NFS hard mounts versus soft mounts

On most Unix systems NFS mounts come in your choice of two flavours, hard or soft. The Linux nfs manpage actually has a very good description of the difference; the short summary is that a hard NFS mount will keep trying NFS operations endlessly until the server responds while a soft NFS mount will give up and return errors after a while.

You can find people with very divergent opinions about which is better (cf, 2). My opinion is fairly strongly negative about soft mounts. The problem is that it is routine for a loaded NFS server to not respond to client requests within the client timeout interval because the timeout is not for the NFS server to receive the request, it's for the server to fully process it. As you might imagine, a server under heavy IO and network load may not be able to finish your disk IO for some time, especially if it's write IO. This makes NFS timeouts that would trigger soft NFS mount errors a relatively routine event in many real world environments.

(On Linux, any time a client reports 'nfs: server X not responding, still trying' that would be an IO error on a soft NFS mount. In our fileserver environment, some of these happen nearly every day.)

Many Unix programs do not really expect their IO to fail. Even programs that do notice IO errors often don't and can't do anything more than print an error message and perhaps abort. This is not a helpful response to transient errors, but then Unix programs are generally not really designed for a world with routine transient IO errors. Even when programs report the situation, users may not notice or may not be prepared to do very much except, perhaps, retry the operation.

(Write errors are especially dangerous because they can easily cause you to permanently lose data, but even read errors will cause you plenty of heartburn.)

Soft NFS mounts primarily make sense when you have some system that absolutely must remain responsive and cannot delay for too long for any reason. In this case a random but potentially very long kernel imposed delay is a really bad thing and you'd rather have the operation error out entirely so that your user level code can take action and at least respond in some way. Some NFS clients (or just specific NFS mounts) are only used in this way, for a custom system, and are not exposed to general use and general users.

(IO to NFS hard mounts can still be interrupted if you've sensibly mounted them with the intr option. It just requires an explicit decision at user level that the operation should be aborted, instead of the kernel deciding that all operations that have taken 'too long' should be aborted.)

PS: My bias here is that I've always been involved in running general use NFS clients, ones where random people will be using the NFS mounts for random and varied things with random and varied programs of very varied quality. This is basically a worst case for NFS soft mounts.

unix/NFSHardVsSoft written at 00:41:15; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.