Wandering Thoughts archives

My annoyance with Chrome's cut and paste support under X

In the past I've vaguely alluded to having problems getting xterm and Chrome to play nicely together for selecting text in xterm and pasting it into Chrome (eg here). At the time I couldn't clearly reproduce my problems, so I wrote it off as due to fallible memory. Well, my memory may be fallible but I also wasn't wrong about Chrome not playing nice with xterm's normal selections. It definitely happens and it's one of the real irritations with using Chrome for me.

Certainly, sometimes I can select text in xterm and paste it into a HTML form field in Chrome with the middle mouse button. But not always. I'm not sure if it most frequently happens with the fields involved in login forms, or if that's just my most common use of moving text from xterm to Chrome. Of course, being unable to paste a complex random password is very irritating. As a result, I basically always resort to my way of making xterm do real Copy and Paste; this is longer, but so far it's always worked.

(Instead of 'select, move mouse over, paste with middle mouse button', it is now 'select, hit Ctrl-Shift-C, move mouse over, use right mouse button to bring up Chrome field menu, select Paste'. I could save a bit of time by remembering that Ctrl-V is paste in Chrome, but it doesn't yet quite seem worth doing that.)

Now that I look this seems to be a known issue, based on Chromium bug #68886. That's been open since 2011, so I doubt things are going to change any time soon. I guess maybe I should memorize Ctrl-V.

(I know, this sounds petty. But sometimes it's the little things that really get under my skin and rob me of confidence in a program. If I don't understand when and why Chrome is not accepting text I'm trying to paste in from xterm, what other unpleasant things are lurking in its behavior that I just haven't stumbled into yet because I don't (yet) use it enough?)

Bad slide navigation on the web and understanding why it's bad

As usual, I'll start with my tweet:

If the online copy of your slide presentation is structured in 2D, not just 'go forward', please know that I just closed my browser window.

This is sort of opaque because of the 140 character limitation, so let me unpack it.

People put slide decks for their presentations online using various bits of technology. Most of the time how you navigate through those decks is strictly linear; you have 'next slide' and 'previous slide' in some form. But there's another somewhat popular form I run across every so often, where the navigation down in the bottom right corner offers you a left / right / up / down compass rose. Normally you go through the slide deck by moving right (forward), but some slides have more slides below them so you have to switch to going down to the end, then going right again.

These days, I close the browser window on those slide presentations. They're simply not worth the hassle of dealing with the navigation.

There are a number of reasons why this navigation is bad on the web (and probably in general) beyond the obvious. To start with, there's generally no warning cue on a slide itself that it's the top of an up/down stack of slides (and not all slides at the top level are). Instead I have to pay attention to the presence or absence of a little down arrow all the way over on the side of the display, well away from what I'm paying attention to. It is extremely easy to miss this cue and thus skip a whole series of slides. At best this gives me an extremely abbreviated version of the slide deck until I realize, back up, and try to find the stacks I missed.

This lack of cues combines terribly with the other attribute of slides, which is that good slides are very low density and thus will be read fast. When a slide has at most a sentence or two, I'm going to be spending only seconds a slide (I read fast) and the whole slide deck is often a stream of information. Except that it's a stream that I can't just go 'next next next' through, because I have to stop to figure out what I do next and keep track of whether I'm going right or down and so on. I'm pretty sure that on some 'sparse slides' presentations this would literally double the amount of time I spend per slide, and worse it interrupts the context of my reading; one moment I'm absorbing this slide, the next I'm switching contexts to figure out where to navigate to, then I'm back to absorbing the next slide, then etc. I get whiplash. It's not a very pleasant way to read something.

Multi-option HTML navigation works best when it is infrequent and clear. We all hate those articles that have been sliced up into multiple pages with only a couple of paragraphs per page, and it's for good reason; we want to read the information, not navigate from page to page to page. The more complicated and obscure you make the navigation, the worse it is. This sort of slide presentation is an extreme version of multi-page articles with less clear navigation than normal HTML links (which are themselves often obscured these days).

I don't think any of this is particularly novel or even non-obvious, and I sure hope that people doing web design are thinking about these information architecture issues. But people still keep designing what I think of as terribly broken web navigation experiences anyways, these slide decks being one of them. I could speculate about why, but all of the reasons are depressing.

(Yes, including that my tastes here are unusual, because if my tastes are unusual it means that I'm basically doomed to a lot of bad web experiences. Oh well, generally I don't really need to read those slide decks et al, so in a sense people are helpfully saving my time. There's an increasing number of links on Twitter that I don't even bother following because I know I won't be able to read them due to the site they're on.)

Sidebar: Where I suspect this design idea got started

Imagine a slide deck where there you've added some optional extra material at various spots. Depending on timing and audience interest, you could include some or all of this material or you could skip over it. This material logically 'hangs off' certain slides (in that between slide A and F there are optional slides C, D, and E 'hanging off' A).

This slide structure makes sense to represent in 2D for presentation purposes. Your main line of presentation (all the stuff that really has to be there) is along the top, then the optional pieces go below the various spots they hang off of. At any time you can move forward to the next main line slide, or start moving through the bit of extra material that's appropriate to the current context (ie, you go down a stack).

Then two (or three) things went wrong. First, the presentation focused structure was copied literally to the web for general viewing, when probably it should be shifted into linear form. Second, there were no prominent markers added for 'there is extra material below' (the presenter knows this already, but general readers don't). Finally, people took this 2D structure and put important material 'down' instead of restricting down to purely additional material. Now a reader has to navigate in 2D instead of 1D, and is doing so without cues that should really be there.

Why your Apache should have mod_status configured somewhere

Recently, our monitoring system alerted us that our central web server wasn't responding. I poked it and indeed, it wasn't responding, but when I looked at the server everything seemed okay and the logs said it was responding to requests (a lot of them, in fact). Then a little bit later monitoring said it was responding again. Then it wasn't responding. Then my attempt to look at a URL from it worked, but only really slowly.

If you're a long-term Apache wrangler, you can probably already guess the cause. You would be correct; what was going on was that our Apache was being hit with so many requests at once that it was running out of worker processes. If it got through enough work in time, it would eventually pick up your request and satisfy it; if it didn't, you timed out. And if you were lucky, maybe you could get a request in during a lull in all the requests and it would be handled right away.

Once we'd identified the overall cause, we needed to know who or what was doing it. Our central web server handles a wide variety of URLs for a lot of people, some of which can get popular from time to time, so there were a lot of options. And nothing stood out in a quick scan of the logs as receiving a wall of requests or the like. Now, I'm sure that we could have done some more careful log analysis to determine the most active URLs and the most active sources over the last hour or half hour or something, but that would have taken time and effort and we still might have missed sometime. Instead I took the brute force approach: I added mod_status to the server's configuration, on a non-standard URL with access restrictions, and then I looked at it. A high volume source IP jumped out right away and did indeed turn out to be our problem.

Apache's mod_status has a bad reputation as an information leak and a security issue, and as a result I think that a lot of people don't enabled it these days. Our example shows why you might want to reconsider that. Mod_status offers information that's fairly hard to get in any other way and that's very useful (or essential) when you need it, and it's definitely possible to enable it securely. Someday you will want to know who or what is bogging down your server (or at least what it's doing right now), and a live display of current requests is just the thing to tell you.

(This should not be surprising; live status is valuable for pretty much anything. Even when this sort of information can be approximated or reconstructed from logs, it takes extra time and effort.)

Why I think Let's Encrypt won't be a threat to commercial CAs for now

One of the questions these days is what effects Let's Encrypt and their well publicized free TLS certificates will have on existing commercial CAs. While LE has become a very popular CA and there have been some amusing signs of nervousness from existing CAs, my personal belief is that Let's Encrypt is not going to be a threat to the business of commercial CAs for the medium term, say the next few years.

One reason I say this is pragmatic past history. LE is not the first or only free SSL CA, and that existing free SSL CA does not seem to have made any particularly large dent in the business of other CAs. Now, there are certainly some differences between LE and this other CA that makes this not quite an apples to apples situation; LE is very well known now and issues short-duration certificates through easy automation, while the other free CA is less well known and issues year long certificates through a more annoying and onerous web-based process that not everyone can take advantage of. However, I think it's still a sign of some things.

(The other free CA also charges for certificate revocation or early replacement, which did not endear them to a lot of people during Heartbleed.)

Beyond that, there are a number of potential reasons that Let's Encrypt certificates are not necessarily all that compelling for many organizations. In no particular order:

• LE doesn't issue wild card certificates.
• Even with their recently raised rate limits, 20 new certificates a week is still potentially a real limit for decent sized organizations (who can have a lot of hosts and subdomains they want TLS certificates for).
• The short duration of LE certificates pretty much requires developing and deploying new automation to automatically renew and roll over LE certificates. With commercial CAs, a moderate sized place can just buy some certificates and be done for the next few years, with minimal staff time required.

  (At least some CAs will even mail you helpful reminder emails when your certificates are approaching their expiry times.)

Finally, as a commodity basic TLS DV certificates are already available for very little money. It's my impression that a lot of organizations are not that price sensitive about TLS certificates; they simply don't care that they could save $10 or $20 a year per certificate, because the aggregate savings are not worth thinking about at their size. They might as well go with whatever is either simpler for them now or already familiar. Let's Encrypt is compelling for the price sensitive hobbyist, but for a decent sized organization I think it's only interesting for now.

(And for a large organization, the rate limits probably make it infeasible to rely on LE for new certificates.)

However, I do expect this to change over the longer term. What will really threaten existing CAs is the addition of turn-key integration of LE certificates in more and more pieces of software. Once you can set an option in your web server of choice to automatically get and maintain LE certificates for each virtual host you have or add, I think that a lot of people in a lot of organizations will just turn that option on rather than wrestle with all of the little hassles of commercial CAs. Once LE becomes the easiest choice, existing CAs have much less to offer to anyone who fits into LE's rate limits (basically CAs are down to 'we'll sell you wild-carded DV certificates').

(I'm ignoring EV certificates as another thing entirely, but if CAs feel actively threatened by LE I expect them to start trying to push EV certificates much more strongly. One obvious danger sign would be if CAs start trying to persuade browser vendors to restrict certain JavaScript or browser capabilities to only EV-validated sites instead of all HTTPS sites.)

(This is sort of an elaboration of a comment I left on Tanguy Ortolo's entry about this issue (via Planet Debian).)

Why I don't use HTTP Key Pinning and I'm not likely to any time soon

One of the additional security measures you can take for a HTTPS web site is HTTP Public Key Pinning. HPKP is designed to guard against the fundamental SSL CA problem by limiting what validly signed keys can be used for your website. This way it doesn't matter if an attacker gets one of the zillion different CAs out there to issue a certificate for your website; hopefully a lot of browsers won't accept it anyways because you've told them better. A number of people like the idea of HPKP and are sad that it is used only by a trace number of websites out there.

I have looked a bit at HPKP and I'm not going to use it any time soon (either on my personal sites or at work). Fundamentally HPKP has several problems for most people. First, it's relatively hard to even set up HPKP so that it works. To simplify, a valid HPKP configuration requires you to have either a second spare TLS certificate or a second CA that you're confident you'll get a TLS certificate from if you need another one. If you go with a spare TLS certificate, you must make sure this TLS certificate is always recoverable; if you lose both your primary and your secondary certificates, you are off the air until your HPKP pinning times out. With the second CA, you must be absolutely sure that you can get a TLS certificate from the CA if you need it, and of course you're vulnerable if an attacker can get a TLS certificate for you from that specific CA.

(Note that as far as I know, you cannot specify 'this current TLS certificate or any certificate from my current CA'; your backup key must be completely independent from your current one. Also technically you don't need your secondary TLS certificate to be issued and signed by a CA, just to be pregenerated and known. This may give you some freedom. And you can specify more than one backup pin.)

Measurements in the field indicate that a lot of people get this one wrong (here's one source). Either they serve HPKP headers with only one pin or they provide two pins that aren't fully independent.

The second problem is that HPKP is very dangerous, and the more it protects you the more dangerous it is. If you set up a valid HPKP configuration and then lose both your live pin and your backup pin, you've bricked your site for people who have your HPKP information saved. The more specific your pins are and the longer you advertise them as valid, the more you're protected but also the easier it is to lose them and the longer the damage from a loss will last.

The final problem with HPKP for most sites is that it protects against what is now a very rare event that's quite risky for the attacker. It's vanishingly unlikely that your typical HTTPS site is going to be targeted by someone who can generate a valid TLS certificate for it and is willing to (quite possibly) burn a compromised CA by attacking you. This makes the reward of deploying HPKP very small compared to the risks.

If you're a big high risk site that attackers would get a big payoff from compromising, sure, HPKP makes a lot of sense. But then if you're this sort of site, you've probably already done things like ask the Chrome team to pre-pin your TLS certificates. If you'd never even dream of bugging the Chrome team for that (partly because you're too small and partly because it's a massive risk if something goes wrong), strong HPKP doesn't really make much sense.

(It might make sense to deploy reporting-only HPKP, but then you have to find or set up a reporting endpoint and then monitor things for reports, which should normally be big alerts (assuming you got your HPKP pinning right). Even with services like this, that's a fair amount of work for a fairly low likelihood risk.)

(Note that HPKP pinning can get complicated fast, since you can pin more than your current key plus one backup. With my current low level of knowledge, I'm not going to try to say anything more than that.)

Some things I believe about importance and web page design

Here are some things that I believe about web page design.

If you have a bright red element on a page that does not otherwise use red as part of its colour scheme, this element should not merely be the most important thing on the page, it should be so important that the reader must look there first and foremost. In at least the west, we're habituated to red as an alert colour, probably the primary alert colour. A solitary red thing is practically screaming at you, jumping up and down and demanding your attention. It should be worth it and really, it had better be.

If you have a bright red element that you lock as a floating element so that it's always present even when the page scrolls, this element should be so important that the reader needs to always pay attention to it. A constantly present red element is a constant alert yelling at you. I actually tend to think that if it is so important, maybe there shouldn't be anything else on the page (or at least no other content).

(In fact, in general any always present element should be very important. Screen space is a precious resource on at least mobile devices (which are an increasing amount of web browsing), so any space you reserve for headers, footers, sidebars, and so on is space that shrinks an already small content area even smaller. People with small screens can get quite irritated about this and yes, they would much rather scroll up to get back to your navigation than have it take up a quarter or a fifth of their screen all the time.)

If you have or are tempted to have a red element on your pages, especially an always visible one, I very strongly think that you should be considering whether it really is that important. Is it an alert that visitors should be paying some amount of attention to all the time, to the degree that some of them may find the result unreadable? Is it more important than the actual content of the web page? Or is it perhaps somewhat less important than that, and so maybe it should not be red, or always visible, or especially not both at once.

(My personal view is that there is very little information that is that important. Your web pages are hopefully about your content, after all, at least for normal websites. About the only case for such an all-consuming alert that I can come up with is content that is actually now dangerous and is being retained only for historical reasons; here, people not reading the content is a feature instead of a bug.)

In not unrelated news, the PowerDNS docs web pages for the 3.x release currently look like this [png]; 3.x is not the latest release, but it is one that is still very much out in the field (it's in Ubuntu 14.04 LTS and Fedora 23, for example). I personally do not consider 'you are reading the 3.x documentation, here is the latest documentation' to be the most important and pretty much all-consuming thing on the page, but it's not my website.

(Nor will I be even attempting to send in a patch, because the only patch I would write is one that deleted the entire 'this is older documentation' alert. There is no possible way of fixing this within anything that looks and acts like the current setup, and I have to assume that the people who created the alert feel that it is really that important. In that case we have a disagreement not about presentation, which can be patched with HTML and CSS, but about web page information architecture.)

I've started using the Firefox addon Self-Destructing Cookies for some stuff

I previously wrote about two models for dealing with cookies in Firefox and how I was someone who followed the 'never accept' model while Self-Destructing Cookies followed the 'accept then remove' model. Well, as it happens I've recently started using SDC in some of my Firefox profiles, for what I think of as a good reason. Namely, it's really easy to get started with SDC.

My main browser is not logged in to sites, in addition to basically not accepting cookies. For a few sites I use regularly or want convenient fast access to, this is inconvenient enough that I maintain a separate Firefox profile; this profile is set up to retain cookies so that I stay logged in. I recently realized that the simple setup of these profiles meant that I was retaining all cookies, not just the cookies for the site I care about. When I actually looked, I found that I'd built up quite a collection of tracker cookies that I really didn't like.

I could have reacted to this by setting up a strong anti-cookie addon in those profiles and adopting a default-deny policy. But while I can clearly identify what site I have to allow for my login cookies to work, some of these places may need side cookies and the like. I decided that I didn't really have the energy to tackle this; I wanted a simple solution that was easy to enact and almost surefire. This is a situation that Self-Destructing Cookies is ideal for. I could install it, tell it to permanently keep cookies from the main site, and be done.

So far I'm quite happy with using SDC this way. It's been a no fuss, no muss way of choking off a great deal of cookie abuse at minimal cost in time and attention.

(As with almost all Firefox addons, I've found that I have to tell it to be quiet about notifications. For some reason, many addons assume that I'm eager to hear about everything they're doing and blocking for me. This is very much not the case. I'm perfectly well aware that the web is a horrorshow of trackers and privacy abuse, and I don't care about the details of who is abusing me this time.)

(Because this is for profiles that I don't keep running all the time, I don't know if SDC leaks memory in normal ongoing use, as all too many addons do.)

Firefox addons seem unfortunately prone to memory leaks

As is now traditional, I'll start with my tweets:

I look forward to a hypothetical day when it's difficult for Firefox extensions to leak memory like sieves. That day is not today.
It's depressing that at least half the time I audition new Firefox extensions, they immediately cause my session to bloat up madly.

It's like clockwork by now; I'll decide to try out a new extension (this time around I believe it was Cookie Monster), and at most a few hours later I'll see that my Firefox has monstered out into high and steadily growing memory usage. So much for that extension.

I have to assume that this is not a universal experience with Firefox addons, especially popular ones like Greasemonkey and Stylish. Clearly there's something unusual about either how I use Firefox or how my collection of addons interact with each other that makes me especially prone to it. I don't think it's just leaving Firefox running with pages open, because I'm sure that at least some people these days leave tabs permanently open on GMail or Twitter or Slack or Facebook or any number of other 'constant on' sites. Perhaps it's my use of windows over tabs, or combining multiple windows with some having tabs, but even that feels stretched.

(I don't think it's using Firefox Nightly either; I've tried out extensions in a stock Firefox and observed bloat symptoms there if I push it.)

It would be nice if Firefox Electrolysis would fix this, because then I'd really get something for the problems it's going to bring. But I don't actually believe that that's going to happen, and in fact I expect e10s to make my situation worse, because it'll undoubtedly cause me to have to change some extensions (and thus hunt for ones that not only work but don't leak).

(It's possible that e10s could help here. Given the pervasiveness of addon memory leaks, there are clearly Firefox addon APIs that are hard to use without leaking memory. If e10s replaced them with more leak-free APIs, we could win. But I don't really expect this because I don't think this is a core part of Electrolysis's purpose.)

Incidentally, this is one reason I am so grumpy when Firefox does something that breaks one or more of my extensions. My current set of extensions is extremely carefully chosen to not just work but also to not leak memory. Finding replacements with both attributes is both a bunch of work and also not in any way guaranteed. I've certainly had to drop addons in the past when they leaked and there was no non-leaking replacement.

(I suppose for big addons like Greasemonkey I should try to find their bug reporting system and report the issue, just on the odd chance that this gets things fixed. But making bug reports is exhausting.)

Turning over a rock on some weird HTTP requests to our web server

I recently made the mistake of looking at our Apache access.log, and in fact watching it live with 'tail -f'. Me being me, I can't just let what I saw sit quietly, so now I'm here to tell you about the big weirdness I saw. Put simply, it was a whole rapid burst of requests that looked like:

IP - - [28/Feb/2016:17:18:38 -0500] "GET /mmievslc.txt HTTP/1.1" 404 [...]
IP - - [28/Feb/2016:17:18:39 -0500] "GET /mmievslc.txt HTTP/1.1" 404 [...]
IP - - [28/Feb/2016:17:18:39 -0500] "GET /mmievslc.txt HTTP/1.1" 404 [...]

When I started digging, I saw multiple IPs making requests like this for multiple different 8-character .txt URLs in the root of our web server (none of which have ever existed). On random spot checks, they almost all happen in bursts (although there can be pauses), and there are a lot of them.

How many? Yesterday, we saw 34,500 such requests (about 10% of the total HTTP requests), from 116 different IPs and for 122 different names. The top three IPs all made over 1000 requests each; the median made 233 requests. Every such request had the same user-agent:

"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

(Only 50 other requests from 13 different IPs used this user-agent.)

On a random spot check of IP addresses doing this, I can't find any that aren't in China. Some but not many of the IP addresses are listed in things like the SBL; others claim to be entirely clean in my blocklist checks.

I spot-checked the IPs doing this yesterday against the IPs doing this today and about two thirds of them are different; checking yesterday against the day before yielded the same result. So there seems to be a different set of sources doing this over time.

We have multiple virtual hosts on this web server, and only two of them are affected; the main departmental web server name and another one (which saw far less volume of these requests). There's nothing obvious that's different between unaffected hosts and affected ones.

And what makes this really mysterious is I have no idea what these requests are supposed to accomplish. Are they an attack of some sort? Are they an accidental side effect of other software? Are they being done deliberately in order to create some sort of useful side effect? Are they traffic cloaking or obfuscation of some sort? Who knows. I may have turned over this rock, but I have no idea how to understand what's scuttling around underneath it.

Mozilla, Symantec, SHA-1 certificates, and the balance of power

In theory, all CAs are supposed to have stopped issuing SHA-1 certificates on January 1st. In Payment Processors Still Using Weak Crypto (via), Mozilla has now announced that they will allow Symantec to issue a limited number of SHA-1 certificates. The reactions I've seen are reasonably harsh. While I don't entirely disagree, I have an additional cynical perspective that's based on the balance of power between CAs and browsers.

Let us be blunt here: Symantec wants to issue these certificates. They are undoubtedly getting to charge a large amount of money for them (especially under the circumstances) and we have plenty of evidence that many CAs do not care one whit about phasing out SHA-1. Browser demands to the contrary are an irritating distraction from the important business of taking money for random numbers.

Browser people are caught in a difficult three-way situation. On the one hand, they hold significant power given the only purpose of most TLS certificates is to display a lock in the browser. On the other hand, browsers are generally commodities themselves. If a browser stops working for you, which includes 'letting you browse HTTPS sites that you want to browse', most people are going to switch to one that does work. The result is that browsers are engaged in a giant game of CA chicken with each other, much like XHTML chicken and DRM chicken. If all browsers remove a popular CA for violating policies, all is fine. But if one or more browsers blink and do not do so, the remaining strict browsers lose; some decent amount of their users will find important sites not working and move one of the browsers that still include the CA. So if you are a CA, you actually hold a fair amount of power over browser vendors provided that you can deal with them in isolation. Finally, on the gripping hand I think that many browser people genuinely want to do what's right, which includes not screwing people in various ways, especially over risks that are (unfortunately) theoretical at the moment.

If Mozilla were to take a hard line here but no other browser were to do so, it feels likely that Symantec would issue those SHA-1 certificates anyways. If Mozilla were to make Firefox stop trusting Symantec certificates, a lot of people would switch away from Firefox (and it doesn't have a huge user base any more). If Mozilla didn't, its threats to do so in the future to misbehaving CAs would be much less credible. So it comes down to whether other browsers will pull Symantec's root certificates over this. Will they? I suspect not, although we'll find out soon enough.

(For the record: I don't think that the Mozilla people involved made the decision they did because of fear of this happening. I'm sure they're sincere in their desire to do the right thing, and I'm sure the harm to various people of Symantec not issuing these certificates weighed on their minds. But I can't see this situation and not think of the balance of power behind the scenes and would probably happen if Mozilla's decision had gone differently.)