Wandering Thoughts archives

One potential problem with cloud computing for us is the payment model

I spend a certain amount of my spare time trying to think about how we might use cloud computing, and also about reasons we might not be able to. Because of the special nature of universities, one of the potential problems for us is how it changes the payment model. As many people have observed, cloud computing replaces a large one time up front cost (to buy and deploy hardware on premises) with a steady ongoing cost (the monthly cloud bill).

In many environments, this is an attractive change all by itself; it lowers your initial costs, it lets you scale things down if you turn out not to need as much as you expected, it leads to smoother budgets, and so on. Unfortunately, universities are not anything like normal environments. In particular, in universities it is much easier to get 'one time only' money than it is to get an ongoing budget (cf), and even once you've got an ongoing budget the extra challenge is holding on to it for years to come.

The flipside of cloud computing having a steady ongoing cost is that once you buy into cloud computing, you are committed to that cost. Your (cloud-based) infrastructure requires you to keep paying for it every month. Fail to pay at all and you get turned off; be unable to pay for all of it and you have to reduce your infrastructure, shrinking or losing services. By contrast physical hardware bought with one time money is yours now until it falls apart beyond your ability to fix, no matter what happens to budgets in the future. And if does start to fall apart (and it's important), the odds are pretty good that you can scrounge up some more one time money to keep things going.

Perhaps I have just existed in an unusual university environment, but my experience is that ongoing budgets are far from secure no matter what you might have been promised. Sooner or later a big enough budget cut will come up and, well, there you are. This is of course not an issue that's unique to universities, but the lack of a ROI does make it harder to mount certain 'this is worth spending the money' arguments in defense of your ongoing budget.

(As was pointed out to me recently, it's also not enough to just hold on to a fixed-dollars ongoing budget. Your ongoing budget really needs to be adjusted to account for 'inflation', in this case any increases in cloud computing prices or changes in your provider's charging models that mean you pay more.)

On the other side, having a monthly cloud computing bill might make it easier to defend its ongoing budget item precisely because any cuts directly require immediate reductions in services. A budget reduction wouldn't be an abstract thing or living with older hardware for longer, it'd be 'we will have to stop doing X and Y'.

A cynical view on needing SSDs in all your machines in the future

Let's start with my tweets:

@thatcks: Dear Firefox Nightly: doing ten+ minutes of high disk IO on startup before you even start showing me my restored session is absurd.
@thatcks: Clearly the day is coming when using a SSD is going be not merely useful but essential to get modern programs to perform decently.

I didn't say this just because programs are going to want to do more and more disk IO over time. Instead, I said it because of a traditional developer behavior, namely that developers mostly assess how fast their work is based on how it runs on their machines and developer machines are generally very beefy ones. At this point it's extremely likely that most developer machines have decently fast SSDs (and for good reason), which means that it's actually going to be hard for developers to notice they've written code that basically assumes a SSD and only runs acceptably on it (either in general or when some moderate corner case triggers).

SSDs exacerbate this problem by being not just fast in general but especially hugely faster at random IO than traditional hard drives. If you accidentally write something that is random IO heavy (or becomes so under some circumstances, perhaps as you scale the size of the database up) but only run it on a SSD based system, you might not really notice. Run that same thing on a HD based one (with a large database) and it will grind to a halt for ten minutes.

(Today I don't think we have profiling tools for disk IO the way we do for CPU usage by code, so even if a developer wanted to check for this their only option is to find a machine with a HD and try things out. Perhaps part of the solution will be an 'act like a HD' emulation layer for software testing that does things like slowing down random IO. Of course it's much more likely that people will just say 'buy SSDs and stop bugging us', especially in a few years.)

Why I still report bugs

I've written plenty of entries here over the years about why people don't report bugs (myself included). Yet I still report bugs against some projects.

The quiet reality of bug reports that I haven't mentioned so far is that when one of my bug report goes well, it's an amazingly good feeling. When I find a bug, isolate it, maybe peer into the code to identify the cause, file a report, and have the project authors say 'that's a nice find and a good analysis', that's a real rush. It's not even so much that I may get a fix for my issue; it's very much also that I have reached into the depths of a mystery and come out with the right answer. It's even better when it helps other people (both in the future and sometimes right away). This is bug reports as the culmination of debugging, and successful debugging itself is a rush of puzzle solving and a victory over a hard problem.

(It's equally a good feeling, although a somewhat different one, when I file a carefully reasoned bug report in favour of something that the software doesn't currently do and I wind up bringing the project people around to my views.)

More than anything else, this is the feeling that keeps me filing bug reports with hospitable projects. It is the feeling that makes bug reports into something other than grinding work and that makes me proud to have written a good report.

I'm often down on bug reporting because I don't have this experience with bug reporting very often. But neither I nor anyone else should forget that bug reporting can feel good too. It's probably not a small part of why we're all willing to keep making those reports, and I don't want to lose sight of it.

(It's easier to remember the negative bug reporting experiences than the powerfully positive ones, partly because humans have very good memories for the negative.)

(As you might guess, this entry was sparked by the experience of recently filing a couple of good bug reports.)

A modest little change I'd like to see in bug reporting systems

It is my opinion that sometimes little elements of wording and culture matter. One of those little elements of culture that has been nagging at me lately is the specifics of how Bugzilla and probably other bug reporting systems deal with duplicate bug reports; they are set to 'closed as a duplicate of <other bug>'.

On the one hand, this is perfectly accurate. On the other hand, almost all of the time one of my bug reports is closed out this way I wind up feeling like I shouldn't have filed it at all, because I should have been sufficiently industrious to find the original bug report. I suspect that I am not alone in feeling this way in this situation. I further suspect that feeling this way serves as a quiet little disincentive to file bug reports; after all, it might be yet another duplicate.

Now, some projects certainly seem to not want bug reports in the first place. And probably some projects get enough duplicate bug reports that they want to apply pressure against them, especially against people who do it frequently (although I suspect that this isn't entirely going to work). But I suspect that this is not a globally desirable thing.

As a result, what I'd like to see bug reporting systems try out is simply renaming this status to the more neutral 'merged with <other bug>'.

Would it make any real difference? I honestly don't know; little cultural hacks are hard to predict. But I don't think it would hurt and who knows, something interesting could happen.

(In my view, 'closed as duplicate' is the kind of thing that makes perfect sense when your bug reporting system is an internal one fed by QA people who are paid to do this sort of stuff efficiently and accurately. In that situation, duplicate bugs often are someone kind of falling down on the job. But this is not the state of affairs with public bug reporting systems, where you are lucky if people even bother to jump through your hoops to file at all.)

My brush with the increasing pervasiveness of smartphone GPS mapping

One of the things I do with my time is go bicycling with a local bike club. When you go on group bike rides, one of the things you generally want to have is directions for where the ride is going (if only to reassure yourself if you get separated from the group). When I started with the club back in 2006, these 'cue sheets' for rides were entirely a paper thing and entirely offline; you turned up at the start of the ride and the ride leader handed out a bunch of copies to anyone who wanted or needed one.

(By 2006 I believe that people were mostly creating new cue sheets in word processors and other tools, but some old ones existed only in scanned form that had been passed down through the years.)

Time rolled on and smartphones with GPS appeared. Various early adapters around the club started using smartphone apps to record their rides. People put these ride recordings online and other people started learning from them, spotting interesting new ways to get places and so on. Other people started taking these GPS traces and loading them on their own smartphones (and sometimes GPS devices) as informal guides to the route to supplement the official cue sheets. As time went on, some people started augmenting the normal online ride descriptions for upcoming rides with somewhat informal links to online GPS-based maps of the ride route.

Last year the club started a big push to put copies of the cue sheets online, and alongside the cue sheets it started digitizing many of the routes into GPS route files. For some of the rides, the GPS route files started being the primary authority for the ride's route; the printed cue sheet that the ride leader handed out at the start was generated from them. Finally, this year the club is really pushing people to print their own cue sheets instead of having the ride leader give them out at the start. It's not really hard to see why; even last year fewer and fewer people were asking for copies of the cue sheet at the start of rides and more and more people were saying 'I'm good, I've got the GPS information loaded into my smartphone'.

(This year, on the group rides I've lead I could hardly give out more than a handful of cue sheets. And usually not because people had already printed their own.)

It doesn't take much extrapolation to see where this is going. The club is still officially using cue sheets for now, but it's definitely alongside the GPS route files and more and more cue sheets are automatically generated from the GPS route files. It wouldn't surprise me if by five years from now, having a smartphone with good GPS and a route following app was basically necessary to go on our rides. There's various advantages to going to only GPS route files, and smartphones are clearly becoming increasingly pervasive. Just like the club assumes that you have a bike and a helmet and a few other things, we'll assume you have a reasonably capable smartphone too.

(By then it's unlikely to cost more than, say, your helmet.)

In one way there's nothing particularly surprising about this shift; smartphones with GPS have been taking over from manual maps in many areas. But this is a shift that I've seen happen in front of me and that makes it personally novel. Future shock is made real by being a personal experience.

(It also affects me in that I don't currently have a smartphone, so I'm looking at a future where I probably need to get one in order to really keep up with the club.)

OpenBSD and the impact of culture

We use OpenBSD here for firewalls and a few other things, and over the course of doing so we've found a number of bugs and issues. Have we ever reported these to the OpenBSD people?

Ha ha. Of course not. We'd have to be crazy to do that.

The OpenBSD culture has acquired a reputation for being what we could politely call 'abrasive'. But let's call a spade a spade and say 'abusive' instead. As the folklore has it, if you show up with a question or an issue or a bug report that the OpenBSD people feel is stupid or ignorant or wrong, you can expect them to call you an idiot and a waste of their time. Innocently say something that they don't like? You can look forwarded to being upbraided, perhaps by the infamous Theo himself. Maybe you'll even get an explosion of invective.

(When I wrote about how a social obligation to report bugs implies that projects have a social obligation to make reporting bugs not unpleasant, OpenBSD was the big example I had in mind.)

If we cared a lot, if we desperately needed OpenBSD to work right in some case where it doesn't, maybe interacting with this would be worth it. But we don't, it isn't, and so we don't. We'll almost certainly never submit an OpenBSD bug report no matter what issues we run into. I suspect we're not alone in this.

(Certainly in the one case we ran into where we couldn't get OpenBSD to work at all for us, we just abandoned it rather than particularly file bug reports or interact with the OpenBSD community. This wasn't even a decision I was involved in, as it was a project my co-workers were doing.)

If OpenBSD doesn't want to get bug reports except from people who are really dedicated, the folklore about their culture is working really great for them. If they would maybe like some more bug reports, assistance, and so on, it is not being so helpful.

I also rather think that this culture affects how interested people are in using OpenBSD. Especially with free software, the community around the software can really change how usable it is because the community is where you go for information, help, advice, bugfixes, and so on. If there is effectively no community for you, the software becomes an unsupported black box, and those are much less useful and attractive.

Culture matters. Culture has an impact, one that goes far beyond the obvious. And contrary to what people sometimes say, you cannot divorce culture from technical issues or consider code and personalities separately, because how you talk about technical issues influences whether people are willing to talk to you about them at all. Some people will care enough to persist anyways; other people won't.

(And OpenBSD is not particularly an exception here, although it does make a great example that I have personal experience with. Any number of projects have their own cultural issues; I'm sure you can name your own favorite cases.)

Faster SSDs matter to companies because they sell things

The computer hardware industry has a problem: systems mostly aren't getting (much) better any more, especially desktop PCs. The most famous example is that CPU performance has been changing only incrementally for years, especially for single threaded performance. This is a problem because a lot of hardware sales are upgrades and when there's no particular performance improvement you can trumpet people don't bother to upgrade. They do replace old machines eventually, but that's slower and less lucrative (and runs the risk of people leaking out to, eg, tablets and smartphones).

This is where a faster SSD interconnect matters to companies; it's a clear performance improvement they can point to. Whether or not it makes a difference in practice for most people, companies can trumpet 'much faster disk read and write speeds' as well as 'take full advantage of SSDs' and thereby move (more) hardware. No matter what it does in practice, it sounds good.

My general impression is that Intel and the motherboard companies are pretty desperate for things that will move new hardware, and really I can't blame them. So I wouldn't be surprised to see U.2 NVMe support appear in motherboards and systems quite fast, and I honestly hope it works to prop up their fortunes.

(As someone who is well out on the small tail end in terms of my PC hardware, I have a vested interest in a vibrant motherboard market that caters to even relatively weird interests like mine.)

Sidebar: the 'pushing technology' view

On a longer and larger scale view, drastically increased 'disk' access speeds that are essentially specific to SSDs also increase the chances that people will start building filesystems and other things that are specifically designed and tuned for SSDs, or just generally for things that look more like memory than rotating magnetic medium. It's been very useful to be able to pretend SSDs are hard drives, but they aren't really and we may find that systems are quite different and better when we can stop pretending.

(This too is likely to sell new hardware over the long term.)

The next (or coming) way to connect SSDs to your system

Modern SSDs have a problem: flash chips are so fast that they outpace even high speed SATA and SAS links. In the enterprise market the workaround for this is SSD 'drives' that are PCIe cards, but this has all sorts of drawbacks as a general solution. Since the companies involved here are not stupid, they've known this for some time and have come up with a new interconnection system, NVMe aka NVM Express.

The Wikipedia page is a bit confusing to outsiders, but as far as I can tell NVMe is essentially a standard for how PCIe SSDs should present themselves to the host system. NVMe devices advertise that they have a specific PCI device class and promise to have a common set of registers, control operations, and so on; as a result, any NVMe device can be driven by a single common driver instead of each company's devices needing their own driver.

(Most PCI and PCIe devices need specific drivers because there's no standard for how they're controlled; each different device has its own unique collection of registers, operations, and so on. This gives us, eg, a zillion different PCI(e) Ethernet device drivers.)

If this was all that NVMe was, it would be kind of boring because it would be restricted to actual PCIe card SSDs and those are never going to be really popular. But NVMe also has a physical standard called U.2 that lets you pull PCIe out over a cable to a conventional-ish SSD drive. This means that you can have a 2.5" form factor SSD mounted somewhere and cabled up that is an NVMe drive and thus is actually a PCIe device on one of your PCIe busses. Assuming everything works and U.2 ports appear in sufficient quantity on motherboards, this seems likely to compete with SATA for connecting SSDs in general, not just in expensive enterprise setups.

(U.2 used to be called SFF-8639 until this month. As you can tell, the ink is barely dry on much of this stuff.)

If I'm reading the tea leaves right, U.2 is somewhat less convenient than ordinary SATA because it requires cables and connectors that are a bit more than twice as big. This is going to impact port density and wiring density, but there are plenty of ordinary machines which have enough motherboard real estate and enough space for cables that this probably isn't a big concern. On the other hand I do expect a bunch of small motherboard and high density servers to deliberately stay with SATA or SAS for the higher achievable port density.

(PCIe and thus NVMe can also be connected up with a less popular connector standard called M.2. This is apparently intended for plugging bare-board SSDs directly into your motherboard instead of cabling things to mounts elsewhere, although I've read some things suggestion it can be coerced into working with cables.)

Does this all matter to ordinary people balancing the SSD inflection point? Maybe. My view is that it does matter in the long term for computer hardware companies, but that's going to take another entry to explain.

Thinking about people's SSD inflection points in general

What I'm calling the (or a) SSD inflection point is the point where SSDs get big enough and cheap enough for people to switch from spinning rust to SSDs. Of course this is already happening for some people some of the time, so the real question is when it's going to happen for lots of people.

(Well, my real question is when it's going to happen for me, but that's another entry.)

I don't have any answers. I don't even have any particular guesses or opinions. What I do have is an obvious observation.

For most people and most systems, the choice of HDs versus SSDs is not about absolute performance (which clearly goes to SSDs today) or absolute space (which is still by far in the hands of HDs both in terms of price per GB and how much TB you can get in N drives). Instead, unsurprisingly, it is about getting enough space and then if possible making it go faster. People can and do make tradeoffs there based on their feelings about the relative importance of more space and more speed, including ones that make their systems more complicated (like having a small, affordable SSD for speed while offloading much of your data to a slow(er) but big HD). This makes the inflection point complicated and thus the migration from HDs to SSDs is probably going to be a drawn out affair.

We've already seen one broad inflection point happen here, in good laptops; big enough SSDs have mostly displaced HDs, even though people may not have all the space they want. I doubt many laptop users would trade back, even if they have to carefully manage disk space on their laptop SSD.

My suspicion is that the next inflection point will hit when affordable SSDs become big enough to hold all of the data a typical person puts on their computer; at that point you can give people much faster computers without them really noticing any drawbacks. But I don't have any idea how much space that is today; a TB? A couple of TB? Less than a TB for many people?

(My impression is that for many people the major space consumer on home machines is computer games. I'm probably out of touch on this.)

The impact on you of making 'bad' bug reports

One of the reasons that making bug reports is stressful for people who can do them well is that we're aware of the quiet but very real social downside of submitting 'bad' bug reports in most contexts. One of the important practical benefits of making (good) bug reports is that it builds up your good karma with the people behind whatever you're reporting bugs on. This good karma makes it more likely that they will take you seriously in the future, interact nicely with you, and perhaps even fix your bugs or answer your questions. This is ultimately a matter of name recognition; the developers will see a report or a question from you and go 'oh yeah, I remember that name, that's going to be worth reading'.

Bad bug reports are the reverse. They flush good karma down the drain and build up bad associations in people's minds. If you submit bad bug reports, people are going to remember you but it won't be in a good way. You'll be remembered as someone who's not worth paying attention to, someone that they kind of wish wouldn't file bugs and argue with them about the bugs you've filed and so on, and all of this will make the developers less likely to do you any favours or work on your problems or whatever.

Unfortunately, people generally remember bad interactions much better than they do good ones. You can guess what the corollary here is, but I'll say it outright:

It's often better for you to not file a bug report at all than to file a bad bug report.

Unless you can make significant contributions (submitting good patches and so on), you accumulate good karma (aka 'name recognition') only slowly. One bad bug report can thus offset a whole lot of good ones.

These social consequences of 'bad' bug reports serve as an obvious disincentive to file bug reports at all. Unless you're very sure that you're filing a good report (but not too good), it's better to sit on your hands. You're probably not going to get a timely fix for your problem anyways and sitting on your hands definitely avoids the potential downsides.

(By the way, one of the signals that an abrasive bug reporting environment sends to outside people is that there are probably to be significant bad social consequences for making 'bad' bug reports. Unless you've found something that is 110% sure to be a real bug that the developers will care about and that you can report clearly and in an unarguable way, well, it's really better to stay silent.)

Now it's time to add more unhappy news. I put 'bad' in quotes deliberately, because in practice bad bug reports includes any bug report that the developers don't agree with. If you file bug reports that are closed as 'NOTABUG' or the equivalent, those build up bad karma; you're going to get remembered as someone who files bug reports for things that aren't actually bugs. This often extends to bug reports about things that the developers don't really care about, because the developers will remember you as someone who's pestering them about unimportant stuff. Of course all of this adds its own level of stress and tension to filing certain bug reports, especially if you're critiquing a design or API issue.

(I sort of wrote a version of this as a comment reply here a few years ago, but I've decided that it's important enough to be in an entry of its own.)