Wandering Thoughts archives

IPv6 is going to be a fruitful source of configuration mistakes

I recently ran into an odd problem. If I visited a website with my regular Firefox session, it came up fine and normally, but if I tried to visit it in my testing Firefox I instead got a generic advertising page for a VPS provider (the VPS provider that the website was hosted on, in fact). Testing now shows that I would have gotten the same results with Chrome or lynx, but that if I had tried this from work instead of from home, everything would have seen the real website.

Of course, you already know the punchline here because I put it in the title of this entry. The website has both an IPv4 and an IPv6 address; the IPv4 address worked while the IPv6 address got the generic VPS ad page. I presume that this is inadvertent and happened because something, somewhere, was not told that this IPv6 address should be recognized as being associated with the website. (There are plausible Apache VirtualHost mistakes that would do this, for example.)

What is interesting in a somewhat depressing way is how odd and unclear the symptoms were. I'm technically inclined and my own sysadmin, so it only took a little bit of head scratching before the penny dropped (and I verified my suspicions). Now imagine this happening (over and over again) to some of your users. Won't that be a fun problem report?

This is not a new problem, but this is the first time I've seen it caused by what was clearly a configuration slipup instead of things like routing problems. I suspect that this sort of thing is going to happen over and over again as we see more and more IPv6 usage. In a sense, configuring things for IPv6 has been (and to a large part still is) something that wasn't necessary for the system to work .

Sidebar: Why my symptoms were what they were

My home machine has a real IPv6 connection, but none of the machines at work do (my office workstation has a 6to4 IPv6 address, which evidently isn't good enough to count as an IPv6 address for this VPS provider). My testing Firefox, Chrome, and lynx all are IPv6 enabled and prefer IPv6 addresses over IPv4 ones, but my regular Firefox session is not (because it is proxied through an IPv4 only ad-stripping proxy).

Why system administration certifications have worked so far

A commentator on my previous entry on sysadmin education asked a very good question:

Now square your view with the fact that MSCE, CCNE, RHCE exist.

In thinking about it, I think that there are several reasons why people get such certificates.

• Historically, they've worked; getting a certificate has generally led to getting paid better.

• Some certificates can be obtained more or less as a side effect of your employer getting you trained on some specific technology that the employer cares about. If your employer needs someone to have a good knowledge of Red Hat (eg, for a RH-based deployment) and sending you on a training course is an efficient way to get you up to speed, any RHCE certificate which you can earn in the process is just a bonus.

• Many certificates can be obtained sufficiently cheaply to be within reach of a job seeker who wants an extra edge for their resume (either to make it stand out or to make up for some other weakness they have).

Of these, I think the overall cost of certificates is a big driver. If it only costs a few hundred dollars to get a certificate, it doesn't take much of a raise or a better job offer (or a job offer that comes sooner) for you to make money on the whole deal.

(I also think that significantly more expensive training and certification processes would drastically blunt many of these effects, especially if the training was for general sysadmin knowledge instead of for a specific technology.)

Sidebar: certificates and hot new things

If you have been around technology for long enough, you will have seen certificates sprouting up like weeds in the wake of a popular hot new thing. As it happens, I have a theory about that.

To start with, certificates let employers outsource checking a job seeker's technical chops in a specific area to a third party, the certificate issuer. Rather than quiz the job seeker about their knowledge of technology X yourself, you just look for a certificate in the area (and thus you trust that the people behind it asked enough questions and so on). This outsourcing is especially valuable if the employer, or at least the people involved in the hiring process, don't know enough about technology X to do a decent job of checking out job seekers. If you can't tell a fast talker from the real thing you have no real choice but to rely on people who can, at least in theory. This lack of local expertise is especially likely to happen during the spread of a hot new thing, specifically at the point where it's got enough buzz that everyone wants in but it's still new enough that there aren't many people with actual experience with it.

This also suggests a reason why it's common to sneer at employers who insist on certificates. Who wants to work for people who lack the in-house technical expertise to evaluate your qualifications themselves?

Why formal sysadmin education isn't likely any time soon

A while back I ran across Tom Limoncelli's The road to intentional, formal, system administration education. In it Tom put forward the view that system administration should and in fact needs to move to a more formalized system of education and training. Unfortunately I have to take the opposite side of this; I don't believe that such a move is likely any time soon and I don't believe that trying to make such a move would work out.

The fundamental reason I feel this is an economic one. In almost any scenario, sysadmins getting more training than they currently do are going to demand more money in exchange. This is especially so if entry to system administration becomes more restrictive (eg, if in practice it now requires the new training), which would lower the available talent pool and thus drive up prices. If getting the new additional training has no reward (in terms of better pay) it's not clear why most people would bother spending the money on it.

(Some people will always be interested in immersing themselves more deeply in their field, and there will always be some jobs for more advanced experts. However you need more than a few jobs in order to pull any significant number of people into the additional training.)

However now we get to look at the extremely cynical take on DevOps. One corollary of this view of DevOps is that additional money to pay for better trained sysadmins is almost certainly simply not there. Employers are already not paying for sysadmins as it is; they're going to be even less willing to pay more to get basically the same thing. And if employers are not willing to reliably pay extra for better educated sysadmins, sysadmins are not going to get better educated.

(My cynical view is that turning a sysadmin into a 'real' DevOps person as many places mean it implies turning them into a programmer and a developer. I think that's a fine thing to do, but it's very far from more training and education focused on system administration. If you think that this is the future, you might as well focus on a couple of 'development in the real world' courses for programmers.)

The optimistic view is that I am underestimating the improvements that organizations will see by employing properly trained sysadmins. In the face of the extremely cynical take on DevOps I can't bring myself to believe that there is any really big improvement to be had here, because today organizations that want drastically improved results don't seem to be hiring good sysadmins to get them; they do entirely different things.

The extremely pessimistic view is that would-be sysadmins will have no choice but to get this training in the future because the market for sysadmins will shrink drastically and so 'proper training' will become a minimum requirement to get hired at all. Again I find myself unable to believe in this and think that if would-be sysadmins have to get additional training at all, they are just going to go learn how to be developers; it pays better (as far as I can tell).

(There are some people who could have been developers but who chose to be sysadmins instead, myself included, but I don't think it's all that common these days.)

The extremely cynical take on DevOps

You can take a number of views of what DevOps is about and what it means; for example, I've written about DevOps as a way to deal with the blame problem. That was a relatively cheery view of DevOps, so to go with it here is a really cynical one.

The grim view of what DevOps means is that it shows that people are not getting their money's worth from traditional sysadmins, or at least they feel that they aren't; they feel that traditional sysadmins cost too much and deliver too few benefits. For these people it's more cost effective overall to turn developers into partial, part time sysadmins than to hire actual sysadmins. Given that developers are relatively costly and in particular generally more costly than a sysadmin, this is a stunning failure on the part of actual system administration. In effect this view says that typical sysadmins are either screamingly inefficient at doing their work or they're adding negative value to the overall organization.

At this point it's customary for traditional sysadmins to rise in righteous anger to point out all of the things that these new-fangled 'devops' organizations are either doing badly or not doing, and talk about all of the things that can go wrong if you don't have real professionals. Unfortunately the proof is in the pudding, namely that real companies are doing this and they have not exploded. Yes, every so often such companies make a mistake that causes experienced sysadmins to roll their eyes and mutter 'how could they not know that', but it's almost always only embarrassing in the end and ultimately doesn't matter; as I wrote before in passing, it'll generally take a real long tail event to kill an otherwise sound company.

(In another world it might be possible to argue that companies that take this view are either terribly mistaken and thus doomed, or have simply not run into challenges that require real sysadmin expertise yet. In this world, some of the companies taking the DevOps view are quite successful and have run into real problems. So clearly taking a DevOps view doesn't doom your company and still leaves you able to solve hard system administration problems.)

Why Exim has a single queue for all email

In a recent entry I mentioned that Exim puts all pending email into a single queue instead of having multiple queues (with one per destination domain or the like). Given that multiple queues make it much easier to insure that a single slow destination doesn't affect email to other places, you might wonder why Exim has made such a choice. The answer is that a single queue is effectively forced on Exim because of a core design decision.

All mailers have two conceptual stages for processing a message: they take the initial top level addresses and map them into destinations, then deliver the email to the various destinations. Most mailers do this mapping process once and then save all of the destination addresses, but Exim instead repeats the mapping process every time it retries a message. And this is what makes the difference.

When you do this mapping process only once, it's easy to have per-destination queues; you determine the destinations at the start, add the message to the appropriate queues, and you're done; you can then go through each queue separately. When the mailer is redoing the mappings every time (and the results can change), what you generally wind up with is not queues but limits on how many simultaneous deliveries you can have to any single destination. You can implement this, but it requires much more coordination and complexity than simply starting N delivery processes for a particular queue.

(You wind up with a situation where each delivery process has to check in with something every time it wants to deliver to a new destination.)

For a mailer that redoes mappings every time, the straightforward approach is what Exim has: a single queue of messages and completely independent delivery processes (with per-message locking). The downside of this is that you can wind up with a bunch of your delivery processes all trying to deliver to the same unresponsive destination.