From a comment on an earlier entry:

What's surprising is not the failure of XHTML. Rather it's the enthusiasm for cargo-cult XHTML that persists to this day: HTML files with the XHTML 1.0 DTD decl and superficial XHTML features like <br /> tags, but served without the XHTML mime type that would actually make them XHTML.

I'm not surprised in the least. In fact, this is a great illustration of the web authoring folkore that I talked about in that entry.

The thing to understand about the web is that people do not author content to standards. I don't mean that in the sense that they reject standards; I mean that in the sense that almost no web authors know what the standards actually require, because they didn't learn how to write HTML by reading the standard or authoritative descriptions of it. Instead people learn through what could be called a folk process, where information passed from person to person in various ways.

(There are many reasons why this happened, but I suspect that one is that there have been so many web standards that people have needed some sort of trusted guide to sort out which one to pay attention to. Clearly the standards organizations themselves cannot play this role.)

In this environment, very few people really know about XHTML and how to use it. Instead most people know a cloud of folklore surrounding XHTML; they know that it is the right thing to author content in and they know some of the markers of 'writing XHTML', like using <br />, so they use these on their pages. These people are not cynical or idiots; instead they are well intentioned but undereducated. In some sense they are the inevitable result of repeated proselytization that writing to web standards is important.

In fact they may not even know that <br /> is XHTML, they may just have vaguely heard or vaguely remember that it is the right way to write singleton tags these days. You might laugh, but this used to describe me.

The uncomfortable truth is that writing to standards requires an expert and not all that many people are interested in becoming experts on HTML. Especially when the best practices for HTML keep changing.

(Someone is about to say 'just author with standard conforming software'. Great. How do you know that the software is really standards conforming? How do you know enough to even ask that question? This is how we get people confidently using software <X> and putting 'valid XHTML' badges on their web pages when their pages are neither XHTML nor valid. I'm sure you can fill in some values for <X> here.)