A while back I gave DuckDuckGo a try (for various reasons that I'm not going to try to summarize here). One of the things the experience brought home to me is how hard it is for anyone to compete with Google in search, at least general Internet search. The crawling is hard enough on its own, not just because of the scope but also because you have to tacitly persuade a critical mass of webmasters to not block your robot, either out of hand or when it misbehaves (and apparently everyone's web spider misbehaves sooner or later, even Google's). But my DuckDuckGo experience has led me to conclude that the real hard bit is getting your query results to be good.

Part of this is simple quality of implementation issues, which should not be surprising. For all that people harsh on Google results and Google's recent moves to personalize them more and more, Google has spent engineer-decades of effort on improving search results and tuning them. It would be a little bit surprising if that work was easily duplicated or improved on.

But I think that this is also partly because long-term Google users have quietly learned how to write search queries that Google likes. And in fact 'learned' is not quite the right word for the process, because I don't think it's been a conscious one. Instead we've just acclimatized ourselves to Google much as it has to us, absorbing (and creating) an idiosyncratic collection of tricks and tools for getting the results we want. It's highly unlikely that these reflexive tricks will work the same or be as effective on any other search engine. The result is that until people unlearn those reflexes and acclimatize to the new search engine, no search engine is going to work as well as the old, familiar Google.

(I'm assuming roughly equivalent basic results. If a new search engine gives better enough results than Google's, it can overcome this effect. How much better depends on how big the effect is, which varies from person to person.)

This was pretty much the feeling that I had when I used DuckDuckGo. I was reasonably confidant that DDG knew the information that I wanted; I just didn't know the tricks to make it appear and to bubble up to the top of search results. As a result I went back to Google; whether or not the basic quality of search results is a bit better on one or the other and regardless of other factors, in practice Google was easier for me to get the results I wanted.

(There's also an related issue with negative results, but that's another entry.)