Acclimatization makes competition in web search engines hard(er)

October 8, 2012

A while back I gave DuckDuckGo a try (for various reasons that I'm not going to try to summarize here). One of the things the experience brought home to me is how hard it is for anyone to compete with Google in search, at least general Internet search. The crawling is hard enough on its own, not just because of the scope but also because you have to tacitly persuade a critical mass of webmasters to not block your robot, either out of hand or when it misbehaves (and apparently everyone's web spider misbehaves sooner or later, even Google's). But my DuckDuckGo experience has led me to conclude that the real hard bit is getting your query results to be good.

Part of this is simple quality of implementation issues, which should not be surprising. For all that people harsh on Google results and Google's recent moves to personalize them more and more, Google has spent engineer-decades of effort on improving search results and tuning them. It would be a little bit surprising if that work was easily duplicated or improved on.

But I think that this is also partly because long-term Google users have quietly learned how to write search queries that Google likes. And in fact 'learned' is not quite the right word for the process, because I don't think it's been a conscious one. Instead we've just acclimatized ourselves to Google much as it has to us, absorbing (and creating) an idiosyncratic collection of tricks and tools for getting the results we want. It's highly unlikely that these reflexive tricks will work the same or be as effective on any other search engine. The result is that until people unlearn those reflexes and acclimatize to the new search engine, no search engine is going to work as well as the old, familiar Google.

(I'm assuming roughly equivalent basic results. If a new search engine gives better enough results than Google's, it can overcome this effect. How much better depends on how big the effect is, which varies from person to person.)

This was pretty much the feeling that I had when I used DuckDuckGo. I was reasonably confidant that DDG knew the information that I wanted; I just didn't know the tricks to make it appear and to bubble up to the top of search results. As a result I went back to Google; whether or not the basic quality of search results is a bit better on one or the other and regardless of other factors, in practice Google was easier for me to get the results I wanted.

(There's also an related issue with negative results, but that's another entry.)

Comments on this page:

From at 2012-10-08 04:52:13:

The thing that has kept me using duckduckgo is that I can customise the regions. So I have separate search keywords for different languages. Google and Bing, by contrast, always try to push me to stuff in the local language.

I think I've become more effective for the fact that I use a mix of search engines because I've come to know their strengths. With DDG, it is easy to add !bing or !google into the search bar if I think it is worth getting a second opinion on a search. I also sometimes use for one word searches it will often get you the best site. Wolfram Alpha also has it's place.

Written on 08 October 2012.
« Python can execute zip files
The negative results problem with search engines »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Oct 8 00:35:34 2012
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.