== Why _nofollow_ is useful and important No less a person than Google's Matt Cutts recently spoke up about [[herding Googlebot http://www.mattcutts.com/blog/bot-obedience-herding-googlebot/]] and more or less recommend using the _noindex_ meta tag on pages instead of _nofollow_ on links to them (on the grounds that it's more of a sure thing to mark pages noindex than to make sure that all links are marked nofollow). I must respectfully disagree with this, because in one important respect meta _noindex_ isn't good enough. The big thing that _nofollow_ does that meta _noindex_ can't do is that it makes [[good web spiders RespectTheNofollow]] not fetch the target page *at all*. Which means that you didn't have to send it, and for dynamic pages that you didn't have to generate it. (This is especially important for heavily dynamic websites that have [[a lot of automatically generated index pages of various sorts DynamicSitePerilScope]].) I really don't want to be burning my CPU cycles to generate pages that web spiders will just throw away again; frankly, it's annoying as well as wasteful. This is a good part of why I am so twitchy about [[spiders respecting _nofollow_ RespectTheNofollow]]. (In fact I care more about this than about helping Google reduce redundancy in their indexes, which is one reason why WanderingThoughts has lots of _nofollow_ but no meta _noindex_. Plus, getting good indexing for a blog-oid thing is much harder than just sprinkling some _noindex_ magic over bits.) === Sidebar: why not _robots.txt_? In theory, _robots.txt_ is supposed to be the way to tell web spiders to avoid URLs entirely. However, there are two problems with in practice. First, the format itself is inadequate for anything except blocking entire directory hierarchies. Second, it's the wrong place; the only thing that really knows whether a page should be spidered is the thing generating the page.