Wandering Thoughts archives

Link: Search engine page size limits for indexing

Search Engine Indexing Limits: Where Do the Bots Stop? takes an experimental approach to seeing how big a page various search engine bots will fetch, and how much of large pages they index. I find this an interesting question because it affects how you organize your content and generate indexes to it, especially for dynamic websites with auto-generated aggregate pages.

One area not investigated in the article is how far down the pages the search engine bots will go looking for links to follow. I smell a followup project for someone.

(From Ned Batchelder, who has interesting information on the size of his own blog pages as a result of this.)