Link: Search engine page size limits for indexing

May 7, 2006

Search Engine Indexing Limits: Where Do the Bots Stop? takes an experimental approach to seeing how big a page various search engine bots will fetch, and how much of large pages they index. I find this an interesting question because it affects how you organize your content and generate indexes to it, especially for dynamic websites with auto-generated aggregate pages.

One area not investigated in the article is how far down the pages the search engine bots will go looking for links to follow. I smell a followup project for someone.

(From Ned Batchelder, who has interesting information on the size of his own blog pages as a result of this.)

Written on 07 May 2006.
« SCGI versus FastCGI
A really stupid web spider »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun May 7 23:18:26 2006
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.