== Link: Search engine page size limits for indexing

[[Search Engine Indexing Limits: Where Do the Bots Stop?
http://www.sitepoint.com/article/indexing-limits-where-bots-stop]] takes
an experimental approach to seeing how big a page various search engine
bots will fetch, and how much of large pages they index.
I find this an interesting question because it affects how you organize
your content and generate indexes to it, especially for dynamic websites
with auto-generated aggregate pages.

One area not investigated in the article is how far down the pages the
search engine bots will go looking for links to follow. I smell a
followup project for someone.

(From [[Ned Batchelder
http://www.nedbatchelder.com/blog/200605.html#e20060506T132734]], who
has interesting information on the size of his own blog pages as a
result of this.)