Chris's Wiki :: blog/web/WgetNoMoreHere Commentshttps://utcc.utoronto.ca/~cks/space/blog/web/WgetNoMoreHere?atomcommentsDWiki2018-12-17T16:12:30ZRecent comments in Chris's Wiki :: blog/web/WgetNoMoreHere.By Chris Siebenmann on /blog/web/WgetNoMoreHeretag:CSpace:blog/web/WgetNoMoreHere:0f900c99ca9a9e4d624bfb030563bb2cd87ecd50Chris Siebenmann<div class="wikitext"><p>Belatedly: a crawl delay wouldn't fix things, because fast fetching is
only one of wget's problems. I also believe that for most people, any
substantial crawl delay is equivalent to disabling wget; people mostly
don't want to be sitting there for days waiting for wget to finish.</p>
<p>As far as suggesting wget options, that would require research work and
that's up to the people who want to use wget on my site. I don't, so
they get to do the work. If I actively wanted people to make copies of
here I would have suggested ways of doing so, but as it is I am merely
passively okay with it.</p>
</div>2018-12-17T16:12:30ZBy Grant Taylor on /blog/web/WgetNoMoreHeretag:CSpace:blog/web/WgetNoMoreHere:4878f17eb866083b61286c61b7f1dbad33667e92Grant Taylorhttps://dotfiles.tnetconsulting.net/home.html<div class="wikitext"><p>I agree with your speed bump.</p>
<p>I have long advocated to friends and colleagues that they should close, and lock, every gate, even if it's a lowly 3 foot tall picket fence.</p>
<p>It's enough to provide a clear demarcation that someone must make a small effort to step over. A distinction that is quite important in a number of circumstances.</p>
</div>2018-12-08T17:56:38ZBy Anonymous on /blog/web/WgetNoMoreHeretag:CSpace:blog/web/WgetNoMoreHere:c3b6384e903f54b8a49429d2ad11bf7bea12b3feAnonymous<div class="wikitext"><p>If someone did want to use wget to mirror your site, this would be a good place for you to suggest the options you would like them to use.</p>
</div>2018-12-05T13:57:53ZFrom 8.25.197.27 on /blog/web/WgetNoMoreHeretag:CSpace:blog/web/WgetNoMoreHere:150df1aa6e0562400d114a183fa510911eb91d82From 8.25.197.27<div class="wikitext"><p>What about something like this:</p>
<p>User-agent: wget
Crawl-Delay: 10</p>
<p>in your robots.txt file.</p>
</div>2018-12-04T20:43:52Z