Re: Robots and Spiders, oh my!

Brian W. Fitzpatrick wrote:
> httpd-2.0's repository contains 3,603 files, weighing in at about 41MB.
> It also has 62 tags and branches.
>
> Let's say that we convert this repository to Subversion, preserving
> history.
>
> If some dumb crawler comes along and decides to crawl
> httpd-2.0/[trunk|tags|branches], it's going to suck somewhere in the
> neighborhood of 2.5GB of bandwidth as it grabs every tagged and branch
> revision on it's way to trunk.
>
> See the problem?

What about recommending that robots.txt contains:

User-agent: *
Disallow: /repos/tags
Disallow: /repos/branches

Or a variation, depending on the repository layout.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Fri Mar 12 15:23:25 2004