On Fri, 2004-03-12 at 08:22, Edmund Horner wrote:
> Brian W. Fitzpatrick wrote:
> > httpd-2.0's repository contains 3,603 files, weighing in at about 41MB.
> > It also has 62 tags and branches.
> >
> > Let's say that we convert this repository to Subversion, preserving
> > history.
> >
> > If some dumb crawler comes along and decides to crawl
> > httpd-2.0/[trunk|tags|branches], it's going to suck somewhere in the
> > neighborhood of 2.5GB of bandwidth as it grabs every tagged and branch
> > revision on it's way to trunk.
> >
> > See the problem?
>
> What about recommending that robots.txt contains:
>
> User-agent: *
> Disallow: /repos/tags
> Disallow: /repos/branches
>
> Or a variation, depending on the repository layout.
Ah! Now that sounds like an *excellent* idea.
-Fitz
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Fri Mar 12 16:25:36 2004