Robots and Spiders, oh my!

Garrett just pointed out to me that the ASF's version control guidelines
(http://www.apache.org/dev/version-control.html) recommended checking
out a project_root instead of the trunk of that project (I fixed them
already).

While I was updating the guidelines and contemplating the difficulties
that one might have trying to checkout http://svn.collab.net/svn,
something occurred to me:

    What would happen if a robot crawled a big repository with a whole
    lotta tags and branches?

Shouldn't we have a big bold warning box in the book telling people to
create a robots.txt file in their DOCUMENT_ROOT That contains:

    User-agent: *
    Disallow: /

We've had this on svn.collab.net for ages, and I'm thinking we should
really let people know about it.

And while I'm at it, how about advising people who use Subversion
working copies as websites to put something like this in their
httpd.conf files:

    # Disallow browsing of Subversion working copy administrative
    # directories.
    <DirectoryMatch "^/.*/\.svn/">
        Order deny,allow
        Deny from all
    </DirectoryMatch>

Thoughts?

-Fitz

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Fri Mar 12 05:01:10 2004