[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: Subversion and Very,Very Large Repositories

From: Russ Brown <rbrown_at_ebuyer.com>
Date: 2006-02-28 15:38:42 CET

On Tue, 2006-02-28 at 09:03 -0500, Roger Ashby wrote:
> I administer a repository of images (about 300,000
> tifs,pdfs,pngs,jpgs, and some zips with no single file larger than
> 700MB, however only 1/3 of those are more then a MB in size and of
> those that are they mostly range between 1-40MB) that currently totals
> about 700GB data. I've been tasked with coming up with a revision
> control/digital asset management solution on a very limited budget. I
> ultimately settled upon using svk (because of the lightweight checkout
> copy management) with a subversion repository in the background. I
> decided to incrementally capacity test the subversion repository a few
> Gigs at a time. I have however run into an issue, it seems that the
> repository is being saved in one big file, and I've reach the max file
> limit for my machine.
>

What backend are you using? For fsfs each revision is stored in a
separate file. I can't comment on how DBD works though from this
perspective, but fsfs definitely does not store everything in one file.

> My first question is, are there options for svk or subversion that
> dictate how the repository/depot stores it's files so I don't run into
> this file size issue.
>

Make sure you're using fsfs. Unless one revision (most likely the first
one) is greater than your filesystem's max file size limit, you will be
fine.

> My second question is, does it make sense to have this much data (it
> could grow to several TB over the next 5 years) under revision control
> can subversion handle it or should I be using some other solutions.

I can only comment as far as my experience has taken me so far. My svk
depot (mirroring my work's repository) is 2.1GB in size and is now at
21324 revisions. No problems doing day to day stuff whatsoever. Now
several TB I do not know about... :)

-- 
Russ
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Received on Tue Feb 28 23:03:09 2006

This is an archived mail posted to the Subversion Users mailing list.