Re: Problem Loading Huge Repository

From: Daniel Shahaf <d.s_at_daniel.shahaf.name>
Date: Fri, 17 Jun 2011 18:59:48 +0300

Thomas Harold wrote on Fri, Jun 17, 2011 at 11:50:45 -0400:
> On 6/17/2011 10:54 AM, Daniel Shahaf wrote:
> >Thomas Harold wrote on Fri, Jun 17, 2011 at 10:31:43 -0400:
> >>And if you have a choice of file systems for the repository to be stored
> >>on, make sure that it's something which can deal with a few hundred
> >>thousand tiny files. On Linux, I'd suggest going with ext4 over ext3.
> >>While db/revs in a FSFS repository can have its revisions packed to
> >>reduce the file count, the db/revprops folder still consists of 1 tiny
> >>file for every revision in the project in a FSFS repository.
> >
> >revprops/ is sharded.
> >
> >And in 1.7 (including the recent 1.7.0-alpha1) it is packed, too.
> >
>
> Good. Another of the many reasons that we're looking forward to 1.7.
>
> Even with the sharding, those little revprop files are causing us
> issues during backups (hotcopy -> rdiff-backup). Being able to pack
> those revprop files is going to make a big difference as the backup
> process will only have to track 2000-2200 files instead of 30,000 to
> 50,000.
>
> (We have a few long-lived repositories with up to 25k revisions.
> And I just finished splitting a 22GB repository with 15-16k revs
> into a bunch of smaller repositories. Now the nightly backup can
> look at doing a hotcopy on only the repositories with changes in the
> last 5 days.)