[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: Repository Size Growing VERY Quickly

From: Daniel Berlin <dberlin_at_dberlin.org>
Date: 2006-07-21 03:51:00 CEST

J Kramer wrote:
>
>
> On 7/20/06, *Daniel Berlin* <dberlin@dberlin.org
> <mailto:dberlin@dberlin.org>> wrote:
>
> J Kramer wrote:
> > All,
> >
> > At my workplace we use SVN to manage our software development. We
> > commit rather frequently as that seems good policy, but the size
> of the
> > SVN repository (on the server) is growing quite rapidly--too
> rapidly for
> > us.
>
> You need to quantify these terms with numbers.
>
>
> In the past week the repository has grown from about 500 MB to 1.7 GB.
> We should be able to take care of some of that by unversioning some
> things that are currently versioned, but most of our binary files have
> to be versioned.

How big are these files each?

>
>
>
> It depends on the size of the binary files, and how they are changing.
> If we are talking gigabyte files, you'll probably want to up the window
> size a bunch (at the cost of memory and time usage) to get better delta
> performance.
>
> This requires modifying one of the include files.
>
>
>
> Could you explain more about that?

In order to be able to deal with unlimited size files, without keeping
them all in memory at once to be able to perform deltas, we break the
streams we are performing deltas on into seperate "windows" of a certain
size. The default window size is 102400 bytes. That means any file you
store in subversion that gets stored as a set of deltas, will have
N/102400 delta windows, where N is the file size. The downside to this
scheme is that stream alignment ends up mattering a lot. If you end up
with >WINDOW SIZE amount of new data towards the very beginning, you are
going to throw off the entire rest of the delta stream, and it will
never see anything it thinks can be a copy (unless there is other
redundancy in the stream).

For source code files, 102,400 is a perfectly reasonable window size.

For something like video files, it's not.

You can up the window size, but in general, we are going to use (2 *
window_size) temporary memory to perform a delta.

> I'm not familiar with that parameter.
>
> Also, for files that currently exist in the repository that aren't
> needed, how can we expunge them from the repository?
You can't without dumping /filtering/loading.
We don't design the system to allow you to obliterate old data.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Received on Fri Jul 21 03:52:10 2006

This is an archived mail posted to the Subversion Users mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.