Yes, that basically will have to be our plan... going forward. Now to try to deal with what we have right now.
I am a bit puzzled why the commit requires any information about other folders seemingly not associated in any way with this particular commit though. I'm sure the database has it for its reasons, so I won't question that too much.
-----Original Message-----
From: Andy Levy [mailto:andy.levy_at_gmail.com]
Sent: Wednesday, September 28, 2011 9:16 AM
To: Trevor Schaffer
Cc: Stefan Sperling; users_at_subversion.apache.org
Subject: Re: revs files growing over time, relatively
On Wed, Sep 28, 2011 at 11:01, Trevor Schaffer
<TrevorSchaffer_at_smarttech.com> wrote:
> We definitely use svn copy for revisions, but I think the issue is because our tags are too flat vs not flat enough.
>
> E.g. tags/builds/ is where we put all of our tags (all done with svn copy)
> And over time, this folder has over 7000 tags in it. So now, ever new tag we put into /tags/builds has a listing of every other tag from tags/builds in the revs/db/<rev> file. Now when you have 4 lines in each rev file for each entry in the tags folder, that's 28000 lines of text, which gives us the roughly 600KB of data. And you can see how this grows over time. So obviously, in hindsight... we need to lessen the tags significantly, but current processes in our company will not allow us to do that quickly, or easily.
Would it be possible to clean up your tags directory, maybe on a
quarterly or annual basis? So instead of having a flat /tags/builds
"dumping ground", you have;
/tags/builds/
/tags/builds/2011Q1
/tags/builds/2011Q2
/tags/builds/2011Q3
/tags/builds/2011Q4
Your tags go into /tags/builds as they do today, but then at the end
of each quarter, you move that quarter's builds into the corresponding
directory.
> -----Original Message-----
> From: Stefan Sperling [mailto:stsp_at_elego.de]
> Sent: Wednesday, September 28, 2011 8:23 AM
> To: Trevor Schaffer
> Cc: users_at_subversion.apache.org
> Subject: Re: revs files growing over time, relatively
>
> On Wed, Sep 28, 2011 at 07:57:35AM -0600, Trevor Schaffer wrote:
>> Server: Fedora Core 13 64-bit
>> SVN: 1.6.16 running on Apache 2.2.16
>>
>> We have a repository that currently is in the 128K revision count, and is now standing at 40+GB in size. Over time it's been growing faster, and we noticed a trend on svn copies... the db/revs/* files are getting bigger over time.
>>
>> After cracking one open, and another random commit, I saw that the
>> commit entry not only lists information about the current path and
>> it's siblings, but it also enumerates all parents, and their siblings
>> as well... all the way to the root.
>
> Yes, this is how "cheap copies" in Subversion work.
> There is no way of changing this.
> It's called the "bubble up effect" which is described neatly at
> http://red-bean.com/kfogel/beautiful-code/bc-chapter-02.html
>
>> This is a big issue for us, as we
>> do a lot of build tagging, which means that every build tag commit
>> lists the other thousands of build tags as well, which is why they are
>> growing over time. At the current size of 600KB/commit for a single
>> svn copy into our /tags area, performing 20,000 commits covers about
>> 12GB of size, which I think is quite significant. Truthfully,
>> probably 30+GB of this repo is just svn copies worth of commits.
>
> I am surprised that the copy references take that much space.
> How deeply nested are the tag folders in the tree?
> You could try to flatten out the hierarchy of your repository.
>
>> In the meantime, I'm going to try to dump+load the repo in the hopes that there were some optimizations in svn 1.6 that will help. We've had this repo running for nearly 5 years, since svn 1.4.
>
> Yes, try that. 1.6 and upwards support "representation-sharing" which
> might cut down the size of the repository significantly.
>
> You could also try packing your repositories with 'svnadmin pack'.
> This will cause the repository to use less inodes and free up space
> lost to disk blocks which aren't completely filled up with data.
>
> I am curious about how you create tags. Do you really run 'svn copy'
> or do you maybe use 'svn import' to create them (which would bloat the
> repository significantly if representation-sharing is disabled)?
>
>
>
Received on 2011-09-28 17:22:02 CEST