Sorry, I forgot to CC the list. I lurk on a lot of lists, but I very
rarely post. Thanks for catching the oversight. (Note to self: Reply
to all is your friend).
Thanks,
Kirby
cmpilato@collab.net wrote:
> No need for this to NOT be public (hope you agree, Kirby).
>
> ------- Start of forwarded message -------
> Message-ID: <3C5AEB5D.10007@birddog.com>
> Date: Fri, 01 Feb 2002 13:24:13 -0600
> From: "Kirby C. Bohling" <kbohling@birddog.com>
> MIME-Version: 1.0
> To: cmpilato@collab.net
> Subject: Re: Disk/Memory Consumption of SVN
> References: <3C5AE289.2050805@birddog.com> <x7elk5f2uh.fsf@pascal.ch.collab.net>
> Content-Type: text/plain; charset=us-ascii; format=flowed
> Content-Transfer-Encoding: 7bit
>
> Good god, you guys are repsonsive... The beauty of interested
> developers... Everytime we install the software at our shop, we do tag
> the CVS repository do an export, and build from the export.
>
> Went to a heavily updated file and did a CVS log and grepped the output
> for the tag "INSTALL_RELEASE.*", which is every install we do. We do
> frequent installs, average about 1 a business day.
>
> For simplicity here is the script:
>
> #!/bin/sh
>
> CVS_MODULE=source
> SVN_DIRNAME=second
> echo "Starting conversion"
>
> tac installTags | (
> read oldTag
>
> rm -rf /u1/repository/second initial second
>
> echo "Creating subversion repository"
> svnadmin create /u1/repository/second
>
> echo "Checking out $oldTag of $CVS_MODULE for initial import"
> cvs -Q export -d initial -r $oldTag $CVS_MODULE
>
> echo "Importing into subversion"
> svn import -q -m"Initial Import of revision $oldTag"
> file:///u1/repository/second initial
>
> echo "Checking a working copy out of subversion"
> svn co -q file:///u1/repository/second
>
> while read newTag
> do
> date
> echo "Old Tag: $oldTag New: $newTag" ;
> rm -rf old new
> cvs -Q export -d old -r $oldTag $CVS_MODULE
> cvs -Q export -d new -r $newTag $CVS_MODULE
>
> echo "Diff'ing the versions."
> diff -u -N -r old new > patch-$oldTag-$newTag
>
> echo "Applying the patch"
> ( cd second ; patch -p1 < ../patch-$oldTag-$newTag )
> echo "$newTag" > $SVN_DIRNAME/cvsTag
>
> while ( cd second ; svn status | grep "^?" > /dev/null )
> do
> ( cd second ; svn status | grep "^?" | cut -f6- -d" " |
> xargs -r svn add )
> done
>
> ( cd second ; svn status | grep '^!' | cut -f6- -d" " | xargs
> -r svn delete )
>
> if svn status "$SVN_DIRNAME" | grep -v "^[AMRD]"
> then
> exit 1
> fi
>
> echo "Checking into subversion"
> svn ci -m"Historical modification from CVS Tag: $oldTag to
> $newTag" "$SVN_DIRNAME"
>
> echo "Sleeping for 1 seconds."
> sleep 1
> oldTag=$newTag
> done
> )
>
> It isn't particularly efficient, but it gets me some data. The only
> cavet is if you pressed control C during a tag, this will remove the
> files and recreate them in the next repository. I have watched those
> looked for those and remove the troublesome tags from installTags.
>
> I tried using db_archive there, and it told me this:
>
> db_archive: region error detected; run recovery.
> db_archive: Ignoring log file: log.0000000082: unsupported log version 5
> db_archive: PANIC: Invalid argument
> db_archive: open: DB_RUNRECOVERY: Fatal error, run database recovery
>
> So then I did:
>
> db_recover
>
> Now all my __db files are gone... oops! Small path problem... I was
> using the db3 utils that come with my RedHat path, as opposed to the
> utils that I built for this. Opps! I will re-run and find out what is
> going on. Just FYI, this took 4.2 hours of running check in. It is my
> opinion that this is a VM problem because I am using all of the ram on
> the box. I have all the patch files so I can estimate the rough size of
> changes if you are interested in the performance of this. I will send
> more information when I get the stuff rebuilt. There are roughly 800MB
> of log files so it appears to be a log file issue, not a repository
> issue. By the write times it appears to have just cycled thru the 10M
> files and create a new one, and the blowing away the old ones would be safe.
>
> Thanks for the feed back,
>
> Kirby
>
>
> cmpilato@collab.net wrote:
>
>>Kirby, thanks for your interest in Subversion. Some thoughts:
>>
>> - Berkeley DB uses a write-ahead journaling filesystem, and the logs
>> it uses (found in path/to/repos/db/*.log) can get to be really
>> big, really fast. Have you tried using the db_archive utility
>> (which comes with BDB) to determine which of those log files can
>> be safely removed, and then removed those? (NOTE to the dev-list:
>> we really need to automate this process, code-internally if
>> possible)
>>
>> - When you say you "exported out successive version of CVS", does
>> this means you simply cycled through all the tags that were made
>> over the course of the 9 months? Just wondering.
>>
>>"Kirby C. Bohling" <kbohling@birddog.com> writes:
>>
>>
>>
>>>SubVersionists,
>>>
>>>I am currently interested in replacing CVS. I have evaluated several
>>>different source repositories. SVN is very interesting, and looks like
>>>a candidate for my replacement (I will wait for it to be finished of
>>>course).
>>>
>>>So in the interest of tinkering with it, I took my current CVS
>>>repository and checked out about 9 months worth of install tags (think
>>>230 revisions). The source code is roughly 10-12MB. I did an export
>>>
>>>from CVS, checked into SVN then exported out successive versions of CVS
>>
>>>did a diff, applied the patch and SVN checked in the result. Once I got
>>>this up and running then I could play with it to my hearts content with
>>>a real amount of data in it.
>>>
>>>I have two concerns to point out. I have searched the archives for
>>>similar things, but I have yet to find them. One is that the same
>>>repository in CVS takes about 130MB of disk space, and has 2 years worth
>>>of revisions. The SVN was at 1.3GB when I completely ran the filesystem
>>>out of space and had 8-9 months and was missing several of the huge
>>>files from early in the CVS tree that had been removed. The second is
>>>while check in's take a lot of memory, check outs take more. I have
>>>roughly 300MB of VM on my linux machine, and I literally couldn't check
>>>out trees that I had used for a year in CVS. The directory had 1000
>>>5-10K files that I was using as data for a regression test. I can give
>>>you as much or as little information as you want if this is a new issue.
>>> I probably can't give you the real data (NDA's are like that *grin*).
>>>
>>>I think SVN has a serious shot at being a CVS killer, and addressing
>>>these two issues will go a long way with me to getting me to switch when
>>>SVN is ready for prime time. I was unsure if I should present them here
>>>or put them in as bugs/features. My apologies if I choose poorly.
>>>
>>> Thanks,
>>> Kirby
>>>
>>>
>>>---------------------------------------------------------------------
>>>To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
>>>For additional commands, e-mail: dev-help@subversion.tigris.org
>>>
>>>
>
>
>
> ------- End of forwarded message -------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Sat Oct 21 14:37:03 2006