kfogel@collab.net wrote:
> J Robert Ray <jrray@imageworks.com> writes:
>
>>I am developing a script (using the svn perl bindings) to do a mass
>>rearrange of a large software project. In very simple terms, my
>>script does many, many SVN::Client::ls() (non-recursive) calls, with
>>the occasional SVN::Client::mkdir() and SVN::Client::move().
>>
>>These are the only calls I am making, and they are all against
>>respository URLs, I am not using a working copy.
>>
>>As I run my script, and it is off sniffing around the repository via
>>ls, I am running 'watch db_stat -c' in another terminal.
>>
>>""
>>150 Number of current lockers.
>>1681 Maximum number of lockers at any one time.
>>""
>>
>>The 'Number of current lockers' count grows rapidly and eventually
>>meets up with the max, and the script dies with a 400 error. I can
>>always go to the apache server log and dig up a message like this one:
>>
>>(20014)Error string not specified yet: Berkeley DB error while opening
>>'representations' table for filesystem
>>/mnt/d3/subversion/move_test/db:\nCannot allocate memory
>
>
> This is very exciting! We've encountered the same error message from
> Subversion's own repository a couple of times, and have heard
> occasional reports of it happening elsewhere. But this is the first
> thing resembling a reproduction recipe we've gotten.
>
> Can you share your script with us? (And the repository would be good
> too, although probably just looking at the script would be enough to
> start tracing down the bug in Subversion.)
Script is attached. I can't share my repository, it's corporate source
code and >3GB anyway...
>
>
>>Vital stats:
>>server: svn 1.0.6, apache 2.0.59
>>client: svn 1.1.0, perl 5.8.0, swig 1.3.21
>>berkeley db: 4.2.52
>
>
> If you upgrade the server to 1.1.0, does it still have the problem?
> (If you have time to try it with svnserve and svn://, that could be
> useful information too.)
Yes, 1.1.0 client and server has the same problem. When I try using
svn://, I get a slightly better diagnostic output. This error is
returned to the client:
Berkeley DB error: Berkeley DB error while opening 'nodes' table for
filesystem /net/soft_scratch/users/jrray/software_copy/db:
Cannot allocate memory: bdb: Logging region out of memory; you may need
to increase its size at ./svn_restructure.pl line 46
My DB_CONFIG settings are currently at factory defaults.
After the above failure, 'svnadmin recover' can no longer recover the
database:
Repository lock acquired.
Please wait; recovering the repository may take some time...
svn: DB_RUNRECOVERY: Fatal error, run database recovery
svn: bdb: Logging region out of memory; you may need to increase its size
svn: bdb: Recovery function for LSN 17571 340797 failed on backward pass
svn: bdb: PANIC: Cannot allocate memory
svn: bdb: PANIC: fatal region error detected; run recovery
svn: bdb: PANIC: fatal region error detected; run recovery
...
In a different test repository when I had this problem I bumped up the
DB_CONFIG values to look like this:
set_cachesize 0 8388608 8
set_lg_regionmax 524288
set_lg_bsize 2097152
set_lg_max 8388608
After that I could recover the database, and my script runs without
hitting the lock limit it seems. I'm ready to retry this with another
test repository if you want to guide me through some experiments.
This is all starting to look like a berkeley db configuration issue to
me and not necessarily a subversion bug, but that's your determination
to make.
There is still that neon problem, though.
Thanks for your help,
- Robert
#!/usr/local/bin/perl
use strict;
use warnings;
use File::Spec;
use SVN::Client;
use constant SVN_ROOT => 'http://localhost/svn/repos';
use constant NEED_SLASH => 1;
my $svn;
sub dir_exists ($)
{
my ($url) = @_;
$url =~ s|\%|%25|g;
eval {
$url .= '/' if NEED_SLASH;
print "ls $url\n";
my $result = $svn->ls($url, 'HEAD', 0);
};
if ($@) {
die $@ unless $@ =~ /non-existent in that revision/;
return 0;
}
return 1;
}
sub parent_dir ($)
{
my ($url) = @_;
# determine the parent URL of this URL
# strip trailing /
$url =~ s|/+$||;
die "Could not determine parent directory of '$url'\n"
unless $url =~ m|^(.*?)/[^/]+$|;
return $1;
}
my %existing_dir_cache;
sub recursive_mkdir ($);
sub recursive_mkdir ($)
{
my ($dest_url) = @_;
return if exists $existing_dir_cache{$dest_url};
if (dir_exists $dest_url) {
$existing_dir_cache{$dest_url} = 1;
return;
}
# path doesn't exist, make its parent
recursive_mkdir(parent_dir($dest_url));
$dest_url =~ s|\%|%25|g;
# now make this path
print "mkdir $dest_url\n";
$svn->mkdir([$dest_url]);
}
sub ensure_dest ($)
{
my ($dest_url) = @_;
# ensure that the destination does not already exist,
# and then create all the intermediate subdirectories
# up to the destination path
die "Destination path already exists: '$dest_url'\n"
if dir_exists $dest_url;
# we want to make $parent_url exist
recursive_mkdir(parent_dir($dest_url));
}
sub safe_move ($$)
{
my ($src_url, $dst_url) = @_;
ensure_dest($dst_url);
$src_url =~ s|\%|%25|g;
$dst_url =~ s|\%|%25|g;
$src_url =~ s|/+$||;
print "mv $src_url $dst_url\n";
$svn->move($src_url, 'HEAD', $dst_url, 0);
}
sub migrate_branch (\%$)
{
my ($args, $branch_label) = @_;
my $top_url = join('/', SVN_ROOT, $args->{from_root}, $branch_label);
# find branches that need to be moved
$top_url .= '/' if NEED_SLASH;
print "ls $top_url\n";
my $branches = $svn->ls($top_url, 'HEAD', 0);
for my $branch (keys %{$branches}) {
my $subpath;
my $branch_url = join('/', SVN_ROOT, $args->{from_root}, $branch_label, $branch);
# speed optimization: get the directory listing of the directory above $src_url,
# and see if the subdir exists in that listing. This is faster than fetching
# the contents of the branch
my @from_path_bits = split('/', $args->{from_path});
if (@from_path_bits) {
my $wanted_dir = pop @from_path_bits;
my $search_dir = join('/', $branch_url, @from_path_bits);
$search_dir =~ s|\%|%25|g;
# this directory must contain an entry '$wanted_dir'
my $exists = {};
eval {
$search_dir .= '/' if NEED_SLASH;
print "ls $search_dir\n";
$exists = $svn->ls($search_dir, 'HEAD', 0);
};
if ($@) {
die $@ unless $@ =~ /non-existent in that revision/;
print "ls failed with $@\n";
}
next unless exists $exists->{$wanted_dir};
}
my $src_url = join('/', $branch_url, $args->{from_path});
my $dst_url = join('/', SVN_ROOT, $args->{to_root}, $branch_label, $branch, $args->{to_path});
safe_move($src_url, $dst_url);
}
}
sub migrate_path (%)
{
my (%args) = @_;
# trunk is easy
my $src_url = $args{from_path}
? join('/', SVN_ROOT, $args{from_root}, 'trunk', $args{from_path})
: join('/', SVN_ROOT, $args{from_root}, 'trunk');
my $dst_url = join('/', SVN_ROOT, $args{to_root}, 'trunk', $args{to_path});
return unless dir_exists $src_url;
# find branches and tags that need to be moved
migrate_branch(%args, 'branches');
migrate_branch(%args, 'tags');
safe_move($src_url, $dst_url);
}
my %move_paths = (
'libs/lib1' => 'src/libs/lib',
'libs/lib2' => 'src/libs/lib2',
);
while (my ($key, $value) = each %move_paths) {
my ($from_root, $from_path) = split '/', $key, 2;
# create a new $svn each loop
$svn = new SVN::Client;
migrate_path(from_root => $from_root,
from_path => $from_path || '',
to_root => 'new_home',
to_path => $value);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Received on Wed Oct 13 22:03:46 2004