Hi folks,
I have some tools that make use of the perl bindings to Subversion 1.3.
As we are dealing with moderate amounts of data in long running
processes, we are managing memory using the SVN::Pool interface but it
appears that we are leaking memory. The leak I am focusing on right now
is related to how I am calling SVN::Fs::file_contents, (though I have
found other leaks in at least my usage of apply_textdelta and send_string).
I am including a simple test script which triggers the same behavior
exhibited in the actual tool. The script simply iterates through $count
creations of a svn repos, testing for and getting a stream for the file
at $path. If $leak is set, the stream object is read and the contents
assigned to a variable, causing some number of bytes to be leaked --
otherwise if no read happens, no memory leaks. Here is examples of the
summary output:
# 10 iterations without reading from the file stream:
$ ./svn_find_leak.pl /tmp/test6 /1/roles 10
Starting memory usage: 5628
Ending memory usage: 5884
growth in 10 iterations: 256 bytes
# 1000 iterations without reading from the file stream:
$ ./svn_find_leak.pl /tmp/test6 /1/roles 1000
Starting memory usage: 5628
Ending memory usage: 5884
growth in 100 iterations: 256 bytes
# 10 iterations reading from the file stream:
$ ./svn_find_leak.pl /tmp/test6 /1/roles 10 leak
Starting memory usage: 5628
Ending memory usage: 5948
growth in 10 iterations: 320 bytes
# 1000 iterations reading from the file stream:
$ ./svn_find_leak.pl /tmp/test6 /1/roles 1000 leak
Starting memory usage: 5628
Ending memory usage: 11904
growth in 1000 iterations: 6276 bytes
I think I am missing something in how I'm handling reading from the
stream. In the SVN::Core manpage there is this warning for SVN::Stream:
Note that some functions take a stream to read or write, while it does
not close it but still hold the reference to the handle. In this case
the handle won't be destroyed properly. You should always use correct
default pool before calling such functions.
Unfortunately, I don't understand this statement. :-) It seems to
indicate that one really ought to be making use of implicit 'default'
pools rather than keeping track of them yourself. I *do* know that our
early experimentation with using the 'default' pool behavior available
from SVN::Pool did not seem to work as expected. Pools marked as default
cleared through $pool->clear() or undefined were not removed from the
poolstack, which would simply continue to grow in depth as the process run.
I feel more comfortable explicitly allocating pools and passing them to
every SVN:: function, but if there are examples of better patterns, I am
willing to give them a try. Right now the best tips have come from
Garrett Rooney's article on the SVN client API and Greg Steins notes on
best practices linked from apr.apache.org.
Here is the test script. It will only work on Unix like systems and if
your ps works remotely like ours does. (seems to work (and leak) on both
RHEL 4 and FreeBSD 4.11). Also, we are using Perl 5.8.5 and 5.6.1.
#!/usr/local/bin/perl -w
use strict;
use lib qw(/home/y/lib/perl5/site_perl);
use SVN::Core;
use SVN::Repos;
use SVN::Fs;
my $repos = shift; # arg 1 - path to a subversion repository
my $path = shift; # arg 2 - path in repository to some file (must exist)
my $count = shift; # arg 3 - # of iterations to run for
my $leak = shift; # arg 4 - set to something to leak
my $start_mem = get_mem();
my $prev_mem = $start_mem;
for (0..$count) {
my $p = SVN::Pool->new(undef);
my $svn = SVN::Repos::open($repos, $p);
my $fs = $svn->fs();
my $youngest = $fs->youngest_rev($p);
my $root = $fs->revision_root($youngest, $p);
my $type = SVN::Fs::check_path($root, $path, $p);
if ($type == $SVN::Core::node_none) {
die "The file $path does not exist in $repos @ r$youngest!";
} elsif ($type == $SVN::Core::node_dir) {
die "$path is a directory in $repos @ r$youngest!";
}
# allocate a subpool for operations on this file. this is so that
# the test looks like our actual use-case, but if you do away with
# this and just allocate from $p, the leak will still occur.
my $sp = SVN::Pool->new($p);
my $stream = SVN::Fs::file_contents($root, $path, $sp);
# we only leak if we try to read from $stream
if (defined $leak) {
my $contents = <$stream>;
}
# cleanup
$stream = undef;
$sp = undef;
$svn = undef;
$p = undef;
my $mem = get_mem();
print "memory usage $mem\n";
print "memory growth ",$mem - $prev_mem, " bytes \n";
$prev_mem = $mem;
print "=" x 75, "\n";
}
my $end_mem = get_mem();
my $increase = ($end_mem - $start_mem);
print "Starting memory usage: $start_mem\n";
print "Ending memory usage: $end_mem\n";
print "growth in $count iterations: $increase bytes\n";
sub get_mem {
my $mem = `ps -o rss -p $$ | tail -1`;
chomp($mem);
return $mem;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Received on Wed Apr 12 23:10:44 2006