[PATCH] Problems dumping your repository?

THE PROBLEM:

   Those of you who have mature projects in Subversion repositories
   might have a little bit of problem dumping those repositories with
   `svnadmin dump' if the layers of deltification for any of your
   paths are too large. That is, if the undeltification of any path
   in any revision means applying a delta to a delta to a delta to a
   delta to ... well, enough times that you exhaust your stack space,
   the svnadmin application will be sacked by the OS.
   
   The Subversion repository is just such a repository (at least, when
   I try to dump it on my machine).
   
   There is a solution to this, that is both simple and complicated.

THE SIMPLE SOLUTION:

   What you should do is, in reverse revision order, run `svnadmin
   undeltify' on every X revisions, where X the number of recursive
   delta applications you think your system can hack. That is, if I
   have 1568 revisions in my repos, I might run:
   
      /* skip 1568 because HEAD is always fulltext */
      svnadmin undeltify repos_path 1557 /
      svnadmin undeltify repos_path 1547 /
      svnadmin undeltify repos_path 1537 /
      ...
      svnadmin undeltify repos_path 27 /
      svnadmin undeltify repos_path 17 /
      svnadmin undeltify repos_path 7 /
   
   You need to do this in reverse order, because ... well, I'll leave
   that as an exercise to the reader.
   
   To assist you in this, please see the included Perl script, which
   is run like so:
   
      ./undeltify.pl REPOS-PATH [START-REV:END_REV] [INCREMENT]
   
   So,for the example above, I would run:
   
      ./undeltify.pl 1557:7 10

THE (SLIGHTLY MORE) COMPLICATED PREREQUISITE FOR THE SIMPLE SOLUTION:

   As of revision 2092, the last revision of Subversion that works
   with your old repositories, `svnadmin undeltify' was horribly
   inefficient in how it interacted with Berkeley's locking and memory
   usage sub-systems. But the only real complication is that you need
   to apply the included patch to your revision-2092 source tree, and
   rebuild svnadmin.

   NOTE: This patch is Goodness for post-2092 svnadmin as well, so I
   will commit to the Subversion repos. But, of course, if you've
   rolled back your working copy to revision 2092, you won't have the
   committed version of this patch. Duh. Hence this email.

THE GOODZ:

   First, the Perl script undeltify.pl mentioned above, then the patch
   to svnadmin's main.c:

-------------------------- 8< snip 8< -------------------------------
#!/usr/bin/perl

use strict;

#############################################################################
# SETUP

my $svnadmin_cmd = '/home/cmpilato/bin/svnadmin-2092b';

#
#############################################################################

sub do_usage
{
    print "ERROR: usage: $0 REPOS [START-REV:END-REV] [INCREMENT]]\n\n";
    exit;
}

sub do_undeltify # ($repos, $start_rev, $end_rev, $increment)
{
    my $repos = shift @_;
    my $start_rev = shift @_;
    my $end_rev = shift @_;
    my $increment = shift @_;
    my $i = $start_rev;
    while (1)
    {
        print "--- Undeltifying revision $i...";
        `$svnadmin_cmd undeltify $repos $i /`;
        print "done.\n";
        if ($start_rev > $end_rev)
        {
            $i = $i - $increment;
            last if ($i < $end_rev);
        }
        else
        {
            $i = $i + $increment;
            last if ($i > $end_rev);
        }
    }
}

my $next_arg;
my $repos;
my $start_rev;
my $end_rev;
my $increment;
my $youngest;

# REPOS argument is required.
$next_arg = shift @ARGV;
if ($next_arg eq '')
{
    &do_usage();
}

# Use the REPOS argument to first figure out the youngest revision in
# the repository.
$repos = $next_arg;
$youngest = `$svnadmin_cmd youngest $repos`;
chomp $youngest;

# Setup the default argument list, a backwards walk of all revisions
# in the repository.
$start_rev = $youngest - 1;
$end_rev = 1;
$increment = 1;

# Parse the remaining arguments.
$next_arg = shift @ARGV;
if ($next_arg ne '')
{
    if ($next_arg =~ /^(\d+)\:(\d+)$/)
    {
        $start_rev = $1;
        $end_rev = $2;
        $next_arg = shift @ARGV;
        if ($next_arg ne '')
        {
            if ($next_arg =~ /^\d+$/)
            {
                $increment = $next_arg;
            }
            else
            {
                &do_usage();
            }
        }
    }
    elsif ($next_arg =~ /^\d+$/)
    {
        $increment = $next_arg;
    }
    else
    {
        &do_usage();
    }
}

# Validate the input.
if (($start_rev > $youngest)
    or ($end_rev > $youngest)
    or ($start_rev < 1)
    or ($end_rev < 1))
{
    print "ERROR: You've specified an invalid revision.\n";
    print "ERROR: Valid revisions are those between 1 and $youngest.\n\n";
    exit;
}

print "Undeltifying `$repos', revs $start_rev - $end_rev (by $increment).\n";
&do_undeltify ($repos, $start_rev, $end_rev, $increment);
print "Finished. Happy computing!\n\n";
exit;

-------------------------- 8< snip 8< -------------------------------

Index: ./subversion/svnadmin/main.c
===================================================================
--- ./subversion/svnadmin/main.c
+++ ./subversion/svnadmin/main.c Thu Jun 6 12:02:25 2002
@@ -102,6 +102,56 @@
 }
 
 
+static svn_error_t *
+do_deltify_undeltify (svn_fs_root_t *root,
+ const char *path,
+ int is_deltify,
+ apr_pool_t *pool)
+{
+ /* do the (un-)deltification */
+ printf ("%seltifying `%s'...", is_deltify ? "D" : "Und", path);
+ if (is_deltify)
+ SVN_ERR (svn_fs_deltify (root, path, 0, pool));
+ else
+ SVN_ERR (svn_fs_undeltify (root, path, 0, pool));
+ printf ("done.\n");
+ return SVN_NO_ERROR;
+}
+
+
+static svn_error_t *
+deltify_undeltify (svn_fs_root_t *root,
+ const char *path,
+ int is_deltify,
+ apr_pool_t *pool)
+{
+ apr_hash_t *entries;
+ apr_pool_t *subpool;
+ apr_hash_index_t *hi;
+ int is_dir;
+
+ /* (Un)deltify PATH. */
+ SVN_ERR (do_deltify_undeltify (root, path, is_deltify, pool));
+
+ /* If PATH is not a directory, we're done. */
+ SVN_ERR (svn_fs_is_dir (&is_dir, root, path, pool));
+ if (! is_dir)
+ return SVN_NO_ERROR;
+
+ /* Else, read PATH's entries and recurse. */
+ subpool = svn_pool_create (pool);
+ SVN_ERR (svn_fs_dir_entries (&entries, root, path, pool));
+ for (hi = apr_hash_first (pool, entries); hi; hi = apr_hash_next (hi))
+ {
+ const void *key;
+ apr_hash_this (hi, &key, NULL, NULL);
+ deltify_undeltify (root, svn_path_join (path, key, subpool),
+ is_deltify, subpool);
+ svn_pool_clear (subpool);
+ }
+ svn_pool_destroy (subpool);
+ return SVN_NO_ERROR;
+}
 
 
 /*** Argument parsing and usage. ***/
@@ -577,7 +627,6 @@
     case svnadmin_cmd_undeltify:
       {
         svn_revnum_t the_rev;
- int is_dir = 0;
         svn_fs_root_t *rev_root;
         const char *node;
         int is_deltify = (command == svnadmin_cmd_deltify);
@@ -599,22 +648,8 @@
         /* open the revision root */
         INT_ERR (svn_fs_revision_root (&rev_root, fs, the_rev, pool));
 
- /* see if PATH represents a directory (this doubles as an
- existence check!) */
- INT_ERR (svn_fs_is_dir (&is_dir, rev_root, node, pool));
-
         /* do the (un-)deltification */
- printf ("%seltifying `%s' in revision %" SVN_REVNUM_T_FMT "...",
- is_deltify ? "D" : "Und", node, the_rev);
- if (is_deltify)
- {
- INT_ERR (svn_fs_deltify (rev_root, node, is_dir ? 1 : 0, pool));
- }
- else
- {
- INT_ERR (svn_fs_undeltify (rev_root, node, is_dir ? 1 : 0, pool));
- }
- printf ("done.\n");
+ INT_ERR (deltify_undeltify (rev_root, node, is_deltify, pool));
       }
       break;
 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Thu Jun 6 20:37:28 2002