Delta combiner stress test results

I ran some stress tests on the delta combiner today, using revision
2703, and comparisons whth /trunk at the same revision. Here's how I tested:

    * I used the dump file for the Subversion repository (up to revision
      2662) to create two repositories, one with the branch svn and one
      with the trunk svn. Here's what the repositories look like, after
      cleaning out the log files:

[brane@silmaril s]$ ll repo-trunk/db/
total 57921
-rw-r--r-- 1 brane None 328 Jul 25 06:06 DB_CONFIG
-rw-r--r-- 1 brane None 8192 Jul 25 06:06 __db.001
-rw-r--r-- 1 brane None 270336 Jul 25 06:06 __db.002
-rw-r--r-- 1 brane None 327680 Jul 25 06:06 __db.003
-rw-r--r-- 1 brane None 704512 Jul 25 06:06 __db.004
-rw-r--r-- 1 brane None 16384 Jul 25 06:06 __db.005
-rw-r--r-- 1 brane None 1826816 Jul 25 22:16 changes
-rw-r--r-- 1 brane None 16384 Jul 25 22:16 copies
-rw-r--r-- 1 brane None 1720320 Jul 25 22:16 nodes
-rw-r--r-- 1 brane None 2097152 Jul 25 22:16 representations
-rw-r--r-- 1 brane None 73728 Jul 25 22:16 revisions
-rw-r--r-- 1 brane None 49037312 Jul 25 22:16 strings
-rw-r--r-- 1 brane None 3211264 Jul 25 22:16 transactions
[brane@silmaril s]$ ll repo-branch/db/
total 41033
-rw-r--r-- 1 brane None 328 Jul 25 10:13 DB_CONFIG
-rw-r--r-- 1 brane None 8192 Jul 25 10:13 __db.001
-rw-r--r-- 1 brane None 270336 Jul 25 10:13 __db.002
-rw-r--r-- 1 brane None 327680 Jul 25 10:13 __db.003
-rw-r--r-- 1 brane None 704512 Jul 25 10:13 __db.004
-rw-r--r-- 1 brane None 16384 Jul 25 10:13 __db.005
-rw-r--r-- 1 brane None 1826816 Jul 25 21:46 changes
-rw-r--r-- 1 brane None 16384 Jul 25 21:46 copies
-rw-r--r-- 1 brane None 1720320 Jul 25 21:46 nodes
-rw-r--r-- 1 brane None 2088960 Jul 25 21:46 representations
-rw-r--r-- 1 brane None 73728 Jul 25 21:46 revisions
-rw-r--r-- 1 brane None 31752192 Jul 25 21:46 strings
-rw-r--r-- 1 brane None 3211264 Jul 25 21:46 transactions
    

      The branch repo is smaller, because it deltifies files regardless
      of their size. On the trunk, files larger than the delta window
      (100k) aren't deltified.

    * I created out four working copies, to cover all the possible
      combinations: trunk svn/trunk repo, trunk svn/branch repo, branch
      svn/trunk repo and branch svn/branch svn.

    * I then did the following tests in all four flavours:
      svn co -r500 # Check out an early version -- lots of
      undeltification, few files
      svn up -r2500 # Move to recent version -- less undeltification,
      many files
      svn up -r1500 # Move to older version -- more undeltification,
      less files

    * After each set of tests, I recursively compared the working copies
      (except the .svn directories). All the working copes were
      identical, _except_ that the trunk svn/branch repo combination
      corrupted files larget than 100k. I assume there's a latent bug in
      the undeltification code on /trunk that shows up when trying to
      undeltify files larger than the window size. We didn't notice it
      because there are no such files in our repositories. The branch
      sbuversion doesn't have this problem.

The timing results are in the attached table. Note that the results are
skewed, especially the "svn up" ones, because I measured ra_local. Based
on how I observed "svn" to behave over ra_dav, I estimate that working
copy handling represents a constant 60s and 6-7M of the results.

The tests show that operations on the branch repo are slightly more
expensive, because there's more undeltification going on. On the plus
side, the branch repo is 30% smaller.

Memory usage in the branch svn was much more constant than in the trunk
svn, but both versions displayed a slight but continuous increase in
working set size. We probably have a memory leak somewhere.

Conclusion: The combiner is ready to be merged on the mainline. Please
test the code in /branches/issue-531-dev on your repositories, and let
me know the results. I'd be especially interested in results from really
huge repositores, e.g., the Linux kernel archives I know some of you
have created.

If everything goes well, I'll merge the combiner onto the trunk on Monday.

-- 
Brane ÄŒibej   <brane_at_xbc.nu>   http://www.xbc.nu/brane/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Thu Jul 25 23:16:54 2002