Hi Neels.
The "svn --version" ratio shows wild variation in the "max" and "avg"
columns across different data sets, while its ratio in the "min" column
stays around 1.0 as expected. There isn't any explanation for this
apart from timing errors is there?
- Julian
On Thu, 2010-09-23 at 15:09 +0200, Neels J Hofmeyr wrote:
> Hi everyone,
>
> in research for an article about svn 1.7, I setup my own benchmarks
> comparing trunk against branches/1.6.x.
>
> It's a pseudo-randomized test aimed at wc-ng. It does a "complete" tree
> creation, edit, copy, edit more, merge, resolve... cycle. Svn operations are
> timed by name. ra_local only. Each run is identical (same random seed).
>
> The whole runtime was measured, including I/O wait and whatnot. There was no
> other load on the machine and no swapping of memory to/fro disk.
>
>
> Results:
>
> Even though some operations are now a lot slower compared to 1.6.x, a few do
> get faster. *Overall*, in a working copy with "balanced" dir/file tree,
> trunk takes about 1.6 times the time 1.6.x takes (no pun).
>
> wc-ng already WINS in an insanely deep WC that's 100 dirs deep x 1 dir wide:
> 1.6.x's memory usage blows up like crazy for simple operations like 'svn
> update' etc.. Trunk no longer exhibits that problem, it stays as thin from
> start to end! Nice. In the same WC, trunk is about twice as fast as 1.6.x.
>
> wc-ng still fails run time wise at most other WC setups. The worst factor I
> saw was about 6 times slower. Overall, it's more a factor range of 1 to 2,
> and doing real actual work takes about 1.6 times as long as 1.6.x did.
>
> Tables...
>
>
> VARIATION TO EXPECT:
>
> There were multiple runs for each client version, and to determine
> variation, I compared runs of the same client version against each other. If
> they had taken identical amounts of time, all these numbers would be 1.0:
>
> 1x100_1.7_2.runs / 1x100_1.7_1.runs
> min max avg operation (unit is factor between runs)
> 0.997 1.006 0.972 status
> 0.970 1.346 1.119 resolved
> 1.026 1.005 1.019 resolve
> 0.994 0.854 0.976 propset
> 0.993 1.518 1.172 delete
> 1.010 0.986 0.995 update
> 1.045 0.760 0.888 --version
> 1.012 1.087 1.017 merge
> 0.995 1.036 1.025 switch
> 0.977 1.056 0.995 add
> 1.006 0.743 1.017 propdel
> 1.000 0.559 0.964 proplist
> 1.118 0.695 0.968 commit
> 1.015 0.760 0.979 prop mod
> 1.200 0.808 0.947 checkout
> 1.051 1.191 1.109 copy
> 1.001 0.969 0.982 TOTAL RUN
>
> So this is the variation to expect: up to about 20% variation from
> environmental factors. For the rest of the calculations, the various runs
> were averaged to reduce noise a little.
>
> I'm omitting most of the actual time numbers and am skipping straight to the
> factors between 1.6.x and trunk. If you want more numbers, or the script
> itself, just ask.
>
>
> FACTORS:
>
> legend:
>
> min: the shortest run with 1.7 took <x> times as long as the shortest
> run with 1.6.
>
> max: the longest run with 1.7 took <x> times as long as the longest run with
> 1.6.
>
> avg: the average of all the runs with 1.7 is <x> times as long as the
> average of all the runs with 1.6.
>
>
> -----
>
> 4x4: WC is 4 dir-levels deep and each subdir 4 dirs wide, plus each subdir
> has 4 files:
>
> 4x4_1.7_all.runs / 4x4_1.6_all.runs
> min max avg operation (unit is factor between runs)
> 5.686 6.093 5.203 status
> 4.739 20.004 6.958 resolved
> 1.088 0.348 0.896 resolve
> 1.163 0.279 1.113 propset
> 1.132 1.066 1.108 mkdir
> 2.369 1.237 1.523 update
> 0.914 0.246 0.477 --version
> 2.238 2.025 2.258 merge
> 1.463 1.389 1.409 switch
> 1.223 1.168 1.273 add
> 1.131 1.816 1.170 propdel
> 1.140 1.000 1.112 proplist
> 1.464 1.637 1.684 commit
> 1.174 1.797 1.173 prop mod
> 1.610 1.884 1.948 checkout
> 0.891 0.725 0.819 copy
> 1.992 2.114 2.019 delete
> 1.663 1.532 1.625 TOTAL RUN
>
> -> svn status, resolved, merge and checkout suck hard, with multiples of the
> 1.6.x run time.
> -> is that an improvement there for svn copy? (it's a wc->wc copy)
> -> everything else has the tendency to suck, getting up to 50% slower.
> (-> svn resolve and propset (initial creation of a prop) have very low
> factors for the 'max' column -- seems to suggest that 1.6.x had much higher
> variation on resolve and propset run times than trunk. -- milkin' it.)
>
>
> -----
>
> 1x100: WC is 1 dir level deep and has 100 subdirs and 100 files per subdir.
>
> 1x100_1.7_all.runs / 1x100_1.6_all.runs
> min max avg operation (unit is factor between runs)
> 3.144 3.017 3.241 status
> 3.658 3.792 3.456 resolved
> 1.065 0.837 1.065 resolve
> 1.031 0.938 1.015 propset
> 1.645 1.116 1.662 update
> 0.991 1.253 1.080 --version
> 1.385 1.575 1.467 merge
> 1.480 1.512 1.483 switch
> 1.009 0.999 1.018 add
> 1.001 1.413 1.024 propdel
> 1.070 1.593 1.071 proplist
> 1.235 1.637 1.270 commit
> 1.032 1.740 1.047 prop mod
> 1.432 3.704 2.867 checkout
> 0.968 1.148 1.068 copy
> 0.699 1.060 0.762 delete
> 1.324 1.344 1.329 TOTAL RUN
>
> -> compared to an evenly spread WC, checkout got worse but everything else
> seems to suck less.
>
> -----
>
> 100x1: WC is 100 dirs deep and has 1 subdir and 1 file per subdir (like a
> thin snake of subdirs)
>
> 100x1_1.7_all.runs / 100x1_1.6_all.runs
> min max avg operation (unit is factor between runs)
> 2.736 2.832 3.803 status
> 2.397 0.849 2.098 resolved
> 1.085 0.446 0.936 resolve
> 1.213 0.699 1.359 propset
> 1.165 1.891 1.102 mkdir
> 1.655 0.211 0.228 update
> 0.970 3.071 2.085 --version
> 1.454 1.162 1.482 merge
> 0.149 0.117 0.130 switch
> 1.241 0.883 1.011 add
> 1.240 1.639 1.516 propdel
> 1.151 1.716 1.301 proplist
> 1.188 1.242 1.512 commit
> 1.180 1.290 1.257 prop mod
> 1.409 0.133 0.148 checkout
> 0.961 1.371 1.053 copy
> 1.718 5.051 2.285 delete
> 0.632 0.598 0.617 TOTAL RUN
>
> -> with insane dir depth, svn switch, checkout and update TTLY WIN!!
> -> svn status, resolved still suck, slightly less hard.
> -> svn delete sucks the same.
> -> propdel, commit, merge suck moderately (50% worse than 1.6.x)
> -> everything else is basically the same as 1.6.x
>
> -----
>
> So this is the current status from my machine (ThinkPad T41p, Pentium M
> processor 1700MHz, ubuntu linux, and custom svn builds of course). I'll
> probably wait a little and run this thing again.
>
> I probably won't release the article before trunk's speed matches up.
>
> ~Neels
>
Received on 2010-09-23 16:25:12 CEST