Running benchmarks/suite1 on Windows

From: Johan Corveleyn <jcorvel_at_gmail.com>
Date: Tue, 21 Jun 2011 10:37:02 +0200

Hi,

I managed to get the benchmark 'suite1' (tools/dev/benchmarks/suite1,
which is used for the automated benchmarks [svnbench] on svn-qavm)
running on Windows. I did that by translating the 'run' script (bash)
to a Windows "batch" script (I'm not much of a "batch" programmer, but
with some trial and error, and the help of google ;-) ...).

I've put the run.bat in attachment here (just strip the .txt
extension). I'll commit it tonight (don't have access right now), but
some extra eyes in the meantime can't hurt :-).

There is one annoying issue remaining with the run script for Windows
(or the benchmark in general): I get an error when running the 100x1
tests (100 directory levels deep). I reduced it to 50x1, but still get
the error. It's probably some "max path component" or "max path
length" limit on Windows/NTFS that's causing this. Maybe some of the
Windows-devs knows how to work around this? Or should I reduce that
test to 25x1?

This is the error I'm getting:
[[[
Results for dir levels: 50 spread: 1
Traceback (most recent call last):
  File "C:\research\svn\client_build\trunk\tools\dev\benchmarks\suite1\benchmark.py",
line 629, in <module>
    cmd_run(*args)
  File "C:\research\svn\client_build\trunk\tools\dev\benchmarks\suite1\benchmark.py",
line 577, in cmd_run
    run(levels, spread, N)
  File "C:\research\svn\client_build\trunk\tools\dev\benchmarks\suite1\benchmark.py",
line 437, in run
    create_tree(trunk, levels, spread)
  File "C:\research\svn\client_build\trunk\tools\dev\benchmarks\suite1\benchmark.py",
line 247, in create_tree
    create_tree(dn, levels - 1, spread)

... <repeated a couple of times> ...

  File "C:\research\svn\client_build\trunk\tools\dev\benchmarks\suite1\benchmark.py",
line 247, in create_tree
    create_tree(dn, levels - 1, spread)
  File "C:\research\svn\client_build\trunk\tools\dev\benchmarks\suite1\benchmark.py",
line 240, in create_tree
    f = open(fn, 'w')
IOError: [Errno 2] No such file or directory:
'c:\\docume~1\\user\\locals~1\\temp\\tmpmddeli\\wc\\trunk\\dir_2\\dir_4\\d
ir_6\\dir_8\\dir_10\\dir_12\\dir_14\\dir_16\\dir_18\\dir_20\\dir_22\\dir_24\\dir_26\\dir_28\\dir_30\\dir_32\\dir_34\\dir
_36\\dir_38\\dir_40\\dir_42\\dir_44\\dir_46\\dir_48\\dir_50\\dir_52\\dir_54\\dir_56\\dir_58\\file_59'
]]]

For the rest it runs fine. Below are some results (only the 5x5 and
1x100) from comparing trunk_at_1137727 with 1.6.17 (SlikSVN binary) on
Windows XP (32-bit) with local disk, with Antivirus disabled.

Note that it took a *huge* time to run, even without the 100x1 tests
(~ 7 hours). I guess that shows how slow my machine really is ;-),
especially with 1.6. The performance improvements are definitely great
though (especially 'update', which is 80% faster with trunk! And also
'commit', with is down almost 60%).

--------------8<---------------

C:\research\svn\client_build\trunk\tools\dev\benchmarks\suite1>run
C:\research\svn\client_build\trunk\tools\dev\benchmarks\suite1\ma_20_06_2011-22332348
Started at ma 20/06/2011-22:33:23,48

---------------------------------------------------------------------

Results for dir levels: 5 spread: 5
Timings for 5x5_1.6
    N min max avg operation (unit is seconds)
    6 2678.55 2805.30 2735.33 TOTAL RUN
  816 0.08 28.97 0.28 add
   12 0.11 68.28 30.63 checkout
   48 54.75 132.92 96.62 commit
    6 0.13 9.13 1.85 copy
    6 33.42 44.73 36.60 delete
   12 53.86 74.17 67.24 merge
  918 0.09 0.92 0.10 mkdir
6318 0.08 1.45 0.10 prop mod
  306 0.08 0.11 0.09 propdel
12660 0.08 0.53 0.09 proplist
25242 0.08 4.38 0.09 propset
   12 0.08 0.11 0.10 resolve
   12 22.77 38.41 28.58 resolved
   84 0.45 11.55 1.51 status
    6 60.78 64.61 62.58 switch
   90 8.08 124.69 55.27 update

---
Timings for 5x5_trunk
    N   min     max     avg    operation  (unit is seconds)
    6 1345.23 1369.86 1359.09  TOTAL RUN
  816    0.08   13.16    0.17  add
   12    0.13   48.30   23.51  checkout
   48    2.48   77.02   40.89  commit
    6    0.11    0.28    0.15  copy
    6    4.55   11.41    7.32  delete
   12   38.56   51.66   44.50  merge
  918    0.08    0.63    0.09  mkdir
 6318    0.08    0.19    0.09  prop mod
  306    0.08    0.09    0.09  propdel
12660    0.08    0.42    0.08  proplist
25242    0.08    0.33    0.09  propset
   12    0.08    0.09    0.09  resolve
   12    1.42    6.50    2.98  resolved
   84    0.39    1.50    0.91  status
    6   23.08   25.20   24.11  switch
   90    0.42   41.83   10.84  update
---
COMPARE 5x5_1.6 to 5x5_trunk
  TOTAL RUN times: 2735.3 seconds avg for 5x5_1.6
                   1359.1 seconds avg for 5x5_trunk
      min              max              avg         operation
   0.50|-1333.313    0.49|-1435.437    0.50|-1376.242  TOTAL RUN
   1.00| +0.000     0.45|-15.813     0.62| -0.108   add
   1.15| +0.016     0.71|-19.985     0.77| -7.116   checkout
   0.05|-52.266     0.58|-55.907     0.42|-55.731   commit
   0.87| -0.016     0.03| -8.844     0.08| -1.701   copy
   0.14|-28.875     0.25|-33.327     0.20|-29.286   delete
   0.72|-15.296     0.70|-22.516     0.66|-22.740   merge
   0.84| -0.015     0.68| -0.297     0.89| -0.011   mkdir
   1.00| +0.000     0.13| -1.265     0.91| -0.008   prop mod
   1.00| +0.000     0.85| -0.016     0.91| -0.008   propdel
   1.00| +0.000     0.79| -0.109     0.94| -0.006   proplist
   1.00| +0.000     0.07| -4.047     0.91| -0.008   propset
   1.00| +0.000     0.86| -0.015     0.87| -0.013   resolve
   0.06|-21.343     0.17|-31.906     0.10|-25.591   resolved
   0.86| -0.063     0.13|-10.047     0.60| -0.604   status
   0.38|-37.704     0.39|-39.407     0.39|-38.472   switch
   0.05| -7.656     0.34|-82.859     0.20|-44.427   update
("1.23|+0.45"  means factor=1.23, difference in seconds = 0.45
factor < 1 or difference < 0 means '5x5_trunk' is faster than '5x5_1.6')
---------------------------------------------------------------------
Results for dir levels: 1  spread: 100
Timings for 1x100_1.6
    N   min     max     avg    operation  (unit is seconds)
    6   33.19   37.64   34.08  TOTAL RUN
    6    0.27    1.14    0.41  add
   12    0.11    1.53    0.43  checkout
   48    0.53    2.47    0.90  commit
    6    0.11    0.13    0.12  copy
    6    0.36    0.44    0.39  delete
   12    0.77    2.30    1.03  merge
  186    0.08    0.13    0.09  prop mod
   24    0.09    0.11    0.10  propdel
  288    0.08    0.11    0.09  proplist
  588    0.08    1.11    0.10  propset
   12    0.08    0.20    0.10  resolve
   12    0.09    0.13    0.11  resolved
   84    0.08    0.16    0.11  status
    6    0.34    0.36    0.35  switch
   90    0.11    1.00    0.24  update
---
Timings for 1x100_trunk
    N   min     max     avg    operation  (unit is seconds)
    6   30.66   32.14   30.97  TOTAL RUN
    6    0.22    0.22    0.22  add
   12    0.13    1.45    0.50  checkout
   48    0.16    2.39    0.79  commit
    6    0.11    0.17    0.15  copy
    6    0.20    0.22    0.21  delete
   12    0.80    2.02    0.99  merge
  186    0.08    0.09    0.09  prop mod
   24    0.08    0.09    0.09  propdel
  288    0.08    0.09    0.08  proplist
  588    0.08    0.30    0.09  propset
   12    0.08    0.09    0.08  resolve
   12    0.09    0.13    0.11  resolved
   84    0.08    0.11    0.09  status
    6    0.36    0.47    0.40  switch
   90    0.09    0.84    0.23  update
---
COMPARE 1x100_1.6 to 1x100_trunk
  TOTAL RUN times:  34.1 seconds avg for 1x100_1.6
                    31.0 seconds avg for 1x100_trunk
      min              max              avg         operation
   0.92| -2.532     0.85| -5.501     0.91| -3.102   TOTAL RUN
   0.82| -0.047     0.19| -0.922     0.53| -0.195   add
   1.15| +0.016     0.95| -0.078     1.15| +0.066   checkout
   0.29| -0.375     0.97| -0.079     0.88| -0.107   commit
   0.99| -0.001     1.38| +0.047     1.21| +0.026   copy
   0.57| -0.156     0.50| -0.219     0.53| -0.182   delete
   1.04| +0.032     0.88| -0.281     0.96| -0.042   merge
   1.00| +0.000     0.75| -0.031     0.91| -0.008   prop mod
   0.84| -0.015     0.85| -0.016     0.90| -0.009   propdel
   1.00| +0.000     0.85| -0.016     0.93| -0.007   proplist
   1.00| +0.000     0.27| -0.812     0.90| -0.009   propset
   1.00| +0.000     0.46| -0.109     0.83| -0.017   resolve
   1.00| +0.000     1.00| +0.000     0.97| -0.003   resolved
   1.00| +0.000     0.71| -0.046     0.86| -0.014   status
   1.05| +0.016     1.30| +0.109     1.17| +0.057   switch
   0.85| -0.016     0.84| -0.156     0.93| -0.018   update
("1.23|+0.45"  means factor=1.23, difference in seconds = 0.45
factor < 1 or difference < 0 means '1x100_trunk' is faster than '1x100_1.6')
=========================================================================
comparing averaged totals..."
Timings for total_1.6
    N   min     max     avg    operation  (unit is seconds)
   12   33.19 2805.30 1384.70  TOTAL RUN
  822    0.08   28.97    0.28  add
   24    0.11   68.28   15.53  checkout
   96    0.53  132.92   48.76  commit
   12    0.11    9.13    0.98  copy
   12    0.36   44.73   18.49  delete
   24    0.77   74.17   34.13  merge
  918    0.09    0.92    0.10  mkdir
 6504    0.08    1.45    0.10  prop mod
  330    0.08    0.11    0.09  propdel
12948    0.08    0.53    0.09  proplist
25830    0.08    4.38    0.09  propset
   24    0.08    0.20    0.10  resolve
   24    0.09   38.41   14.34  resolved
  168    0.08   11.55    0.81  status
   12    0.34   64.61   31.46  switch
  180    0.11  124.69   27.75  update
---
Timings for total_trunk
    N   min     max     avg    operation  (unit is seconds)
   12   30.66 1369.86  695.03  TOTAL RUN
  822    0.08   13.16    0.17  add
   24    0.13   48.30   12.00  checkout
   96    0.16   77.02   20.84  commit
   12    0.11    0.28    0.15  copy
   12    0.20   11.41    3.76  delete
   24    0.80   51.66   22.74  merge
  918    0.08    0.63    0.09  mkdir
 6504    0.08    0.19    0.09  prop mod
  330    0.08    0.09    0.09  propdel
12948    0.08    0.42    0.08  proplist
25830    0.08    0.33    0.09  propset
   24    0.08    0.09    0.08  resolve
   24    0.09    6.50    1.54  resolved
  168    0.08    1.50    0.50  status
   12    0.36   25.20   12.26  switch
  180    0.09   41.83    5.53  update
---
COMPARE total_1.6 to total_trunk
  TOTAL RUN times: 1384.7 seconds avg for total_1.6
                   695.0 seconds avg for total_trunk
      min              max              avg         operation
   0.92| -2.532     0.49|-1435.437    0.50|-689.672  TOTAL RUN
   1.00| +0.000     0.45|-15.813     0.62| -0.108   add
   1.15| +0.016     0.71|-19.985     0.77| -3.525   checkout
   0.29| -0.375     0.58|-55.907     0.43|-27.919   commit
   0.99| -0.001     0.03| -8.844     0.15| -0.837   copy
   0.57| -0.156     0.25|-33.327     0.20|-14.734   delete
   1.04| +0.032     0.70|-22.516     0.67|-11.391   merge
   0.84| -0.015     0.68| -0.297     0.89| -0.011   mkdir
   1.00| +0.000     0.13| -1.265     0.91| -0.008   prop mod
   1.00| +0.000     0.85| -0.016     0.91| -0.008   propdel
   1.00| +0.000     0.79| -0.109     0.93| -0.006   proplist
   1.00| +0.000     0.07| -4.047     0.91| -0.008   propset
   1.00| +0.000     0.46| -0.109     0.85| -0.015   resolve
   1.00| +0.000     0.17|-31.906     0.11|-12.797   resolved
   1.00| +0.000     0.13|-10.047     0.62| -0.309   status
   1.05| +0.016     0.39|-39.407     0.39|-19.207   switch
   0.85| -0.016     0.34|-82.859     0.20|-22.222   update
("1.23|+0.45"  means factor=1.23, difference in seconds = 0.45
factor < 1 or difference < 0 means 'total_trunk' is faster than 'total_1.6')
Had started at ma 20/06/2011-22:33:23,48,
       done at di 21/06/2011- 5:38:03,57
C:\research\svn\client_build\trunk\tools\dev\benchmarks\suite1\ma_20_06_2011-22332348
-- 
Johan

text/plain attachment: results.txt

text/plain attachment: run.bat.txt

Received on 2011-06-21 10:37:59 CEST

This message: [ Message body ]
Next message: Stefan Sperling: "Re: svn bisect"
Previous message: Julian Foad: "Re: svn bisect"
Next in thread: Daniel Klíma: "Re: Running benchmarks/suite1 on Windows"
Reply: Daniel Klíma: "Re: Running benchmarks/suite1 on Windows"

Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]