On Tue, Sep 22, 2015 at 12:57 AM, Johan Corveleyn <jcorvel_at_gmail.com> wrote:
> Last week I performed some performance tests on various types of
> backends (FSFS formats 6 and 7, and FSX) served by 1.9, on machines
> that are similar to our company's production svn setup. This is a
> Solaris Sparc t5 with back-end on a Netapp cluster mounted via NFS;
> client is a Windows client over a 100 Mbit LAN.
>
> Disclaimer: this is not a scientific test. Most parts were not fully
> isolated from production, with normal fluctuations in available
> bandwidth (netapp cluster, network, ...). YMMV.
>
>
> == Summary ==
>
> FSFS f7 (packed) and FSFS f6 without dir-deltification (created by
> 1.8, unpacked) go head to head. Large log/blame-requests and export
> are somewhat faster on f7, smaller log/blame/list-requests are
> somewhat faster on f6_1.8. Disk storage: f7 is 40% smaller than f6_1.8
> (probably because of dir-deltification). It seems that f7 is faster
> when disk caches are cold (but this is hard to test / quantify (how
> cold is "cold")). I think I'll go with f7 for the cold cache
> performance on large requests, the smaller footprint (and better
> admin-ability because of the packed form), and the faster performance
> on 'export'.
>
> Further:
>
> * f6 with dir-deltification (the default when creating a
> --compatible-version=1.8 repository with svnadmin 1.9) is a no-go:
> it's 2 to 3 times slower than f7 and f6_1.8 for log and blame.
>
> * The --block-read option makes f7 even faster for large requests
> (record numbers for export, huge log and huge blame requests), but
> also much slower for small requests. With block-size=8 the advantage
> for large requests mostly remains, while small requests get a bit
> faster (but still slower than f7 without block-read, or especially
> f6_1.8). That leaves me with a dilemma: which use-case is more
> important. I think I won't use block-read atm ... would be nice if
> those regressions for small requests could be improved though
> (ideally, the system would be able to guess whether block-read is
> interesting for a particular high-level request).
>
> * FSX is rather slow in most tests (specifically: it's fine when disks
> are cold, but doesn't speed up much when disk caches become hot -- cpu
> usage is permanently high though, so I think FSX is cpu bound in many
> of my tests). Also: not included in the numbers below, but export with
> FSX was significantly faster with --cache-revprops *disabled* (up to
> around 120 seconds vs 165 seconds when disk caches were hot). In other
> words: although I did run all FSX tests with cache-revprops enabled,
> that seems to slow down export.
>
> * Unpacked f7 and FSX are rather slow (didn't fully test them, it was
> immediately obvious) -- these back-ends are best used in packed form.
> OTOH, f6 gest best performance in unpacked form (pack manifest lookup
> is overhead).
>
>
> == Details & Numbers ==
>
> Backend variations:
> * f6_1.8
> repos created & loaded with 1.8 (no dir/prop-deltif)
> on-disk size: 10 GB
> server: svnserve -d --foreground -M128 -c0 -r repos_1.8
> * f6_1.9
> repos created with 1.9-beta1 with --compatible-version 1.8
> & loaded with 1.9-beta1 (has dir/prop-deltif enabled by default)
> on-disk size: 6.1 GB
> server: svnserve -d --foreground -M128 -c0 -r repos_1.9_f6
> * f7_pkd
> repos created & loaded with 1.9-beta1 + packed
> on-disk size: 5.8 GB
> server: svnserve -d --foreground -M128 -c0 -r repos_1.9_f7_packed
> * f7_pkd_br
> same as above, but with block-read enabled
> server: svnserve -d --foreground -M128 -c0 --block-read yes -r
> repos_1.9_f7_packed
> * f7_pkd_br8
> same as above, but with block-read enabled, and block-size=8
(fsfs.conf)
> server: svnserve -d --foreground -M128 -c0 --block-read yes -r
> repos_1.9_f7_packed
> * fsx_pkd
> repos created with --fs-type fsx & loaded with 1.9-beta1 + packed
> on-disk size: 5.3 GB
> server: svnserve -d --foreground -M128 -c0 --cache-revprops yes -r
> repos_1.9_fsx_packed
>
> svnserve: 1.9.0-beta1 (provided by philip - Wandisco) on Solaris Sparc
> client: svnbench 1.9.1 x64 (Sliksvn) on Windows 7
> Network between client and server: 100 Mbit LAN
>
> Hardware:
> * Server: Solaris t5 sparc x64 (16 x sparcv9 3600 Mhz) 24 GB RAM
> * Back-end: Netapp cluster (don't know the details), mounted via NFS
> (The volume used here is relatively slow -- we have faster arrays for
> other systems, but this one is what we have for SVN right now)
>
> Test Method:
> * Per backend variation:
> * Per type of test (export, log subtree, blame-huge, list, log file,
> blame-normal):
> * Start svnserve
> * Run test from client 4 times
> * Stop svnserve
>
> Infrastructure observations:
> * 'iostat' on the backend volume showed high busy-percentages for each
> first run, the three subsequent iterations consistently showed 0 % busy.
> First run is still interesting, showing "cold" results.
> * 'top' showed low cpu usage for each first run (waiting for I/O), high
cpu
> percentages for the three subsequent iterations.
>
> Times in seconds. The tables below are best viewed with a fixed-width
font.
>
> null-export subtree (2,638 dirs; 21,713 files = 178 MB; 129,864 props = 2
MB)
> ===================
> run \ repos | f6_1.8 | f6_1.9 | f7_pkd | f7_pkd_br | f7_pkd_br8 | fsx_pkd
|
>
-----------------------------------------------------------------------------
> 1st (~cold) | 1053 | 147 | 374 | 162 | 73 | 227
|
> 2nd | 81 | 71 | 67 | 85 | 56 | 165
|
> 3rd | 80 | 81 | 65 | 86 | 56 | 167
|
> 4th | 79 | 65 | 65 | 85 | 57 | 167
|
>
> null-log -v subtree (62,458 revs; 98,900 msg lines; 299,171 changes)
> ===================
> run \ repos | f6_1.8 | f6_1.9 | f7_pkd | f7_pkd_br | f7_pkd_br8 | fsx_pkd
|
>
-----------------------------------------------------------------------------
> 1st (~cold) | 1558 | 802 | 611 | 176 | 152 | 342
|
> 2nd | 291 | 865 | 252 | 126 | 136 | 211
|
> 3rd | 272 | 742 | 229 | 130 | 134 | 213
|
> 4th | 266 | 722 | 232 | 133 | 134 | 208
|
>
> null-blame huge-deep-history (13,998 revs; 13,997 deltas; 33873 MB in
deltas)
> ============================
> run \ repos | f6_1.8 | f6_1.9 | f7_pkd | f7_pkd_br | f7_pkd_br8 | fsx_pkd
|
>
-----------------------------------------------------------------------------
> 1st (~cold) | 531 | 968 | 351 | 254 | 381 | 308
|
> 2nd | 342 | 585 | 340 | 230 | 293 | 293
|
> 3rd | 332 | 555 | 349 | 232 | 241 | 298
|
> 4th | 332 | 582 | 308 | 233 | 239 | 319
|
>
> null-list -R subtree (2,639 dirs; 21,713 files; 0 locks)
> ====================
> run \ repos | f6_1.8 | f6_1.9 | f7_pkd | f7_pkd_br | f7_pkd_br8 | fsx_pkd
|
>
-----------------------------------------------------------------------------
> 1st (~cold) | 6.8 | 11.5 | 6.7 | 10.7 | 7.0 | 16.2
|
> 2nd | 5.3 | 7.0 | 6.3 | 10.1 | 6.9 | 15.0
|
> 3rd | 4.9 | 6.2 | 6.3 | 9.9 | 6.8 | 14.1
|
> 4th | 5.0 | 6.4 | 6.3 | 10.2 | 6.8 | 15.2
|
>
> null-log -v file (434 revs; 957 msg lines; 13,955 changes)
> ================
> run \ repos | f6_1.8 | f6_1.9 | f7_pkd | f7_pkd_br | f7_pkd_br8 | fsx_pkd
|
>
-----------------------------------------------------------------------------
> 1st (~cold) | 17.5 | 30.7 | 15.3 | 23.7 | 8.0 | 16.5
|
> 2nd | 3.8 | 10.2 | 3.9 | 14.1 | 6.8 | 11.6
|
> 3rd | 3.6 | 8.4 | 3.9 | 13.9 | 7.1 | 11.4
|
> 4th | 3.5 | 8.6 | 3.9 | 13.7 | 6.8 | 11.3
|
>
> null-blame normal (542 revs; 542 deltas; 22 MB in deltas)
> =================
> run \ repos | f6_1.8 | f6_1.9 | f7_pkd | f7_pkd_br | f7_pkd_br8 | fsx_pkd
|
>
-----------------------------------------------------------------------------
> 1st (~cold) | 6.9 | 27.6 | 19.9 | 38.1 | 15.7 | 29.7
|
> 2nd | 4.1 | 15.8 | 5.3 | 32.1 | 14.2 | 27.6
|
> 3rd | 4.1 | 13.3 | 5.3 | 33.3 | 14.6 | 28.0
|
> 4th | 4.1 | 13.3 | 5.5 | 32.6 | 14.5 | 28.0
|
>
For those who are interested: a couple of months ago we did the final
migration of our production repository to FSFS format 7 (dump/load-ed from
old format 3 (svn 1.5) repository). It's running very smoothly ever since.
Before migrating I tested different variations for some fsfs.conf
parameters, to measure performance a bit. See numbers below (may not be
directly comparable to the old numbers of previous tests above -- history
has grown since then of course). These are not scientific tests, and I only
did a limited number of iterations due to lack of time, so YMMV.
TL;DR: We chose default settings for FSFS format 7 (packed), except for
dir/props-deltification, which we disabled. As it turned out,
dir/props-deltification in our case had a large performance cost for export
and blame (with huge history). Other knobs we tested
(max-linear-deltification and compression-level) didn't have that much of
an impact.
Repository variations:
* f3: FSFS format 3 (svn 1.5 compatible): 15 GB
* f7: FSFS format 7 packed (default settings): 6.8 GB
* f7c1: f7 with compression-level=1: 6.9 GB
* f7c0: f7 with compression-level=0: 10 GB
* sd: f7 with max-linear-deltification=1: 8.7 GB
* sdndp: f7 with max-linear-deltification=1 and
enable-dir/props-deltification=false (forgot to note disk space)
* ndp: f7 with enable-dir/props-deltification=false: 12 GB
* ndpc1: f7 with enable-dir/props-deltification=false and
compression-level=1: 12 GB
Client: SVN 1.9.3 / Windows 7 64bit - Intel i7-3740QM 2.7 GHz 16 GB RAM
Server: SVN 1.9.3 / Solaris 10 - t5 sparc x64 (32 x sparcv9 3600 Mhz) 64 GB
RAM
Server config: HTTPS / SVNInMemoryCacheSize 128 MB / SVNPathAuthz off
Back-end filesystem: NetApp filer, mounted via NFS
Network between client and server: 100 Mbit LAN
Times in seconds
null-export large-subtree (2,984 dirs; 22,602 files = 188 MB; 157,695 props
= 4 MB)
======================
run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1|
---------------------------------------------------------------------
1st (~cold) | 442 | 112 | 247 | 258 | 256 | 319 | 125 | 132 |
2nd | 73 | 114 | 103 | 102 | 83 | 85 | 75 | 72 |
3rd | 73 | 124 | 95 | 101 | 107 | 75 | 73 | 75 |
4th | 72 | 112 | 93 | 98 | 150 | 74 | 71 | 61 |
null-log -v large-subtree (72,731 revs; 110,870 msg lines; 365,983 changes)
======================
run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1|
---------------------------------------------------------------------
1st (~cold) | 3165 | 362 | 688 | 618 | 756 | 888 | 439 | 454 |
2nd | 1454 | 367 | 378 | 376 | 396 | 351 | 330 | 331 |
3rd | 1551 | 366 | 374 | 502 | 395 | 350 | 334 | 335 |
4th | 1398 | 364 | 371 | 380 | 392 | 351 | 357 | 335 |
null-blame large-xmlfile-huge-history (15,183 revs; 15,179 deltas; 38266 MB
in deltas)
======================
run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1|
---------------------------------------------------------------------
1st (~cold) | 2135 | 678 | 775 | 785 | 2114 | 2019 | 536 | 535 |
2nd | 1850 | 800 | 718 | 684 | 1698 | 1599 | 667 | 502 |
3rd | 1927 | 1048 | 796 | 693 | 1733 | 1597 | 516 | 605 |
4th | 1854 | 704 | 700 | 687 | 1696 | 1665 | 515 | 492 |
null-list -R large-subtree (2,985 dirs; 22,602 files; 0 locks (?))
=======================
run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1|
---------------------------------------------------------------------
1st (~cold) | 24 | 13 | 13 | 13 | 13 | 12 | 12 | 11 |
2nd | 16 | 12 | 13 | 13 | 13 | 12 | 7 | 7 |
3rd | 15 | 12 | 7 | 13 | 13 | 12 | 12 | 11 |
4th | 15 | 12 | 7 | 7 | 13 | 11 | 11 | 11 |
null-log -v file-avghistory (449 revs; 977 msg lines; 14,064 changes)
========================
run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1|
---------------------------------------------------------------------
1st (~cold) | 35 | 9 | 19 | 21 | 30 | 28 | 9 | 9 |
2nd | 20 | 12 | 8 | 8 | 6 | 5 | 6 | 5 |
3rd | 2 | 15 | 12 | 8 | 6 | 5 | 1 | 5 |
4th | 19 | 8 | 8 | 8 | 1 | 5 | 6 | 5 |
null-blame average-file (602 revs; 602 deltas; 26 MB in deltas)
===================
run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1|
---------------------------------------------------------------------
1st (~cold) | 8.3 | 10.6 | 14.1 | 16.9 | 24.5 | 25.3 | 9.7 | 8.6 |
2nd | 6.1 | 9.4 | 9.2 | 19.1 | 8.7 | 7.4 | 6.6 | 6.4 |
3rd | 6.2 | 10.0 | 10.1 | 2.6 | 1.6 | 7.1 | 6.8 | 6.3 |
4th | 6.0 | 2.1 | - | - | - | - | 1.3 | 6.5 |
null-export small-tree (48 dirs; 394 files = 2 MB; 2,650 props = 70 KB)
==========================
run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1|
---------------------------------------------------------------------
1st (~cold) | 9.2 | 2.3 | 5.1 | 10.2 | 6.4 | 5.5 | 2.9 | 2.9 |
2nd | 1.9 | 2.4 | 2.3 | 3.8 | 2.2 | 2.3 | 2.1 | 2.0 |
3rd | 1.9 | 2.2 | 2.0 | 5.8 | 1.7 | 1.9 | 1.5 | 1.7 |
4th | 1.6 | 1.5 | 1.6 | 3.3 | 1.5 | 1.8 | 1.7 | 1.4 |
null-list branches/ (337 dirs; 0 files; 0 locks)
=======================
run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1|
---------------------------------------------------------------------
1st (~cold) | 0.51 | 0.48 | 0.66 | 0.98 | 0.64 | 0.53 | 0.46 | 0.48 |
2nd | 0.46 | 0.53 | 0.52 | 0.58 | 0.48 | 0.45 | 0.49 | 0.46 |
3rd | 0.47 | 0.44 | 0.43 | 0.56 | 0.50 | 0.46 | 0.50 | 0.54 |
4th | 0.39 | 0.41 | 0.42 | 0.54 | 0.47 | 0.47 | 0.48 | 0.55 |
null-log -v file-small-history (30 revs; 44 msg lines; 939 changes)
========================
run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1|
---------------------------------------------------------------------
1st (~cold) | 2.7 | 0.92 | 2.54 | 3.92 | 3.03 | 2.98 | 1.08 | 1.08 |
2nd | 1.3 | 0.84 | 0.91 | 1.86 | 0.81 | 0.74 | 0.76 | 0.76 |
3rd | 1.3 | 0.88 | 0.91 | 1.75 | 0.80 | 0.76 | 0.79 | 0.84 |
4th | 1.3 | 0.83 | 0.90 | 1.11 | 0.79 | 0.71 | 0.80 | 0.76 |
null-export trunk (43,650 dirs; 204,837 files = 5.7 GB; 1,412,000 props =
36 MB)
=================
run \ repos | f3 | f7 | f7c1 | f7c0 | sd | sdndp| ndp | ndpc1|
---------------------------------------------------------------------
1st (~cold) | 2249 | 979 | 1078 | 1213 | 1087 | 1205 | 929 | 896 |
2nd | 1048 | 969 | 962 | 954 | 901 | 961 | 974 | 844 |
3rd | 974 | 1084 | 1000 | 974 | 884 | 845 | 893 | 967 |
4th | 956 | 987 | 961 | 1059 | 882 | 840 | 839 | 841 |
--
Johan
Received on 2016-10-10 15:22:01 CEST