[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

The Performance Elephant is Dead

From: Stefan Fuhrmann <stefanfuhrmann_at_alice-dsl.de>
Date: Sun, 04 Jul 2010 00:28:42 +0200

Go and grab your share at

  
https://tortoiseanalyze.svn.sourceforge.net/svnroot/tortoiseanalyze/OptimizedSVN/branches/prototype

It is SVN 1.7 + APR 1.3 + ZLIB 1.2.5 tuned for svnadmin and
threaded svnserve performance with FSFS. All tests pass on Linux.

-- Stefan^2.

What is this all about?
-----------------------

A couple of weeks ago, I started working on SVN's server performance.
Without access to a development repository, I found it hard to
develop a compelling solution to the underlying problems. Thus,
the baby-step patches I sent to this and other lists got often
rejected.

So, I decided to set up a project in my own repository and to
see how my ideas would turn out in the end. For instance, the
"cache item pinning" feature got dropped and replaced by an
even more effective solution.

Now, I got a working code base that already served many TBytes
without complaint and many code fragments can directly be
copied into the SVN /trunk - after due review, of course.

What has been changed?
----------------------

* APR: improve performance when scanning a file backward
* APR: defer (and often eliminate) seek operations
* FSFS only: file handle cache to keep files open as long as
             feasible and to reuse their buffer content.
* FSFS only: in-process shared fulltext and txdelta caches
             (similar to memcached but without the latency penalty)
* zlib: major deflate() and adler32() speedup, minor inflate() speedup

* simple and fast serialization "framework"
* let stream_readline() fetch data chunks instead of single bytes
* partial object access in svncache_*
* tons of local optimizations (buffer sizes, copy elimination,
  loop unrolling etc.)

* svnserve: wire-compression level is now a command line option
* svnserve, svnadmin: data and file handle cache sizes can now
                      be set from the command line

* back-ported client-relevant changes to 1.6.x to get a baseline
  for c/o performance

What was accomplished?
----------------------

* sustained real-world data throughput > 140MByte/sec per thread
  (wire-compression off)
* > 800 MBytes/sec threaded throughput (wire-compression off)
  @25% server CPU load, client limited
* > 120 MBytes/sec threaded throughput (wire-compression on)
  @50% server CPU load, client limited
* e.g. svn trunk export in 0.34 sec, c/o in 1.3sec
  kde trunk export in 58 sec (8.4GB, ~400.000 files & folders)

* drastic reduction in OS overhead, especially file open & close
* e.g. 15x I/O reduction in svnadmin verify, 200x file reduction

* client runtime dominated by wire-(de-)compression and MD5 checksumming

What is left to do?
-------------------

* code cleanup (e.g. move new files to more appropriate places)
* commenting
* finalize zlib (first version submitted a while ago was well
  received) and APR patches (dito, but no feedback from the APR
  guys whatsoever)

* help people get bits and pieces of the code into SVN /trunk
* test and analyze on Windows

* turn the built-in performance data collection code into
  a "framework"
* use that to tune caches and their usage for various commands
* use the new serialization code in more places

* add a shared mmaped file backed to the membuffer_cache to
  facilitate pre-fork servers
* tune svn_mod_dav + httpd where appropriate

Performance measurements
----------------------

see attachment
(for comparison: http://svn.haxx.se/dev/archive-2010-05/0180.shtml)

Test system
-----------

2x XEON 5550 (8 cores total), 2.66GHz, hyper-threading disabled, turbo boost disabled
24GB RAM
4x128GB cheapo ssd on RAID-0 controller w/ 256 MB cache

LINUX 2.6.28-18-generic SMP
64 bits

repositories on ext4
export to tempfs

Repository mirrors used
-----------------------

criterion tsvn tsvn_packed apache.org kde.org
repo size 5.3G 4.8G 30.4G 58.4G
revisions 18,937 18,937 943,322 1,125,690
export items 3,840 3,840 1,941 399,419
export size 81M 81M 38M 8.4G

Repository verification (technically a dump discarding the output)
------------------------------------------------------------------

1.7.trunk: ~/1.7/svnadmin verify -q $repos
prototype: ~/prototype/svnadmin verify -q -F 512 -M 16000 $repos

* tsvn (hot OS file cache because this repo is quite small)
  1.7.trunk 4m02.246s real, 3m36.450s user, 0m25.086s sys
  prototype 0m56.497s real, 0m54.879s user, 0m01.620s sys

  1.7.trunk 4802975 r/o files, 4858443 seeks, 21449142982 bytes in 5278236 reads
  prototype 21805 r/o files, 272935 seeks, 1406509016 bytes in 353616 reads

* apache (cold OS caches)
  1.7.trunk 473m44.992s real, 426m41.080s user, 34m33.394s sys
  prototype 87m24.859s real, 74m44.800s user, 3m02.275s sys

  1.7.trunk 392465497 r/o files, 377499431 seeks, 1719424827656 bytes in 454170506 reads
  prototype 1695138 r/o files, 28943613 seeks, 136101585284 bytes in 33321548 reads

* kde (cold OS caches)
  1.7.trunk 2129m03.571s real, 2025m11.194s user, 87m44.689s sys
  prototype 480m38.854s real, 454m33.124s user, 6m33.397s sys

  1.7.trunk 960393120 r/o files, 871510017 seeks, 4836969421866 bytes in 1353804552 reads
  prototype 4406426 r/o files, 74136903 seeks, 352105428412 bytes in 85352822 reads

I/O for typical client operations (pre-1.7 repository format)
-------------------------------------------------------------

./svn export -q --ignore-externals file:///mnt/archive/svnroot/tsvn_mirror_packed/trunk /dev/shm/t
1.7.trunk 22365 r/o 13488 r/w files, 16151 seeks, 165077368 bytes in 49250 reads, 139533633 bytes in 17475 writes
prototype 3678 r/o 11433 r/w files, 14654 seeks, 154009260 bytes in 26784 reads, 139533634 bytes in 10906 writes

./svn log -v file:///mnt/archive/svnroot/tsvn_mirror_packed/trunk/www
1.7.trunk 6522 r/o files, 8438 seeks, 29087119 bytes in 7916 reads
prototype 706 r/o files, 5712 seeks, 21052707 bytes in 6492 reads

./svn log -q file:///mnt/archive/svnroot/tsvn_mirror_packed/trunk/www
1.7.trunk 5224 r/o files, 5843 seeks, 21132519 bytes in 5932 reads
prototype 706 r/o files, 860 seeks, 3530970 bytes in 2211 reads

Export using the new server, wire-compression off
-------------------------------------------------
(all run with the same svnserve instance)

~/prototype/svnserve -d -T -c 0 -F 512 -M 12000 $repos_parent
~/prototype/svn export --ignore-externals -q svn://localhost/$TOEXPORT /dev/shm/t

1st run tsvn tsvn_packed apache.org kde.org
real 0m5.534s 0m4.803s 0m3.852s 7m27.693s
user 0m0.636s 0m0.700s 0m0.376s 0m57.980s
sys 0m0.444s 0m0.472s 0m0.192s 0m39.298s

2nd run tsvn tsvn_packed apache.org kde.org
real 0m0.747s 0m0.661s 0m0.408s 1m06.505s
user 0m0.372s 0m0.328s 0m0.224s 0m34.462s
sys 0m0.368s 0m0.324s 0m0.172s 0m23.537s

3rd run tsvn tsvn_packed apache.org kde.org
real 0m0.639s 0m0.642s 0m0.342s 0m58.237s
user 0m0.388s 0m0.368s 0m0.208s 0m33.518s
sys 0m0.248s 0m0.268s 0m0.128s 0m24.358s
MByte/s 126 126 110 148

Export using the new server, wire-compression on
-------------------------------------------------
(all run with the same svnserve instance)

~/prototype/svnserve -d -T -c 1 -F 512 -M 12000 $repos_parent
~/prototype/svn export --ignore-externals -q svn://localhost/$TOEXPORT /dev/shm/t

kde.org 1st run 2nd run 3rd run
real 11m41.616s 4m26.154s 3m56.603s
user 01m42.998s 1m13.365s 1m24.585s
sys 00m36.098s 0m24.602s 0m25.774s
MByte/s 12.2 32.3 36.3
Received on 2010-07-04 00:29:48 CEST

This is an archived mail posted to the Subversion Dev mailing list.