Hi all,
answering several replies to my OP:
Bert Huijben wrote:
>> -----Original Message-----
>> From: Stefan Fuhrmann [mailto:stefanfuhrmann_at_alice-dsl.de]
>> Sent: woensdag 12 mei 2010 12:25
>> To: dev_at_apr.apache.org
>> Subject: First SVN performance data
>>
>> Hi there,
>>
>> as I promised, I'm going to conduct some in-depth analysis and
>> comprehensive SVN performance testing.
>> That is very time-consuming process.
>>
>
> I think this belongs on dev_at_subversion instead of dev_at_apr; moving to that
> list.
>
>
I'd hit the wrong button :/
>> "Export" has been chosen to eliminate problems with client-side w/c
>> performance.
>>
>
> Note that this also eliminates delta transfers, which are very common in
> these wc operations. Subversion tries to optimize transfers by only
> transferring binary updates, but you eliminated that.
>
From what I have seen, most time is spent (in that order) in
* wire compression (i.e. before sending data via svn://)
* de-compression while getting data from FSFS
* MD5
Due to the skip-delta storage, I would assume constructing
delta (r1, r2) could be more expensive than delta (r, 0).
But I haven't checked that, yet.
> I would assume that transferring full files as deltas is not the most
> efficient way to transfer these full files. No surprises there. (And
> transferring deltas will hopefully be faster too after this work)
>
With wire compression disabled, the transfer takes about
25% of the wall clock time.
> In the performance measurements I did just before releasing 1.6 (on
> checkout, update, merge) I found that most time was spend on client-io in
> the working copy and on locking, but all of this was removed from your
> testset as you are only testing the subversion fs (filesystem) and IO
> performance.
>
> While I'm certain that these components can use more than a bit of
> optimization to enhance the scalability of Subversion, I'm not sure if these
> are the true performance problems of subversion as noted in our user
> feedback.
>
There is a APR patch that reduces the I/O overhead,
especially under Windows. But there has been no feedback:
https://issues.apache.org/bugzilla/show_bug.cgi?id=49085
However, even without that patch the client-side file I/O
takes only 50% of the available wall clock time (see
user+sys vs. real). So, writing the data twice plus some
DB update (written once!) would still be feasible within
the wall clock time-frame set by the server performance.
> If you are looking at just 'svn export' you completely eliminate the
> reporting phase from an update editor (the client->server reporting), which
> is one of the major components of svn update and svn status --update. You
> just leave the server->client data transfer, the in-memory property handling
> and writing out the possibly translated files on the client.
>
Client-side reporting has never been a real problem
for me because TSVN crawls the folder tree from
time to time keeping the cache hot. Provided wc-ng
will not be a major regression, the actual data download
remains the bottleneck.
> Michael Pilato and Hyrum Wright interviewed some enterprise users earlier
> this year and wrote some reports which indicated that the network latency
> and working copy performance were the true bottlenecks. If I look at
> ^/subversion/trunk/notes/feedback, I see checkout, log, merging as primary
> performance issues and this matches the performance issues I see in my day
> to day use of repositories on the other side of the world.
Same here. That is why my focus is on c/o performance.
With the w/c currently undergoing major refactoring,
the server-side is the only "tangible" link in the chain.
For TSVN, I already solved the log performance issue
for most practical purposes ;)
Many of the performance improvements are also relevant
for client code: APR (file I/O), APR-Util (MD5), zlib,
svn libs file access overhead.
> Theoretically svn
> checkout should be pretty similar to the svn export your testing, but
> Subversion completely implements this as updating from r0 to the specified
> revision. Most of the time on these operations is spend on the client
> waiting for disk (libsvn_wc administration, moving and rewriting files) and
> on network-IO caused by latency (not by throughput).
>
O.k. I will have a look at the update editor performance.
My strategy is as follows:
* eliminate "friction" within the server code, use high-volume
requests as a benchmark
-> everything else should automatically be faster than this "worst case"
+ improved SNR for further analysis
* gather data and idea for fs-ng along the way
* find a I/O & network setup that copes the backend performance
-> clear advice to users on how to maximize performance,
if that is crucial to them
* extend optimization efforts to w/c
* optimize I/O patterns to bring other setups closer to the "best case"
> (On the Slik Windows buildbot, which runs all tests on a NTFS ramdrive, I
> see that the tests are CPU bound these days... But that is after eliminating
> all disk IO during the tests. When I run them on a normal workstation disk I
> see different results)
>
Hopefully, SSDs and ample memory will remedy the
I/O performance disparity.
Bolstridge, Andrew <andy.bolstridge_at_intergraph.com <mailto:andy.bolstridge_at_intergraph.com?Subject=RE:%20First%20SVN%20performance%20data>> wrote
> ///> From: Stefan Fuhrmann [mailto:stefanfuhrmann_at_alice-dsl.de] /
> ///> /
> ///> * SVN servers tend to be CPU-limited /
> /> (we already observed that problem @ our company /
> /> with SVN 1.4) /
>
> Lovely figures, but I'm guessing CPU will be more of a bottleneck when
> you run on a server with 24Gb RAM and 4 SSDs in RAID-0 configuration.
> The server will allocate so much to cache that everything is
> super-quick, and what disk IO there is, will also be super quick.
>
If you look at the bottom end of the attachment, you will see
the actual amount of RAM used during the tests. It says that
the amount of memory your server should have is roughly
the size of your working copy.
Also, a SSD raid is *slower* than an SSD directly hooked
to a simple controller chip. I built my workstation a while
ago and had to compensate for the crappy flash controllers
used at that time -> proper raid controller w/ memory.
> Do you have benchmarks on a more 'representative' server? Say, a dual
> core with
> 2Gb RAM and 4 HDD disks in RAID5 running as a VM image, and SVN running
> over http?
>
I could use my old workstation but it doesn't have a RAID5.
At our company, we run SVN 1.4 servers in a VM (1G phys.
RAM) accessing a SAN. Since many developers work on
the same projects, the VM has been proven CPU-bound.
But maybe the following numbers would interest you. I hooked
a cheap USB harddisk to my notebook and compared it to
its internal SSD drive. Tests were executed on WinXP 32
using the TSVN repository:
flat sharded packed
usb 1st run 2:11.641 2:39.468 1:32.343
usb 2nd run 0:03.921 0:03.578 0:03.375
ssd 1st run 0:07.031 0:09.656 0:05.859
ssd 2nd run 0:03.750 0:03.406 0:03.296
normalized by runtime for "packed"
usb 1st run 1.42 1.73 1.00
usb 2nd run 1.16 1.06 1.00
ssd 1st run 1.20 1.65 1.00
ssd 2nd run 1.14 1.03 1.00
normalized by fasted setup
usb 1st run 39.94 48.38 28.02
usb 2nd run 1.19 1.09 1.02
ssd 1st run 2.13 2.93 1.78
ssd 2nd run 1.14 1.03 1.00
From these numbers, you can see:
* use either packed 1.6 format or flat 1.4 format
* your storage location must be cachable on the server
Johan Corveleyn wrote:
>> > * SVN servers tend to be CPU-limited
> If you're using an NFS connected SAN, it's a whole different ballgame.
> Ok, that may not be the fastest way to run an SVN server, but I think
> it's often set up that way in a lot of companies, for various
> practical reasons. I'd like SVN to be fast in this setup as well (as
> are many other server applications).
>
At some point in future, you will have to decide upon new
server hardware. It is important to understand that SVN
is a database server and as such has specific needs. You
probably spend large amounts of money on your backup
solution, including safe / vault, service contracts etc.
If that is the case, the following rationals could guide your
decisions:
* a backup solution that handles "terror bytes" of data
within the defined backup / restore window is expensive.
If keeping your data safe is worth a certain amount of
money, you should be allowed to spend at least the same
amount of money for making the data usable.
* only a few repositories are actually "hot". Document
and binary repositories usually are not and / or work
reasonably well from with I/O (due to item size).
* RAM is ~32G/1kEUR to 256G/10kEUR. The latter
is enough for >10 KDE-sized projects, i.e. several
1000 developers.
* Flash is ~512G/1kEUR. Most companies won't need
more than 1T for their hot repositories. Use it as a
write-through cache and copy new revisions to the SAN.
IOW, spend 5 grand on a pizza box and make hundreds
of users happy. Those users also happen to be very expensive
and critical to your company's success.
>
>> > * packed repositories are ~20% faster than non-packed,
>> > non-sharded
>>
>
> Also, I don't see those ~20% in your test numbers (more like ~5%), but
> maybe you have other numbers that show this difference?
>
It's 2.5 ./. 2.0 secs for "export hot data from 1.7".
In the numbers above it is somewhere between
15 and 40%. The key factor is probably the file
system being used (credentials checking etc).
>
>> > * optimal file cache size is roughly /trunk size
>> > (plus branch diffs, but that is yet to be quantified)
>> > * "cold" I/O from a low-latency source takes 2 .. 3 times
>> > as long as from cached data
>>
>
> Ok, but unless you can get almost the entire repository in cache,
> that's not very useful IMHO. In my tests, I mainly focused on the
> "first run", because I want my server to be fast with a cold cache.
> Because that's most likely the performance that my users will get.
> It's a busy repository, with different users hitting different parts
> of the repository all the time. I just don't think there will be a lot
> of cache hits during a normal working day.
>
You don't need the entire repository to be in cache
- just as my numbers show. Also, in teams with more
than one person working on the same development
line, data written once but read many times.
Think about continuous integration, for instance. I
began investigation that issue when load peaks caused
checkout times to grow from 8 minutes to over an hour.
Adding a second CPU almost eliminated the effect.
> Also, if the test with cached data is 2-3 times faster than from the
> SSD RAID-0, that's another indication to me that there's a lot of time
> spent in I/O. And where there's a lot of time spent, there is
> potentially a lot of time to be saved by optimizing.
>
>
I certainly will take a closer look at I/O. However, it
seems to me that at least the amount of data being
transferred is close to optimal (judging from the amount
of data being processed by zlib).
From the numbers above, you can see that a not-top-notch
SSD directly hooked to the controller is much faster than
the supposedly fast RAID0 configuration. It is all about
latency - not so much OS overhead like "open file".
>> > * a fully patched 1.7 server is twice as fast as 1.6.9
>> >
>> > "Export" has been chosen to eliminate problems
>> > with client-side w/c performance.
>>
>
> I mainly focused on log and blame (and checkout/update to a lesser
> degree), so that may be one of the reasons why we're seeing it
> differently :-) . I suppose the numbers, bottlenecks, ... totally
> depend on the use case (as well as the hardware/network setup).
>
The log performance issue has been solved more or less
in TSVN. In 1.7, we also brought the UI up to speed
with the internals: even complex full-text searches over
millions of changes are (almost) interactive.
>
>> > If I look at
>> > ^/subversion/trunk/notes/feedback, I see checkout, log, merging as primary
>> > performance issues and this matches the performance issues I see in my day
>> > to day use of repositories on the other side of the world.
>>
>
> Ok, so you agree log is one of the important performance issues. That
> one is very much I/O bound on the server (as I described before,
> opening and closing rev files multiple times).
>
To speed up log on the server side, you need to maintain
an index. That's certainly not going to happen before fs-ng.
Otherwise, you will always end up reading every revision file.
Only exception: log on the repo root with no changed path
listing.
-- Stefan^2.
Received on 2010-05-16 12:29:55 CEST