Re: Performance benchmarks

On Sun, Mar 27, 2011 at 4:51 AM, Mark Phippard <markphip_at_gmail.com> wrote:

> On Fri, Mar 25, 2011 at 1:33 PM, Mark Phippard <markphip_at_gmail.com> wrote:
>
> > I have been working on a framework for writing tests to record
> > performance. I have something good enough to share:
> >
> > https://ctf.open.collab.net/sf/projects/csvn
> >
> > It is pretty easy to add new tests if you have ideas on more tests you
> > think we should add. I think I have pretty good coverage of the major
> > functions. The wiki on the site I linked to above has details on how
> > I have constructed the current tests. I am going to put out a call to
> > users for feedback and try to get more people to run the tests and
> > record results.
> >
> > I am not claiming these are anything definitive or even that we will
> > use them to help us make the release decision, but I think it is a
> > start on coming up with some reproducible tests that people can run
> > easily. If after people look at and run the tests they think they are
> > useful or can be tweaked to be useful, then great. If not, then at
> > least I got to write some code for a change :)
> >
> > The tests are written in Java because that is what I know and it gives
> > me good cross platform coverage. However, the Java just drives the
> > command line so all you need to do is have the svn command line in
> > your PATH and that is what it uses for all the work.
>
> These tests are showing some interesting results.
>
> * There are a number of cases where 1.7 shows large performance gains
> over 1.6. For example, a WC with a lot of folders on Windows:
>
>
> https://ctf.open.collab.net/sf/wiki/do/viewPage/projects.csvn/wiki/FolderTests
>
> * Checkout seems to be the biggest remaining hotspot where 1.7 tends
> to be slower, although commit seems to still be slower too.
>
> * Delete and move are slower than I would have expected. Is this
> because we can actually delete the folder now, as opposed to waiting
> until commit time? I actually thought that made would make it faster
> as I thought the folder used to get deleted and then put back (maybe
> just in IDE's).
>
> * There is something seriously bad going on with Anti-Virus on my
> Windows laptop (Symantec Endpoint Protection). I noticed the times
> for commit were way out of whack so I configured my A/V to ignore the
> folder where the benchmarks were running. When I ran the tests again,
> the performance of 1.7 was waaay better. I am about 95% certain the
> A/V is the problem but I need to make sure. While it was great to see
> the good performance, it is still a big concern to see that A/V might
> be having an even bigger impact than it has before. 1.6 also showed
> performance improvements but they were less dramatic than some of the
> 1.7 commands.
>
> Stefan Küng asked me to add svnversion to the benchmark and I have
> done so in the latest release.
>
> I would love to see someone do some tests with the WC on local disk vs
> network mount (1.6 and 1.7). I tried to do it using some virtual
> machines I have access to at CollabNet. The problem is that the
> connection of these boxes to the NetApp with our home folders is too
> slow. Some of the checkouts (even using 1.6) were running for an hour
> and I finally killed the test.
>
> Anyway, overall I am encouraged by the results I am seeing so far. I
> look forward to more people running the tests and sharing the results.
>
>
Attached are my results from the benchmark suite. I've provided two versions
of the 1.7.0 run. One was an older build (prior to the optimizations), and
the last one was a recent build.

These were all run over file:///. I'll try and see if I can re-run it over
svn://.

Cheers,
Daniel B.