I did an additional benchmark doing "svn rm dir/*" on a local directory instead of an nfs directory. It runs in 10.4s. Is going from 10.4s to 6m15s acceptable when using a working copy on nfs vs local? I am fine with a certain amount of slowdown when using nfs. But, I don't see this kind of degradation on nfs vs local for other operations. Svn rm seems particularly susceptible.
-----Original Message-----
From: Stefan Sperling [mailto:stsp_at_elego.de]
Sent: Monday, October 31, 2011 9:45 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: users_at_subversion.apache.org
Subject: Re: Apparent "svn rm" scaling problem in 1.7.x
On Mon, Oct 31, 2011 at 09:09:21AM -0600, michael_rytting_at_agilent.com wrote:
> I am starting to see some very bad performance with "svn rm" compared to the 1.6.x line of subversion. I have a directory that is full of files. If I go into the directory and run "svn rm *", it is significantly slower than running svn rm on the whole directory. While the difference in time taken is significant, the speed is still relatively acceptable for small working copies.
>
> "svn rm dir/*" 3.8s 1.7.1
> "svn rm dir" 0.173s 1.7.1
> "svn rm dir/*" 1.008s 1.6.17
>
> In this case 1.6.17 is nearly 4 times faster. My working copy is nfs mounted and has 200 nodes.
>
What happens here is that each deletion target uses one sqlite transaction.
AFAIK creating sqlite transactions on NFS is expensive.
In the recursive case, only one transaction is used (for 'dir').
In the non-recursive case, the shell expands the target list, and svn ends up using one transaction per target.
To fix this problem, we could change the implementation so that one sqlite transaction is used even for multiple targets.
> When the working copy size starts to get larger, deleting the same directory as above, but within a larger working copy and things really start to slow down.
>
> "svn rm dir/*" 6m15s 1.7.1
> "svn rm dir" 8.5s 1.7.1
> "svn rm dir/*" 1.14s 1.6.17
>
> This working copy is also nfs mounted but I'm up to 23948 nodes (doing
> sqlite3 .svn/wc.db "select count (*) from nodes"). The directory that
> I am deleting has 49 files. The largest is file is 2.5k. I am doing
> the comparison of the same directory of the same repository. In one
> case I am only doing a partial checkout to keep the working copy size
> down where the second test I am doing a complete checkout.
>
> We can also see that subversion 1.6.17 scales very well for working copy size.
Yes, 1.6 performed better on NFS than 1.7 does.
This is because of how sqlite behaves on NFS filesystems.
But 1.7 performs and scale an awful lot better on local disk than 1.6 did, so this was a tradeoff we were willing to make.
I am not sure how much additional performance can be squeezed out of sqlite on NFS. It has been said that not much can be done about it.
But hints are welcome in case anyone has an idea.
Received on 2011-10-31 17:12:42 CET