Re: 1.6.2 poor commit performance with large data

On Tue, Jun 2, 2009 at 1:24 PM, Bolstridge,
Andrew<andy.bolstridge_at_intergraph.com> wrote:
>> -----Original Message-----
>> From: B Smith-Mannschott [mailto:bsmith.occs_at_gmail.com]
>> Sent: Friday, May 29, 2009 11:27 AM
>> To: Bolstridge, Andrew
>> Cc: users_at_subversion.tigris.org
>> Subject: Re: 1.6.2 poor commit performance with large data
>>
> [snip]
>
>> This sounds like the same problem as described in [this thread][1]
>> from a few days ago. Said thread links to [Issue 1964][2] which looks
>> like the same problem.
>>
>> [1]:
>> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessage
>> Id=2352720
>> [2]: http://subversion.tigris.org/issues/show_bug.cgi?id=1964
>>
>> So far, evidence suggests that this issue crops up only under Windows.
>> I haven't seen anything suggesting *why* this is the case, so any
>> information you can provide is likely to be helpful.
>>
>
>
> I've had a look at this running again today. It seems to be throttled in the System process (not any of the sub-processes, so it must be a driver). On my quad-core box System.exe hits 25% CPU and stays there.
> There is a lot of page faults (after transferring 73Mb, I've got 74 million page faults with a delta of roughly 1000 per second).
> Memory usage steadily (but very slowly) increases once the transfer has started to stall, currently showing 169Mb/164Mb in taskmanager.
>
> and then.... I had a brainwave. I turned off the virus checker and suddenly things went back to normal. (I use Trend which isn't quite so bad). The memory usage is still increasing at a rate of 1Mb per second but the page fault delta for System process has dropped to 0.
>
>
> In the end, transferring 9500 mostly cpp source-code files (187Mb) used peak memory of 1.5Gb RAM.
>
> If anyone wants to check the 'leak', I made a heap comparison showing the allocations. I did this using TortoiseSVN (as that has some symbols), unfortunately it doesn't have enough symbols for the svn libraries or it'd be more useful. If anyone has built SVN for Windows, and will create some .pdb symbols, then I'd be happy to re-run my commits and see if I can find out more about this issue.
>
> It appears that the call to libapr_tsvn's apr_allocator_max_free_set is doing the damage, after transferring 130Mb (roughly 7000 files), it has 'leaked' 749Mb. I can’t see what has been calling it without some effort (and symbols) but maybe someone more knowledgeable of SVN source can offer some suggestions. It appears to mostly allocate 8k at a time, but occasionally allocates 16k (if that helps any).
>
> Be aware I've cross-posted this to the TortoiseSVN list too.

The leak has been discovered and fixed and will be in 1.6.3. This
also revealed that the way temp files are created is a cause of
slowness. A fix for that is being looked at and will hopefully be in
1.6.3 as well.

-- 
Thanks
Mark Phippard
http://markphip.blogspot.com/
------------------------------------------------------
http://tortoisesvn.tigris.org/ds/viewMessage.do?dsForumId=4061&dsMessageId=2358841
To unsubscribe from this discussion, e-mail: [users-unsubscribe_at_tortoisesvn.tigris.org].