[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: svnsync crashes on a huge commit

From: Ben Reser <ben_at_reser.org>
Date: Sun, 18 Aug 2013 22:09:50 -0700

On 8/15/13 8:15 PM, Ben Reser wrote:
> Not sure if this is really complete. I don't think the directory case
> or the property case really should be using the dst_rev_pool. In fact
> I really don't understand why the replay_ctx->file_pool exists as well
> as the info->pool. I think we should probably rip out the
> replay_ctx->file_pool, produce appropriate child pools on the info
> struct and then use that. But I haven't done that work to see if it
> works properly.

I finished this up and have nominated the fix for 1.8.x.

I tried continuing to use the shared file_pool and generating new pools
for each file. I didn't find any significant difference between the two
methods in memory usage or speed. The separate pool code I committed
may be slightly faster but the difference is so small it may just be
random differences. I went ahead with this code since it reduced some
code complexity in having to choose which pool to use since all code
paths just always use info->pool.

The previous patch would continue to scale memory based on total
directories opened and the number of properties changed. So it only
fixed part of the problem.

I compared the results of the various strategies using the r18-20 of the
chromium repo using the GNU time command's -v option (not the bash builtin).

        Maximum resident set size (kbytes): 1828144
r1515249 on 1.8.x-svnsync-serf-memory branch:
        Maximum resident set size (kbytes): 210080
r1515269 on 1.8.x-svnsync-serf-memory branch:
        Maximum resident set size (kbytes): 196544

I realize these are different than the previous numbers everyone was
looking at. I know when I looked before I just used the RES column of
top and I got tired of staring at top. In comparison to that I went
from ~500MB for the 1.8.1 code syncing r18 of the chromium repo, ~50MB
for the r1515249 code and ~36MB for the r1515269 code.

Please review this change so we can get it in 1.8.2 when I roll the
tarballs tomorrow.

Reviewers may want to consider the details of the delta editor:
Received on 2013-08-19 07:10:39 CEST

This is an archived mail posted to the Subversion Dev mailing list.