[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: #4667, Merge uses large amount of memory

From: Stefan Fuhrmann <stefan2_at_apache.org>
Date: Sun, 8 Jan 2017 18:06:01 +0100

On 06.01.2017 13:26, Julian Foad wrote:
> I wrote:
>> "svn-mergeinfo-normalizer" fails to execute in the available RAM.
>
> But looking closer, it says "Memory fault" which may indicate some other kind of
> failure rather than not enough memory.
>
> Stefan Fuhrmann wrote:
>> If the tool manages to read the mergeinfo, it will print m/i
>> stats before fetching the log. Does it get to this stage?
>
> One of the examples the customer provided for the memory fault (path names
> obscured):
>
> [[[
> Running the normalizer on one line that is bad works some times, but other times
> it doesn't. With the -v option, here is one of the bad paths (filelist2.txt is
> just: ./H___/S___/T___):
>
> ___:/nfs/2___/l___/Data-Source> svn-mergeinfo-normalizer normalize
> --remove-obsoletes --remove-redundant --remove-redundant-misaligned
> --combine-ranges --targets ~/filelist2.txt --depth empty -v
> Scanning working copy /nfs/2___/l___/Data-Source/H___/S___/T___ ...
> Found mergeinfo on 1 nodes.
> Found 76 branch entries.
> Found 77 merged revision ranges.

As estimated elsethread, this should use ~300MB
for the full working copy.

> Fetching log for https://c___/t___/d___
> .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
......................................................................................................................................................................................
>
> Received 1200983 revisions from 1 to 1200984.
> Received 5720201 path changes.
> Pool has 3020575 different paths.

That adds 400..500 MB, still less than 1GB in total.
So, at least with /trunk, everything should fit into
memory.

> Removing obsolete branches and redundant mergeinfo, combining revision ranges ...
> Trying to elide mergeinfo at path
> /nfs/2___/l___/Data-Source/H___/S___/T___
>
>
> Trying to remove obsolete entries ...
> keep POTENTIAL branch /branches/___/218068-4/Data/H___/S___/T___
> keep POTENTIAL branch /branches/___/235073/Data/H___/S___/T___
> keep POTENTIAL branch /branches/___/241502-5/Data/H___/S___/T___
> keep POTENTIAL branch /branches/___/241502/Data/H___/S___/T___
> keep POTENTIAL branch /branches/___/242103-3/Data/H___/S___/T___
> keep POTENTIAL branch /branches/___/242794-2/Data/H___/S___/T___
> keep POTENTIAL branch /branches/___/242794/Data/H___/S___/T___
> Memory fault

Looks like a segfault in disguise. Could you run
that in a debugger and see where it crashes (stack)?

I'll try to build with pool debugging and run that
through valgrind to see if I can reproduce it.

-- Stefan^2.
Received on 2017-01-08 18:06:06 CET

This is an archived mail posted to the Subversion Dev mailing list.