Re: memory use of svn log -v with long changed paths lists

On Mon, Jun 23, 2014 at 11:55 PM, Stefan Sperling <stsp_at_elego.de> wrote:

> I'm still seeing 180MB of memory used by svn log -v with ruby's r11708
> over ra_local with fsfs7-unpacked (120MB with fsfs6-unpacked).
> The total output generated by svn log is only 5.5MB in size.
> That seems somewhat disproportionate.
>

Yes, it is. The id_t struct is somewhat heavy but in total
a change should be less than 200 bytes. So, everything
that needs more than ~30MB for the ruby example is
suspicious.

> Of course, I'm happy that this invocation of svn log isn't running
> out of memory anymore on my machine, as it did before r1604188!
> So that's a huge step forward but I suppose we can still do better.
>
> It seems caching is partly responsible for the memory growth.
> I'm seeing huge jumps during svn_cache__set() calls, though we might
> still be missing some scratch pools or iterpools to keep memory
> usage low.
>

The serializer itself uses a single, auto-growing buffer.
It should show up as a temporary peak, though.

The current 100 bytes estimation / change is way to low
with the basic struct already taking 144 bytes on a 64bit
system. r1604947 should help here.

> Does anyone know what can be done about this?
>

Use a similar estimate as in r1604947 to skip svn_cache__set
for uncachable items (call svn_cache__is_cachable).

As soon as I arrived in UK, I'll look into this. It's probably
at the point now where I need to counting bytes at every
step to spot inappropriate memory usage.

-- Stefan^2.
Received on 2014-06-24 00:51:22 CEST