[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: memory use of svn log -v with long changed paths lists

From: Stefan Fuhrmann <stefan.fuhrmann_at_wandisco.com>
Date: Tue, 24 Jun 2014 00:50:52 +0200

On Mon, Jun 23, 2014 at 11:55 PM, Stefan Sperling <stsp_at_elego.de> wrote:

> I'm still seeing 180MB of memory used by svn log -v with ruby's r11708
> over ra_local with fsfs7-unpacked (120MB with fsfs6-unpacked).
> The total output generated by svn log is only 5.5MB in size.
> That seems somewhat disproportionate.

Yes, it is. The id_t struct is somewhat heavy but in total
a change should be less than 200 bytes. So, everything
that needs more than ~30MB for the ruby example is

> Of course, I'm happy that this invocation of svn log isn't running
> out of memory anymore on my machine, as it did before r1604188!
> So that's a huge step forward but I suppose we can still do better.
> It seems caching is partly responsible for the memory growth.
> I'm seeing huge jumps during svn_cache__set() calls, though we might
> still be missing some scratch pools or iterpools to keep memory
> usage low.

The serializer itself uses a single, auto-growing buffer.
It should show up as a temporary peak, though.

The current 100 bytes estimation / change is way to low
with the basic struct already taking 144 bytes on a 64bit
system. r1604947 should help here.

> Does anyone know what can be done about this?

Use a similar estimate as in r1604947 to skip svn_cache__set
for uncachable items (call svn_cache__is_cachable).

As soon as I arrived in UK, I'll look into this. It's probably
at the point now where I need to counting bytes at every
step to spot inappropriate memory usage.

-- Stefan^2.
Received on 2014-06-24 00:51:22 CEST

This is an archived mail posted to the Subversion Dev mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.