[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

RE: Memory consumption w/ large directories

From: Sander Striker <striker_at_apache.org>
Date: 2002-02-08 11:24:08 CET

> From: Kirby C. Bohling [mailto:kbohling@birddog.com]
> Sent: 08 February 2002 03:25

> Greg,
> I'm not sure how portable all of it is. It needs some specific patches
> that I only know to work on x86 Linux with a relatively recent glibc,
> and have hints it might work on other glibc platforms, and uses stuff
> from binutils but I will happily contribute it.
> To be honest it is 2 patches (1 to apr and 1 to subversion), and a 3 or 4
> scripts (2 python and a 1-2 shell). Let me know what it is I need to
> do. Some of it needs a bit of refinement as it is only in my
> ..bash_history file :-). I also need to write a doc or README on how I
> have been memory hunting too.

That would be usefull ;)
> Some of the apr stuff Sander liked and I believe considering
> re-implementing into APR (walking the pool tree), and some stuff that

Yes, that is biting me at the moment. I have breakage in testthread.

> allowed me tag specific pools I am "interested" in tracing (I didn't get
> a feel for Sander's opinion after he read the code, he seemed interested
> before seeing the implementation *grin*).

I liked the concept and I liked the hack. It is a very usefull tool.
I just need a bit of time to see how much we can integrate into APR.
I want to try to keep it as portable as possible, but this tool is so
usefull, we might want to drop it in if it even only works on one

I am currently spending my time on a bit of cleanup in the pools code.
Refactoring out stuff in the debug section, etc. I'll get to your code
next... (/me remembers he has to setup an ssl server...)

> Minor rant to mainly to Sander:
> <rant>
> I don't like having to use grep, as I find piping the output of console
> based gdb very difficult to pipe into grep. Somethings I would grep are
> based soley on memory addresses of pools, which could in theory change
> from run to run, and I hate tracking them down. That is the reason for
> the apr_pool_trace_alloc() stuff. IMHO it should be extented to have
> debugging flags that you can set using calls that are macro'ed to no-ops
> in production code so you can say things like this:
> I am interested in allocs in this and all set the I'm interested bit on
> all subpool created from this pool.
> I interested in seeing a backtrace on allocations from this pool.

Instead of outputting, you save the last X allocations so you can spit those
out on demand, right?

> I interested in a when a new pool gets created from this pool.
> For bonus points I am interested in pools whose tag match this regex.
> I'll provide the implementation if your interested.

Yes :) We need to get on the same wavelength to reduce wasted effort
(in the form of code dup or otherwise).

> Digging through the
> output of this in verbose mode was impossible. I knew the pool I was
> interested in and that was the only one I wanted. The total memory

That's not always the case, that you know which pool you want. And,
to be frank, the verbose output is there also to help a 'replay'

> allocated and deallocated in subversion is _*HUGE*_. It goes thru
> memory allocations at an incredible rate (thanks to the speedy pools).
> Trust me the aggregate per file allocations completely dwarf per
> directory allocations, the only reason I noticed it was because the
> lifetime of the objects was wrong.
> </rant>

This is usefull and constructive feedback, I wouldn't even call it a rant. ;)
Any help implementing a better debug mode is appreciated. I wish to
keep the full verbose mode, but I can see that we need a runtime way to
change the output (no output, only a group of pools, etc).

> Wow, I feel better now. Sorry if that came off too strong I think the
> APR stuff is pretty cool. I am trying to figure out how to either mimic
> or use the pooling code at work. It is like some old Objective C code I
> used once in a former life that was easy, memory management was a no
> brainer back then.

:) In the end, everybode likes pools.

> There is some get a backtrace functionality just crammed right where it
> shouldn't be in apr, then a couple of other little tweaks in
> subversion/libsvn_*/*.c to allow me to track the pools I am interested in.

Are the tweaks in there to switch tracking on or something? [since the debug
_implementation_ belongs in APR]

Thanks for your feedback. Drop in on #svn (irc.openprojects.net) someday so
we can talk a bit more interactively.
> Kirby


To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Sat Oct 21 14:37:05 2006

This is an archived mail posted to the Subversion Dev mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.