[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

RE: Memory consumption w/ large directories

From: Sander Striker <striker_at_apache.org>
Date: 2002-02-08 11:24:08 CET

> From: Kirby C. Bohling [mailto:kbohling@birddog.com]
> Sent: 08 February 2002 03:25

> Greg,
>
> I'm not sure how portable all of it is. It needs some specific patches
> that I only know to work on x86 Linux with a relatively recent glibc,
> and have hints it might work on other glibc platforms, and uses stuff
> from binutils but I will happily contribute it.
>
> To be honest it is 2 patches (1 to apr and 1 to subversion), and a 3 or 4
> scripts (2 python and a 1-2 shell). Let me know what it is I need to
> do. Some of it needs a bit of refinement as it is only in my
> ..bash_history file :-). I also need to write a doc or README on how I
> have been memory hunting too.

That would be usefull ;)
 
> Some of the apr stuff Sander liked and I believe considering
> re-implementing into APR (walking the pool tree), and some stuff that

Yes, that is biting me at the moment. I have breakage in testthread.

> allowed me tag specific pools I am "interested" in tracing (I didn't get
> a feel for Sander's opinion after he read the code, he seemed interested
> before seeing the implementation *grin*).

I liked the concept and I liked the hack. It is a very usefull tool.
I just need a bit of time to see how much we can integrate into APR.
I want to try to keep it as portable as possible, but this tool is so
usefull, we might want to drop it in if it even only works on one
platform...

I am currently spending my time on a bit of cleanup in the pools code.
Refactoring out stuff in the debug section, etc. I'll get to your code
next... (/me remembers he has to setup an ssl server...)

> Minor rant to mainly to Sander:
> <rant>
> I don't like having to use grep, as I find piping the output of console
> based gdb very difficult to pipe into grep. Somethings I would grep are
> based soley on memory addresses of pools, which could in theory change
> from run to run, and I hate tracking them down. That is the reason for
> the apr_pool_trace_alloc() stuff. IMHO it should be extented to have
> debugging flags that you can set using calls that are macro'ed to no-ops
> in production code so you can say things like this:
>
> I am interested in allocs in this and all set the I'm interested bit on
> all subpool created from this pool.
>
> I interested in seeing a backtrace on allocations from this pool.

Instead of outputting, you save the last X allocations so you can spit those
out on demand, right?

> I interested in a when a new pool gets created from this pool.
>
> For bonus points I am interested in pools whose tag match this regex.
>
> I'll provide the implementation if your interested.

Yes :) We need to get on the same wavelength to reduce wasted effort
(in the form of code dup or otherwise).

> Digging through the
> output of this in verbose mode was impossible. I knew the pool I was
> interested in and that was the only one I wanted. The total memory

That's not always the case, that you know which pool you want. And,
to be frank, the verbose output is there also to help a 'replay'
implementation.

> allocated and deallocated in subversion is _*HUGE*_. It goes thru
> memory allocations at an incredible rate (thanks to the speedy pools).
> Trust me the aggregate per file allocations completely dwarf per
> directory allocations, the only reason I noticed it was because the
> lifetime of the objects was wrong.
> </rant>

This is usefull and constructive feedback, I wouldn't even call it a rant. ;)
Any help implementing a better debug mode is appreciated. I wish to
keep the full verbose mode, but I can see that we need a runtime way to
change the output (no output, only a group of pools, etc).

> Wow, I feel better now. Sorry if that came off too strong I think the
> APR stuff is pretty cool. I am trying to figure out how to either mimic
> or use the pooling code at work. It is like some old Objective C code I
> used once in a former life that was easy, memory management was a no
> brainer back then.

:) In the end, everybode likes pools.

> There is some get a backtrace functionality just crammed right where it
> shouldn't be in apr, then a couple of other little tweaks in
> subversion/libsvn_*/*.c to allow me to track the pools I am interested in.

Are the tweaks in there to switch tracking on or something? [since the debug
_implementation_ belongs in APR]

Thanks for your feedback. Drop in on #svn (irc.openprojects.net) someday so
we can talk a bit more interactively.
 
> Kirby

Sander

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Sat Oct 21 14:37:05 2006

This is an archived mail posted to the Subversion Dev mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.