[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: Memory consumption w/ large directories

From: Kirby C. Bohling <kbohling_at_birddog.com>
Date: 2002-02-08 18:15:10 CET

Sander Striker wrote:
>>From: Kirby C. Bohling [mailto:kbohling@birddog.com]
>>Sent: 08 February 2002 03:25
>>
>

<Snip>

>> Some of the apr stuff Sander liked and I believe considering
>>re-implementing into APR (walking the pool tree), and some stuff that
>>
>
> Yes, that is biting me at the moment. I have breakage in testthread.
>

Okay, if there is anything I can do let me know. If you're using the
ordering I did, I am not sure that can be threaded and not deadlock.
>
>>allowed me tag specific pools I am "interested" in tracing (I didn't get
>>a feel for Sander's opinion after he read the code, he seemed interested
>>before seeing the implementation *grin*).
>>
>
> I liked the concept and I liked the hack. It is a very usefull tool.
> I just need a bit of time to see how much we can integrate into APR.
> I want to try to keep it as portable as possible, but this tool is so
> usefull, we might want to drop it in if it even only works on one
> platform...
>
> I am currently spending my time on a bit of cleanup in the pools code.
> Refactoring out stuff in the debug section, etc. I'll get to your code
> next... (/me remembers he has to setup an ssl server...)
>

No big hurry on any of it, just wasn't sure what the status was on all
of it. To be honest I had to I can implement most of it in under 20
minutes again on a new check out :-)

>
>>Minor rant to mainly to Sander:
>><rant>
>>I don't like having to use grep, as I find piping the output of console
>>based gdb very difficult to pipe into grep. Somethings I would grep are
>>based soley on memory addresses of pools, which could in theory change
>>from run to run, and I hate tracking them down. That is the reason for
>>the apr_pool_trace_alloc() stuff. IMHO it should be extented to have
>>debugging flags that you can set using calls that are macro'ed to no-ops
>>in production code so you can say things like this:
>>
>>I am interested in allocs in this and all set the I'm interested bit on
>>all subpool created from this pool.
>>
>>I interested in seeing a backtrace on allocations from this pool.
>>
>
> Instead of outputting, you save the last X allocations so you can spit those
> out on demand, right?
>

Nope, I "mark" the pool as I am interested and output all of those pools
everytime I am interested. Normally when I am debugging I do know
exactly which pools I am interested in (well so far) from output that
walks the tree and prints it clues me in. Anything that seems excessive
I am interested in, between the tags/file_line and/or a breakpoint it is
easy to "mark" it.

>
>>I interested in a when a new pool gets created from this pool.
>>
>>For bonus points I am interested in pools whose tag match this regex.
>>
>>I'll provide the implementation if your interested.
>>
>
> Yes :) We need to get on the same wavelength to reduce wasted effort
> (in the form of code dup or otherwise).
>

Sounds good to me. Tell you want, I will code a first pass up sometime
this weekend and push it your way for comment that has some set of
features I find interesting. I won't mind if I duplicate your work this
is *fun*.

>
>>Digging through the
>>output of this in verbose mode was impossible. I knew the pool I was
>>interested in and that was the only one I wanted. The total memory
>>
>
> That's not always the case, that you know which pool you want. And,
> to be frank, the verbose output is there also to help a 'replay'
> implementation.
>

I believe the verbose mode is very useful, I just don't believe it is
the only tool. (If all you have is a hammer everything is a nail).

>
>>allocated and deallocated in subversion is _*HUGE*_. It goes thru
>>memory allocations at an incredible rate (thanks to the speedy pools).
>>Trust me the aggregate per file allocations completely dwarf per
>>directory allocations, the only reason I noticed it was because the
>>lifetime of the objects was wrong.
>></rant>
>>
>
> This is usefull and constructive feedback, I wouldn't even call it a rant. ;)
> Any help implementing a better debug mode is appreciated. I wish to
> keep the full verbose mode, but I can see that we need a runtime way to
> change the output (no output, only a group of pools, etc).
>

*nod* see above.

>> There is some get a backtrace functionality just crammed right where it
>>shouldn't be in apr, then a couple of other little tweaks in
>>subversion/libsvn_*/*.c to allow me to track the pools I am interested in.
>>
>
> Are the tweaks in there to switch tracking on or something? [since the debug
> _implementation_ belongs in APR]
>
> Thanks for your feedback. Drop in on #svn (irc.openprojects.net) someday so
> we can talk a bit more interactively.
>

Uhhh, do you have a recommended IRC client, I've never used IRC in my
life (I am so sheltered...).

        Kirby

>
>> Kirby
>>
>
> Sander
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Sat Oct 21 14:37:05 2006

This is an archived mail posted to the Subversion Dev mailing list.