> From: "Kirby C. Bohling" <kbohling@birddog.com>
> Subject: Re: What is handling it streamily?
> To: Greg Stein <gstein@lyra.org>
> Date: Mon, 04 Feb 2002 16:24:14 -0600
>
> Greg,
>
> Well, from what I can tell, the pool that is a directory baton is being
> over used. I am pretty sure it represents 60-80% of the memory. I know
> I saw the directory baton pool up over 40MB, but I believe that was
> early in the check out process. I had a patch that showed it quite
> clearly, but I did an svn update and now I can't get the code to build
> again :-(. I have a diff around to do it pretty easily again. I have
> tried to follow the directory baton around, and well it makes my blood
> come out my ears... editors, deltas, batons, Oh my!... There are a lot
> of functions and function pointers that make it difficult to follow, but
> pluggable as hell so it is all good. I think I have a working copy from
> a fresh checkout.
>
> It shows all the pools and the sizes. I used tags and file_line from
> the pools to determine exactly which pools and where they were created
> that the amount of memory is being allocated in. I know that a ton of
> memory is consumed creating the entries file over and over again in the
> same directory. I haven't looked at the rest of it. The stuff for
> files gets pretty large also. The number of allocations over 2K is
> small. It would be easy to add statistics for that too.
I hope you simply used ./configure --enable-pool-debug=verbose ?
> I have code that will allow you to take a pool and print a message every
> time you allocate from this specific pool, set a breakpoint there, and
> you can see the call tree of every memory allocation to the pools that
> are a pain.
Ah, show me the code ;) [I like the specific pool part]
> You will lose your mind reading all of the allocations
> ever. It seems you nickel and dime them to death, and keep a lot of
> memory around a ton longer then needed. The lifetimes are all wrong is
> my guess, but I can't tell that because there is too much code I don't
> know about. I know that Mozilla has a tool named "SpaceTracer" I
> believe that tracks memory footprint and lifetime from logs to help you
> find objects that are living too long. I spent a ton of time with
> OpenStep chasing stuff like this down in a former life, it is a lot more
> fun when it isn't your job :-)
I also have the feeling the allocator(s) shouldn't hold on to all the
memory. I've done some testing with different approaches of freeing
some mem and it seems like freeing all mem above a certain threshold
is the way to go. I have one more approach to try though, which aims
at better mem reuse, less waste.
> If you want pathes and/or reports on this stuff let me know, I will send
> it. Some of it requires patching to apr, and some of it is written in
> such a way that you have to have SVN_POOL_DEBUG on, but it is all
> fixable to be relatively sane. Some scripting/gdb/libiberty, and I can
> probably nail down where every last allocation comes from on a
> particuarly pool if that is useful.
The pools code still needs more debugging code to make it good enough
to find where something is going wrong in an application. --enable-pool-debug
in combination --with-efence is not going to cut it in all cases.
> I don't know that the apr people
> will like the apr changes or accept them, which makes a lot of the stuff
> less useful. It's why I haven't submitted it to the list yet, I thought
> I might find a way around the changes to apr. It helps a lot in finding
> the pools.
Don't worry about APR, SVN boundaries. What belongs in APR goes there.
There are enough APR committers here to make changes there. And if it's
about pools I'm here to review your code and give feedback (or drop it
into APR).
> Kirby
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Sat Oct 21 14:37:04 2006