On Thu, Feb 07, 2002 at 07:19:14PM -0600, Kirby C. Bohling wrote:
> Okay,
>
> After a bit of tool/patch building, I have a set of tools that let me
> track down the worst offenders of memory consumption in the subversion.
> I don't know enough about object lifetimes to fix it to be honest.
That's what we're here for :-)
>...
> This back trace allocated 2.3MB during a check in of 208 files in a
> single directory. It is the pool that gets big fast, and I am pretty
> sure it is all related to a little detail about 5 calls deep. This is
> just the worst backtrace. My little set of scripts, will give you a
I see the problem.
libsvn_wc/get_editor.c::add_or_open_file(). Each time that bugger is called,
it calls svn_io_get_dirents() using the directory baton. Then it calls
svn_wc_entries_read() to read the entries file (also using the dir baton).
It calls svn_wc_check_wc(), but that probably won't consume much memory.
In any case, those three items are all done in the directory pool. After
that, it will create the file baton, which has its own subpool.
The logic should be changed to create the subpool *first*, do the operations
in that subpool, and then pass that to make_file_baton().
>...
> Which means that the directory pool is used a *lot* while adding
> several hundred files. And the entry file gets bigger every time a file
> is added, which explains the non-linear behavior. I believe this to be
> the root cause of all of the memory issues I have been seeing. I know
I would think so, yah.
> there are several pieces slated for rewrites/reworks. This is an issue
> that should be considered in all of that.
This one wasn't on deck :-)
After I get through some weird borkenness, and fix some config problem Karl
is seeing, then I'll fix that function.
> By the way, this is the worst of 14 stack traces which all use over
> 0.5MB of memory on the files, 7 of those use over 1MB and 2 of those use
> over 2MB, which for a total means something like 14-15MB of memory
> just to do the check in, and there are 186 unique backtraces that
> allocate memory from a dir_baton created in libsvn_wc.
Neat statistics!
> So, now I have a couple of scripts and various other things lying about
> as patches. I will happily give them out, what is the best way to do
> that? I don't have any file space on a public server, and I figure
> dumping a tar.gz file as an attachment into everybody's inbox won't
> endear me to too many people. I can attach it to issue 622, and I
> should probably post all this information there to. Hmm, that is what I
> will do unless somebody tells me that is a bad idea.
How about we create /trunk/tools/dev, and you commit them into there?
[ theoretically, some of this stuff is also helpful for APR, but we can sort
that out later ]
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Sat Oct 21 14:37:05 2006