Greg Stein wrote:
>On Thu, Feb 07, 2002 at 07:19:14PM -0600, Kirby C. Bohling wrote:
>
>>Okay,
>>
>> After a bit of tool/patch building, I have a set of tools that let me
>>track down the worst offenders of memory consumption in the subversion.
>> I don't know enough about object lifetimes to fix it to be honest.
>>
>
>That's what we're here for :-)
>
>>...
>>This back trace allocated 2.3MB during a check in of 208 files in a
>>single directory. It is the pool that gets big fast, and I am pretty
>>sure it is all related to a little detail about 5 calls deep. This is
>>just the worst backtrace. My little set of scripts, will give you a
>>
>
>I see the problem.
>
>libsvn_wc/get_editor.c::add_or_open_file(). Each time that bugger is called,
>it calls svn_io_get_dirents() using the directory baton. Then it calls
>svn_wc_entries_read() to read the entries file (also using the dir baton).
>It calls svn_wc_check_wc(), but that probably won't consume much memory.
>
>In any case, those three items are all done in the directory pool. After
>that, it will create the file baton, which has its own subpool.
>
>The logic should be changed to create the subpool *first*, do the operations
>in that subpool, and then pass that to make_file_baton().
>
You know, I wouldn't mind reading the entries into the directory pool,
once, and then *keeping them there*, instead of opening and parsing
.svn/entries (your favourite number of) times for each updated /
committed file.
>
>
>>...
>> Which means that the directory pool is used a *lot* while adding
>>several hundred files. And the entry file gets bigger every time a file
>>is added, which explains the non-linear behavior. I believe this to be
>>the root cause of all of the memory issues I have been seeing. I know
>>
>
>I would think so, yah.
>
>>there are several pieces slated for rewrites/reworks. This is an issue
>>that should be considered in all of that.
>>
>
>This one wasn't on deck :-)
>
>After I get through some weird borkenness, and fix some config problem Karl
>is seeing, then I'll fix that function.
>
>>By the way, this is the worst of 14 stack traces which all use over
>>0.5MB of memory on the files, 7 of those use over 1MB and 2 of those use
>> over 2MB, which for a total means something like 14-15MB of memory
>>just to do the check in, and there are 186 unique backtraces that
>>allocate memory from a dir_baton created in libsvn_wc.
>>
>
>Neat statistics!
>
>>So, now I have a couple of scripts and various other things lying about
>>as patches. I will happily give them out, what is the best way to do
>>that? I don't have any file space on a public server, and I figure
>>dumping a tar.gz file as an attachment into everybody's inbox won't
>>endear me to too many people. I can attach it to issue 622, and I
>>should probably post all this information there to. Hmm, that is what I
>>will do unless somebody tells me that is a bad idea.
>>
>
>How about we create /trunk/tools/dev, and you commit them into there?
>
>[ theoretically, some of this stuff is also helpful for APR, but we can sort
> that out later ]
>
>Cheers,
>-g
>
--
Brane Čibej <brane_at_xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Sat Oct 21 14:37:05 2006