Okay,
After a bit of tool/patch building, I have a set of tools that let me
track down the worst offenders of memory consumption in the subversion.
I don't know enough about object lifetimes to fix it to be honest.
Bytes: 2378195 Pool: 0x81587d0 Stack Trace: 0x8124849 0x8100653
0x81006b7 0x8101bf8 0x8101cc9 0x805f874 0x8103ddd 0x81031b6 0x8102d94
0x8102d12 0x81019e9 0x805ff05 0x80601dd 0x8061e5b 0x8061fd3 0x8093db6
0x807f314 0x807f4f9 0x807db76 0x8050e5e 0x804b4e7 0x804dfc7 0x400d3306
0x804b1a1
0x8124849 --> subversion/apr/memory/unix/apr_pools.c:1149
0x8100653 --> subversion/subversion/libsvn_subr/svn_string.c:245
0x81006b7 --> subversion/subversion/libsvn_subr/svn_string.c:261
0x8101bf8 --> subversion/subversion/libsvn_subr/xml.c:379
0x8101cc9 --> subversion/subversion/libsvn_subr/xml.c:405
0x805f874 --> subversion/subversion/libsvn_wc/entries.c:342
0x8103ddd --> subversion/expat-lite/xmlparse.c:1299
0x81031b6 --> subversion/expat-lite/xmlparse.c:955
0x8102d94 --> subversion/expat-lite/xmlparse.c:819
0x8102d12 --> subversion/expat-lite/xmlparse.c:809
0x81019e9 --> subversion/subversion/libsvn_subr/xml.c:273
0x805ff05 --> subversion/subversion/libsvn_wc/entries.c:606
0x80601dd --> subversion/subversion/libsvn_wc/entries.c:814
0x8061e5b --> subversion/subversion/libsvn_wc/get_editor.c:900
0x8061fd3 --> subversion/subversion/libsvn_wc/get_editor.c:976
0x8093db6 --> subversion/subversion/libsvn_delta/compose_editors.c:327
0x807f314 --> subversion/subversion/libsvn_ra_local/checkout.c:215
0x807f4f9 --> subversion/subversion/libsvn_ra_local/checkout.c:272
0x807db76 --> subversion/subversion/libsvn_ra_local/ra_plugin.c:350
0x8050e5e --> subversion/subversion/libsvn_client/checkout.c:110
0x804b4e7 --> subversion/subversion/clients/cmdline/checkout-cmd.c:117
0x804dfc7 --> subversion/subversion/clients/cmdline/main.c:1018
0x400d3306 --> ??:0
0x804b1a1 --> ??:0
This back trace allocated 2.3MB during a check in of 208 files in a
single directory. It is the pool that gets big fast, and I am pretty
sure it is all related to a little detail about 5 calls deep. This is
just the worst backtrace. My little set of scripts, will give you a
breakdown per pool/backtrace combination the amount of memory allocated.
I have some stuff lying about that will do per backtrace allocations.
First the number on the left is the address of the function in the
binary, not useful to anybody but me. The listing on the right is
obviously a file name and a line number.
Starting at the bottom, 0 indexed.
To save everybody some time, key levels where the directory gets copied
or modified, not just passed in a function call are:
lib_ra_local/checkout.c:215
livsvn_wc/get_editor.c:900
libsvn_wc/entries.c:606 --> (look at the svn_xml_make_parser a little
above that too follow that call).
(Gets lost in here, it got stuffed accum, which is attached to the
((entries_accumuatlor*)(XML_parser->user_data))->pool ).
libsvn_wc/entries.c:342 --> pretty much straight down now...
libsvn_subr/svn_string.c -> Blantantly obvious at the strdup call.
Okay nothing interesting happens until you get to level 7 in file
lib_ra_local/checkout.c:215. Right there, you have a dir_baton and you
want to add a file. Because the directory pool has a lifetime that
iterates over the entire directory (obviously), that is a lot of memory.
Which means that the directory pool is used a *lot* while adding
several hundred files. And the entry file gets bigger every time a file
is added, which explains the non-linear behavior. I believe this to be
the root cause of all of the memory issues I have been seeing. I know
there are several pieces slated for rewrites/reworks. This is an issue
that should be considered in all of that.
By the way, this is the worst of 14 stack traces which all use over
0.5MB of memory on the files, 7 of those use over 1MB and 2 of those use
over 2MB, which for a total means something like 14-15MB of memory
just to do the check in, and there are 186 unique backtraces that
allocate memory from a dir_baton created in libsvn_wc.
So, now I have a couple of scripts and various other things lying about
as patches. I will happily give them out, what is the best way to do
that? I don't have any file space on a public server, and I figure
dumping a tar.gz file as an attachment into everybody's inbox won't
endear me to too many people. I can attach it to issue 622, and I
should probably post all this information there to. Hmm, that is what I
will do unless somebody tells me that is a bad idea.
Thanks,
Kirby
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Sat Oct 21 14:37:05 2006