Greg Stein wrote:
>>From: "Kirby C. Bohling" <kbohling@birddog.com>
>>Date: Mon, 04 Feb 2002 16:24:14 -0600
>>...
>> Well, from what I can tell, the pool that is a directory baton is being
>>over used. I am pretty sure it represents 60-80% of the memory. I know
>>I saw the directory baton pool up over 40MB, but I believe that was
>>early in the check out process.
>>
>
> Ouch.
>
> When you say "check out process", you're referring to one of the directory
> batons' pools that are creating during "svn checkout ..." ? (as opposed to
> "svn update" or something)
>
I have created an issue, #622 I believe, search for kbohling as the
issue submitter and that is the one. It has a script attched to it that
will create the problem for you. It is easily extendible to make the
problem as bad as you like. :-) Triple the number of iterations on each
loop and you'll have any Intel box pleading for mercy.
The answer to your question is the command line was this:
svn co -d coStuff file:///u1/repository/myStuff
Import isn't pretty, but it is at least not as punishing as checkout.
Attached to this email is a patch which labels the pools and prints out
the size of each pool after each file is added. Use this and the script
attached to Issue #622, and you'll find the problem pools. It needs
that patch I posted earlier to APR to work correctly. You probably have
to do add --enable-pool-debug to the configure script. I haven't had
time to straight it all out. The some of the labels are extremely
useful. Some of them suck. Hopefully somebody else can fix some of
them up.
>
>>...
>>It shows all the pools and the sizes. I used tags and file_line from
>>the pools to determine exactly which pools and where they were created
>>that the amount of memory is being allocated in. I know that a ton of
>>memory is consumed creating the entries file over and over again in the
>>same directory.
>>
>
> Thus, your patch for using a subpool in svn_wc__entry_modify(). Gotcha. But
> in an earlier mail, you said it was only a few megabytes of the 180 meg that
> you observed.
>
I didn't fix all of the problem :-) Following object lifetime is hard.
> As I mentioned in private email, entries.c is due for a big cleaning. We can
> do the subpool thing at that point, but it's kind of small fish right now.
>
Unless that will address the directory_baton, I don't believe it will
solve the particular issue my script shows. I don't know enough about
entries/deltas/batons/editors to sort it all out. The most interesting
problem is that the contents of the file is *not* the problem. Not from
what I can see, or at least it isn't the biggest problem. I can use up
arbitrary amounts of memory importing and checking out empty files if I
am allowed to put enough files in a single directory.
>
>>I haven't looked at the rest of it. The stuff for
>>files gets pretty large also. The number of allocations over 2K is
>>small. It would be easy to add statistics for that too.
>>
>
> I'd be curious what the largest, single allocation is.
>
A pool for a directory baton with 580 files in it, took 112MB of the
180MB of VM.
>
>>...
>>ever. It seems you nickel and dime them to death, and keep a lot of
>>memory around a ton longer then needed.
>>
>
> This is key. I had assumed that there were a few, incorrect uses of a few
> pools. Instead, it sounds like there is a pretty systemic problem.
>
>
<snip>
>
> What will be most useful is the scripting that you use for this. If we just
> get the results, then we'll fix that one spot. But if we have the tools,
> then we can do even more :-)
>
> [ "give a man a fish, and he won't go hungry for a day. teach him to fish
> and he won't be hungry for the rest of his life." (well, until the lake
> is fished out, at least) ]
>
Uhh, it might be a seriously ugly hack, and require odd
libraries/toolsets, but I will give you what it takes if that is what
you really want... *grin*
>
>>I don't know that the apr people
>>will like the apr changes or accept them, which makes a lot of the stuff
>>less useful. It's why I haven't submitted it to the list yet, I thought
>>I might find a way around the changes to apr. It helps a lot in finding
>>the pools.
>>
>
> As Sander said, we have about a half-dozen committers to APR here. That has
> its downsides every now and then (when changes come to both sides at the
> same time), but on balance, it is a tremendous win.
>
I have submitted it to the list. Both of my patches are not up to par
with what I wanted to submitted, but I figured there are people who can
clean them up or at least use them to diagnose the problems and flame
(politely?) me for quality too. I'll be happy to fix them up, I just
haven't.
Thanks,
Kirby
> Cheers,
> -g
>
>
Index: ./subversion/include/svn_pools.h
===================================================================
--- ./subversion/include/svn_pools.h
+++ ./subversion/include/svn_pools.h Mon Feb 4 19:35:05 2002
@@ -135,6 +135,8 @@
*/
#define svn_pool_destroy apr_pool_destroy
+void svn_pool_display_global_tree( void );
+void svn_pool_display_tree( apr_pool_t *p );
#ifdef __cplusplus
}
Index: ./subversion/libsvn_fs/trail.c
===================================================================
--- ./subversion/libsvn_fs/trail.c
+++ ./subversion/libsvn_fs/trail.c Mon Feb 4 23:16:10 2002
@@ -54,6 +54,8 @@
SVN_ERR (DB_WRAP (fs, "beginning Berkeley DB transaction",
fs->env->txn_begin (fs->env, 0, &trail->db_txn, 0)));
+ apr_pool_tag( trail->pool, strdup( svn_fs_berkeley_path( fs, trail->pool ) ) );
+
*trail_p = trail;
return SVN_NO_ERROR;
}
Index: ./subversion/libsvn_fs/tree.c
===================================================================
--- ./subversion/libsvn_fs/tree.c
+++ ./subversion/libsvn_fs/tree.c Mon Feb 4 23:16:20 2002
@@ -105,6 +105,10 @@
apr_pool_t *subpool = svn_pool_create (pool);
svn_fs_root_t *root = apr_pcalloc (subpool, sizeof (*root));
+ /* XXX: I am a memory leak, please fix me by allocating from
+ * a more global pool. */
+ apr_pool_tag( subpool, strdup( svn_fs_berkeley_path( fs, subpool ) ) );
+
root->fs = fs;
root->pool = pool;
Index: ./subversion/libsvn_wc/get_editor.c
===================================================================
--- ./subversion/libsvn_wc/get_editor.c
+++ ./subversion/libsvn_wc/get_editor.c Mon Feb 4 22:55:29 2002
@@ -144,6 +144,7 @@
struct dir_baton *d = apr_pcalloc (subpool, sizeof (*d));
svn_stringbuf_t *path;
+
if (parent_baton)
{
/* I, the baton-in-creation, have a parent, so base my path on
@@ -156,6 +157,11 @@
path = svn_stringbuf_dup (eb->anchor, subpool);
}
+
+ /* XXX: I am a memory leak. I should get my memory
+ * from the error or possibly the parent pool. */
+ apr_pool_tag( subpool, strdup( path->data ) );
+
if (name)
{
d->name = svn_stringbuf_dup (name, subpool);
Index: ./subversion/libsvn_subr/svn_error.c
===================================================================
--- ./subversion/libsvn_subr/svn_error.c
+++ ./subversion/libsvn_subr/svn_error.c Tue Feb 5 10:46:18 2002
@@ -454,6 +454,31 @@
}
+#ifdef APR_POOL_DEBUG
+
+void svn_pool_display_global_tree( void )
+{
+ apr_pool_walk_global_tree( apr_pool_display_info );
+}
+
+void svn_pool_display_tree( apr_pool_t *p )
+{
+ apr_pool_walk_tree( p, apr_pool_display_info );
+}
+#else
+/* I like breakpoints, the overhead of a function call
+ * for easier debugging is a no brainer for alpha code...
+ * XXX: Turn these into macros when it is all said and done.
+ **/
+void svn_global_pool_display_tree()
+{
+}
+
+void svn_pool_display_tree( apr_pool_t *p )
+{
+}
+#endif
+
/* Wrappers that ensure binary compatibility */
#if !APR_POOL_DEBUG
apr_pool_t *
Index: ./subversion/libsvn_ra_local/checkout.c
===================================================================
--- ./subversion/libsvn_ra_local/checkout.c
+++ ./subversion/libsvn_ra_local/checkout.c Mon Feb 4 19:34:50 2002
@@ -219,6 +219,7 @@
SVN_ERR (send_file_contents (root, dirent_path, file_baton,
editor, iter_pool));
SVN_ERR (editor->close_file (file_baton));
+ svn_pool_display_global_tree();
}
else
Index: ./subversion/clients/cmdline/trace-update.c
===================================================================
--- ./subversion/clients/cmdline/trace-update.c
+++ ./subversion/clients/cmdline/trace-update.c Mon Feb 4 23:24:58 2002
@@ -356,6 +356,9 @@
eb->pool = svn_pool_create (pool);
eb->initial_path = svn_stringbuf_dup (initial_path, eb->pool);
+ /* XXX: I am a memory leak. FIX ME. */
+ apr_pool_tag( eb->pool, strdup( initial_path->data ) );
+
/* Set up the editor. */
trace_editor->open_root = open_root;
trace_editor->delete_entry = delete_entry;
Index: ./subversion/clients/cmdline/main.c
===================================================================
--- ./subversion/clients/cmdline/main.c
+++ ./subversion/clients/cmdline/main.c Mon Feb 4 23:27:01 2002
@@ -763,6 +763,10 @@
apr_initialize ();
pool = svn_pool_create (NULL);
+
+ /* Look at that, no leaking on the tag... :-)*/
+ apr_pool_tag( pool, "main_pool" );
+
memset (&opt_state, 0, sizeof (opt_state));
opt_state.start_revision.kind = svn_client_revision_unspecified;
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Sat Oct 21 14:37:04 2006