Sander Striker wrote:
>>From: "Kirby C. Bohling" <kbohling@birddog.com>
>>Subject: Re: What is handling it streamily?
>>To: Greg Stein <gstein@lyra.org>
>>Date: Mon, 04 Feb 2002 16:24:14 -0600
>>
>>Greg,
>>
>>It shows all the pools and the sizes. I used tags and file_line from
>>the pools to determine exactly which pools and where they were created
>>that the amount of memory is being allocated in. I know that a ton of
>>memory is consumed creating the entries file over and over again in the
>>same directory. I haven't looked at the rest of it. The stuff for
>>files gets pretty large also. The number of allocations over 2K is
>>small. It would be easy to add statistics for that too.
>>
>
> I hope you simply used ./configure --enable-pool-debug=verbose ?
Goodness no. That is entirely too much information. I couldn't find a
good way to get just the information I wanted using verbose short of
writing a perl/sed script. Made a patch to make it simpiler.
>
>
>>I have code that will allow you to take a pool and print a message every
>>time you allocate from this specific pool, set a breakpoint there, and
>>you can see the call tree of every memory allocation to the pools that
>>are a pain.
>>
>
> Ah, show me the code ;) [I like the specific pool part]
Patch attached. The tricky part is you have to set the a breakpoint at
a specific line, it is would be better to a function that has the
specific purpose of being this breakpoint probably. That is how I
normally do it, but some people think I am nuts. The patch has some
known issues, including I probably have the commenting and style wrong.
It has functionality though. Feel free to correct me on both of these
please, they aren't my strong points. I have to re-read the HACKING file.
<snip>
>
> I also have the feeling the allocator(s) shouldn't hold on to all the
> memory. I've done some testing with different approaches of freeing
> some mem and it seems like freeing all mem above a certain threshold
> is the way to go. I have one more approach to try though, which aims
> at better mem reuse, less waste.
>
I believe there is nothing wrong with the pools per se. Just that the
wrong objects are being put in the wrong pools. That is my guess.
>
>>If you want pathes and/or reports on this stuff let me know, I will send
>>it. Some of it requires patching to apr, and some of it is written in
>>such a way that you have to have SVN_POOL_DEBUG on, but it is all
>>fixable to be relatively sane. Some scripting/gdb/libiberty, and I can
>>probably nail down where every last allocation comes from on a
>>particuarly pool if that is useful.
>>
>
> The pools code still needs more debugging code to make it good enough
> to find where something is going wrong in an application. --enable-pool-debug
> in combination --with-efence is not going to cut it in all cases.
>
>
>>I don't know that the apr people
>>will like the apr changes or accept them, which makes a lot of the stuff
>>less useful. It's why I haven't submitted it to the list yet, I thought
>>I might find a way around the changes to apr. It helps a lot in finding
>>the pools.
>>
>
> Don't worry about APR, SVN boundaries. What belongs in APR goes there.
> There are enough APR committers here to make changes there. And if it's
> about pools I'm here to review your code and give feedback (or drop it
> into APR).
>
>
>> Kirby
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
Index: include/apr_pools.h
===================================================================
RCS file: /home/cvspublic/apr/include/apr_pools.h,v
retrieving revision 1.77
diff -u -r1.77 apr_pools.h
--- include/apr_pools.h 5 Feb 2002 12:09:43 -0000 1.77
+++ include/apr_pools.h 5 Feb 2002 16:57:59 -0000
@@ -132,7 +132,7 @@
#define APR_POOL__STRINGIZE(x) #x
#define APR_POOL__FILE_LINE__ __FILE__ ":" APR_POOL_STRINGIZE(__LINE__)
-
+
/** The fundamental pool type */
typedef struct apr_pool_t apr_pool_t;
@@ -615,6 +615,13 @@
* @return The number of bytes
*/
APR_DECLARE(apr_size_t) apr_pool_num_bytes(apr_pool_t *p, int recurse);
+APR_DECLARE(apr_size_t) apr_global_pool_num_bytes( int recurse );
+
+APR_DECLARE(void) apr_pool_trace_alloc( apr_pool_t *p );
+APR_DECLARE(void) apr_pool_display_info( apr_pool_t *p );
+
+APR_DECLARE(void) apr_pool_walk_global_tree( void (func)( apr_pool_t * ) );
+APR_DECLARE(void) apr_pool_walk_tree( apr_pool_t *p, void (func)( apr_pool_t* ) );
/**
* Lock a pool
Index: memory/unix/apr_pools.c
===================================================================
RCS file: /home/cvspublic/apr/memory/unix/apr_pools.c,v
retrieving revision 1.149
diff -u -r1.149 apr_pools.c
--- memory/unix/apr_pools.c 5 Feb 2002 12:09:43 -0000 1.149
+++ memory/unix/apr_pools.c 5 Feb 2002 16:58:00 -0000
@@ -184,6 +184,8 @@
unsigned int stat_alloc;
unsigned int stat_total_alloc;
unsigned int stat_clear;
+ unsigned int trace_alloc;
+ unsigned long trace_total_alloc;
#if APR_HAS_THREADS
apr_os_thread_t owner;
apr_thread_mutex_t *mutex;
@@ -375,7 +377,7 @@
/* If we found nothing, seek the sink (at index 0), if
* it is not empty.
*/
- else if (allocator->free[0]) {
+ else if (index >= MAX_INDEX && allocator->free[0] != NULL) {
#if APR_HAS_THREADS
if (allocator->mutex)
apr_thread_mutex_lock(allocator->mutex);
@@ -424,7 +426,7 @@
static APR_INLINE void node_free(allocator_t *allocator, node_t *node)
{
- node_t *next;
+ node_t *next, *free_nodes = NULL;
apr_uint32_t index, max_index;
#if APR_HAS_THREADS
@@ -454,8 +456,13 @@
/* This node is too large to keep in a specific size bucket,
* just add it to the sink (at index 0).
*/
+#if 0
node->next = allocator->free[0];
allocator->free[0] = node;
+#endif
+
+ node->next = free_nodes;
+ free_nodes = node;
}
}
while ((node = next) != NULL);
@@ -466,6 +473,11 @@
if (allocator->mutex)
apr_thread_mutex_unlock(allocator->mutex);
#endif
+
+ while ((node = free_nodes) != NULL) {
+ free_nodes = node->next;
+ free(node);
+ }
}
APR_DECLARE(void *) apr_palloc(apr_pool_t *pool, apr_size_t size)
@@ -1099,6 +1111,14 @@
pool->stat_alloc++;
pool->stat_total_alloc++;
+
+#ifdef APR_POOL_DEBUG
+ if( pool -> trace_alloc ) {
+ pool -> trace_total_alloc += size;
+ fprintf( stdout, "Allocation from pool: %p size: %lu total: %lu\n", pool,
+ (unsigned long) size, pool->trace_total_alloc );
+ }
+#endif
return mem;
}
@@ -1495,6 +1515,61 @@
return size;
}
#endif
+
+APR_DECLARE(void) apr_pool_trace_alloc( apr_pool_t *p )
+{
+ p->trace_alloc = 1;
+ p->trace_total_alloc = 0;
+}
+
+APR_DECLARE(void) apr_pool_display_info( apr_pool_t *p )
+{
+ fprintf( stdout, "%p;%p;\"%s\";%s;%lu;%lu\n", p, p->parent, p->tag ,
+#if defined(APR_POOL_DEBUG)
+ p->file_line,
+#else
+ "",
+#endif
+ apr_pool_num_bytes( p , 0 ),
+ apr_pool_num_bytes( p , 1 ) );
+ fflush( stdout );
+ fsync( fileno( stdout ) );
+}
+
+APR_DECLARE(void) apr_pool_walk_global_tree( void (func)( apr_pool_t * ) )
+{
+ apr_pool_walk_tree( global_pool, func );
+}
+
+/* It isn't clear it is a useful ordering now that I have thought about it,
+ * it needs to have a queue to give me the order I want. It also probably
+ * isn't even close to thread safe... */
+APR_DECLARE(void) apr_pool_walk_tree( apr_pool_t *p, void (func)( apr_pool_t* ) )
+{
+ struct apr_pool_t* s= 0;
+
+ if( !p )
+ return;
+
+ // Do them in layers....
+ (*func)( p );
+
+ s = p->sibling;
+ while( s )
+ {
+ (*func)( s );
+ s = s ->sibling;
+ }
+
+ apr_pool_walk_tree( p->child, func );
+
+ s = p->sibling;
+ while( s )
+ {
+ apr_pool_walk_tree( s->child, func );
+ s = s->sibling;
+ }
+}
APR_DECLARE(apr_size_t) apr_pool_num_bytes(apr_pool_t *pool, int recurse)
{
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Sat Oct 21 14:37:04 2006