On 4 Nov 2002, Kirby C. Bohling wrote:
> On Mon, 2002-11-04 at 16:06, Karl Fogel wrote:
> > Anyone who's feeling squirrely, please take a look at
> >
> > http://subversion.tigris.org/issues/show_bug.cgi?id=860
> >
> > Toward the end, you'll see Ben Collins-Sussman's summary of the
> > current situation, about how the bug's severity has been reduced, yet
> > there is still some mysterious linearity going on.
> >
> > I'm going to be looking at this too, but this is one of those
> > situations where many eyes might help. Especially those with
> > experience tracing Berkeley DB memory usage and/or APR pools.
> >
>
> Karl,
>
> At the end of the issue, it talks about it might be apache or Berkeley
> DB. However, at the top it mentions that this happens with ra_dav or
> ra_local. So I'm failing to see how it could be apache if it does it
> over ra_local.
>
> I'm kinda curious if this is a WAD (Works as Designed) issue with
> Berkeley DB. You're writting a lot of information to a lot of different
> log files that you're not committing. I don't know enough about
> Berkeley's infrastructure to comment, but has anybody asked them about
> the overhead of having an active transaction that crosses 100-1000 log
> files (log files are still 1M right?), and committing it has to write a
> compressed version of the file out to the logs right so your talking
> between 100-1000 log files (depends on the data and the compression).
> So to take up the 28MB difference between 100M and 900M of data. 900M
> of random data compresses into ~900 1M log files. So is right about 32K
> of overhead per open log file in a transaction assuming no compression.
> 32K sounds like a reasonable number (pretty high given BDB's embedded
> usage, but maybe), so it might be there.
>
> Maybe modify the log file size on the creation to be 10M and see if the
> 100M starts acting like 10M that's my bet.
>
> Maybe Daniel Berlin knows, I'm pretty sure he's the one who set
> straight some things for me about BDB on the list in the past.
Hmmm.
I took at look the berkeley source for log stuff (Now i remember why i
couldn't remember it. It makes me shiver. I must have blocked it out. Not
that it's not well designed, just hard to read.)
It looks like putting into the log, including new log file creation if
necessary, frees all the memory it allocates.
IE it only keeps a live structure for the *current* log in memory at a
given time, regardless of whether the transaction spans logs or not.
It also doesn't mmap log files (it only seems to call os_mapfile, which
is what maps files if it can, when opening the memory pool), so the
memory isn't coming from there either.
--Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Tue Nov 5 07:31:46 2002