Re: Streamy FS writes found detrimental.

On Tue, Feb 26, 2002 at 08:10:14PM -0500, Greg Hudson wrote:
>...
> Here is what the situation boils down to, given my understanding of the
> facts: we have to limit the size of the values we store in the database
> at each key, or we will never have good performance for large files. We
> should give up on streamy FS writes for the moment until we can do that.

Yup, quite probably.

> Although Berkeley DB supports arbitrarily large values, it can only
> write them efficiently if you hold the value in memory all at once. We
> don't want to do that for large file plaintexts, or even for large
> deltas. The only alternative is to append block by block; creating an
> n-byte value that way takes O(n^2) time (and space) in the current
> Berkeley DB implementation.

Sure, I'll buy that.

> As an aside, this whole DB log file issue leaves me with an extremely
> bad taste in my mouth about using Berkeley DB. It's totally
> unacceptable for the default, unassisted mode of operation for a lookup
> database to be this bad space-wise.

I believe we can fix that. Theoretically, the logs should be the same size
as the data shoved into the FS. IOW, put in a 30 meg file, it consumes 60
meg of space until you prune logs. Personally, I'm fine with 2x.

And if we (by default) prune the logs at each commit, then we're quite fine.

I'm not really worried about the log stuff.

> > It might be interesting to look into using duplicate keys for the 'strings'
> > table.
>
> I'm not sure what duplicate keys are.

I gave a URL to the BDB docs that talk about them. Here we go again:

    http://www.sleepycat.com/docs/ref/am_conf/dup.html

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org