Re: pristine store design

Two separate things are getting mixed up here.

1) The fact that storing (only) md5 in the DB introduces the need for
locking where it would otherwise be lockless.

2) Some misty idea of us to split pristines into multiple chunks of a max
size each in order to be able to verify checksums of subsections. *This one
should not be of concern now.*

So let's concentrate on 1).

Surely it would be silly to introduce a skel when we are going to have lots
of volatile data associated with pristines, anyway. Also, storing in the DB
is not a speed concern. The point is to KISS WRT concurrency.

Greg, you said it's not a concern, could you quantify that? I'm not aware
how the locking works concerning db and file system. Can we, like, open the
DB transaction, then mv the temp pristine file into place, then store the DB
row and close the transaction, and thereby guarantee that the set (mv,
write-db-row) is atomic, and does this also hold on windows? Or are we
talking workqueue or something?

Going back to the start, why do we need to store the md5 sum, exactly? I
assume for sending it back to the server without having to calculate it
again. Right? And do we really need that in the first place? If we anyway
read the whole pristine, and if we send the md5 sum in the *end* anyway,
let's get compat by rather having an md5-tee in the sending stream. This is
simply to save ourselves the md5 storing code path.

(Discussed size, compression columns elsewhere, as well as the yet
non-existing mtime, and assuming that the refcount column will go away for
the benefit of a nice and proper query.)

In 1.7, we will really just need the md5. If my assumption is right about
re-calculating md5 being cheap, we can do without an md5-passing-on code
path and we can, independently from that, but in effect, have no metadata
with pristines at all and roll out 1.7 without DB table nor skel. We can add
a DB table when *real* metadata shows up. Is this sane?

~Neels

Greg Stein wrote:
> I didn't miss any part.
>
> You're optimizing writes, when you should worry about reads.
>
> The DB is always open, so reading and writing to it is "cheap".
>
> I don't care about a scheme to seek to the end of a 16Gb chunk. You're
> making stuff up again.
>
> On Mar 3, 2010 9:45 AM, "Stefan Sperling" <stsp_at_elego.de> wrote:
>
> On Wed, Mar 03, 2010 at 12:24:29PM -0500, Greg Stein wrote:
>> You're talking about schemes to verify...
> I didn't say that. I'd very much like svn to verify data it
> reads from the pristine store, on the fly, and point out corrupted
> pristines to the user.
>
>
>> You're talking about splitting files for certain filesystems to help with
>> size limitations, yet...
> Those are side-issues.
>
> What Neels and I are trying to get rid of is the need for
> locking when writing to the pristine store.
>
> You missed the part of not storing data in an sqlite DB which will
> never change once written.
> We need to store the MD5 of every pristine somewhere, for instance.
> If we do store this data in a DB, writing to the pristine store requires
> synchronising access to the DB to keep the DB in a consistent state,
> on top writing the pristine itself.
>
> Writing the pristine itself is already lockless, and also writing
> the MD5 while at it means we wouldn't need any locking.
>
>
>> Putting data in the file means you have to *open* it to read the data.
> We're opening and reading pristines anyway.
> Reading pristines is disk i/o we cannot avoid.
>
> The proposed scheme even minimises I/O in case we need only a chunk
> near the end of a file: Seek across a few SHA1 checksums, read a SHA1
> checksum, then open the pristine with that checksum, instead of seeking
> an entire 16GB pristine until the right block has been found.
>
> Granted, reading an entire huge pristine involves opening a number of
> other prisitines. Not sure which is better.
>
>
>> Again: we are centralizing in order to aggregate data and reduce I/O. Your
>> idea defeats that go...
> Is writing another few bytes to the file slower than writing to the file
> and then opening the DB and modifying the DB, possibly waiting for another
> process to unlock the DB, so we can store the MD5?
>
> Stefan
>