On Mon, Feb 15, 2010 at 08:45, Neels J Hofmeyr <neels_at_elego.de> wrote:
> Hi all,
>
> taking stock of the current state of the pristine store API and finding
> design docs missing, I have created a "design paper" to clarify.
>
> If you could be so kind to glance over it and straighten out my picture, if
> necessary. Upon approval, I'll check it into notes/ so we can edit.
>
> Note, if my view is correct, this design text implies small changes to the
> current state of the API:
> - no need for _pristine_write()
> - need to add _pristine_forget()
> - change the _pristine_checkmode_t enum.
>
> Still missing completely: how to handle a "high-water mark", i.e. how to
> determine which pristines get forgotten first.
As noted elsewhere, you can definitively figure out which pristine
files are required by scanning the "checksum" columns in the tables to
figure out which are referenced. The notion of a "high-water mark" is
not part of the current design because svn *expects* the pristines to
be present. We do not have a capability for demand-load pristines from
a foreign repository. Thus, you need "all" of the pristines, not just
"32 gig" of pristines.
>...
> SOME IMPLEMENTATION INSIGHTS
> ============================
>
> There is a PRISTINE table in the SQLite database with columns
> (checksum, md5_checksum, size, refcount)
>
> The pristine contents are stored in the local filesystem in a pristine file,
> which may or may not be compressed (opaquely hidden behind the pristines API).
> The goal is to be able to have a pristine store per working copy,
For 1.7.
> per user as
> well as system-wide, and to configure each working copy as to which pristine
> store(s) it should use for reading/writing.
For future versions.
> There is a canonical way of getting a given CHECKSUM's pristine file name for
> a given working copy without contacting the WC database (static function
> get_pristine_fname()).
>
> When interacting with the pristine store, we want to, as appropriate, check
> for (combos of):
> db-presence - presence in the PRISTINE table with noted file size > 0
> file-presence - pristine file presence
> stat-match - PRISTINE table's size and mtime match file system
> checksum-match - validity of data in the file against the checksum
There is a fifth mode, potentially not useful, which is SIZE==0. That
means "I know about the pristine, but do not have its contents (yet)."
(altho, it may be that the file *is* there, but a failure occurred
between its write-completion, and the update of the row in PRISTINE).
> file-presence is gotten for free from a successful stat-match (fstat),
> checksum-match (fopen) and unchecked read of the file (fopen).
>
> How fast we consider things:
> db-presence - very fast to moderately fast (in case of "empty db cache")
> file-presence - slow (fstat or fopen)
> stat-match - slow (fstat plus SQLite query)
> checksum-match - super slow (reading, checksumming)
>
>
> On the method of writing pristine files:
> In short: *ALWAYS* copy to pristine temp and checksum along with that, then
> 'mv' into place.
The current design allows for writing directly into the store. If the
checksum is known beforehand, then you don't need to compute it. The
design is also intended to ignore any (partial) file when the row has
SIZE==0. Only when the file has been completely written, will SIZE be
updated, thus signifying a useful pristine.
> To avoid incomplete pristines, we never want to have a write stream to a
> pristine file location. Instead, we write to a tempfile on the same file-
> system device as the pristine store, after which a filesystem 'mv' puts it
> into place and a row is stored in the database. Like that, a pristine file
> is either intact or doesn't exist (unless corrupted by alien ray guns).
Not necessary.
> When fetching a new pristine from the repository, get a temporary-file
> location from the pristine API, receive&checksum to that file, install.
> Typically, we have already checked if the pristine is stored yet.
When a file comes from the repository, we "should" already know its checksum.
Maybe the current code does not, but as we rebuild this stuff... it
WILL. When fetching stuff from the repository, we *always* want to get
the checksum first. Then we can determine whether to fetch it (not in
the pristine store), or to skip the network fetch (found it in the
store).
Thus, we will always have a checksum when we begin to write a new
pristine into the store. And we already have protection against
partial writes.
> When there is some new content (added/modified file in the working copy), a
> pristine with the same content may already exist "coincidentally" -- that
> may be common in certain use scenarios, but is generally less common.
Agreed. The common case is "not present", so the design can optimize
around that.
Note that (branch) checkouts in a shared-pristine environment may want
to optimize for "present". Not something to worry about today, though.
> Discussing optimal ways in different situations:
> - New file is *not* a temporary file ('svn add'/'commit', we are not allowed
> to 'mv' the file away), and a pristine with the same checksum and content
> as this file does *not* exist yet: The file needs to be checksummed and
> copied to the pristine store.
> --> write to pristine-tempfile, checksum along the way, install.
> - New file is *not* a temporary file ('svn add'/'commit'), but a pristine
> with the same content already exists. To catch this case, we would need to
> checksum the file *before* copying to the tempfile and then don't bother to
> copy if a match is found. BUT the file could be changed in-between us
> checksumming in-place, not finding a match and then copying! We should
> copy to a tempfile first anyway.
> - New file *is* a tempfile (we may 'mv' it away) and happens to be on the
> same filesystem device as the pristine store. We could checksum in-place
> and call pristine_install() on the tempfile directly. *But* code, in this
> case, should already have chosen the pristine-store tempfile location to
> begin with.
> So, it does not make sense to optimize here.
> (Depending on hardware, copying across devices can be faster or slower
> than copying within the same device. Can't optimise on that.)
Note that, in many cases, we already have checksums for most of these
files. Or it would be very easy to get the values. Many of these files
are constructed through stream copies, potentially with
(de)translation, and we can perform a checksum, too.
> On overwriting orphaned pristine files that have no database entry:
> "Maybe the user saved his favourite aunt's final poetic words in a file that
> has a name that is this checksum and which ended up in the pristine store
> accidentally ..."
> This is super highly unlikely, given that the file name would have to match
> the checksum. We don't really need to play safe.
> But, if we find a pristine file where the db didn't expect one, it may
> sometimes be faster to checksum that file and find it intact than to
> overwrite with data possibly received from a repository.
> But again, that implies that something must have gone wrong (interrupted at
> just the right time/bug/db corruption) and we don't need to optimise for
> that.
> So, when storing a pristine, we can simply `mv` over an existing checksum
> file in the pristine store, not bothering to even check if it exists.
Generally true, I'd think.
In "repair mode", if you find a pristine file without a database
record, then you'd probably delete the file. Recall that the record
"should" be present if it is referenced from one of the other tables
(and, thus, needed).
(of course, we generally don't have referential integrity turned on,
so it *may* be possible to delete a row from PRISTINE even when you
need it; thus, keeping the file may actually be necessary)
> USE CASES
> =========
>
> Pseudocode implies args for reference to working copy, pool etc. as needed.
> pristine_* = svn_wc__db_pristine_*
> _usable = svn_wc__db_checkmode_usable, etc.
>
>
> Use case "new": "I have locally-created content which should be stored
> -------------- as a pristine."
> Another pristine with identical content may exist coincidentally, in which
> case we technically don't need to write the pristine at all. But, as discussed
> in "On the method to write pristine files" above, we should anyway copy the
> new content to a tempfile to make sure nothing changes it in-between us
> checksumming and copying. So, this is exactly the same as use case "store"!
In most cases, when we do an "add" (or somesuch), we make a copy of
the file into a temporary area, potentially (de)translating that file.
At that time, we can also checksum it. No need to do further I/O (a
move) if you already have the file.
If we do a raw filesystem copy w/o detranslation, then we may not have
a checksum.
I think it would be easier to break up your use cases with what we
know about the source file, and what needs to be done to it.
Detranslation? Copy into a holding area from original source? Copied
from a version-controlled file, so we already have info about it? etc.
Each of these different types of sources will determine a data flow
into the pristine storage.
> Use case "fetch": "I'm in one of the get_pristine_from_repos() places (9).
> ---------------- I want to get a pristine from the repository and store it."
> This will contact the repos to read a revision content. Even though the repos
> provides a checksum for the content, we need to anyway validate that checksum
> on our end.
> Getting the data from the repos, employ use case "store" below.
> (We don't need to bother to checksum our end before storing the pristine,
> checksumming is included in use case "store".)
Does this use case imply that the pristine is NOT present already?
(because, of course, we we already have it, then we don't need to
fetch it from the repository under any situation).
> Use case "store": "I am going to receive data that should be stored
> ---------------- as a new pristine!"
> We are going to receive data which we will checksum (to validate, or even to
Again, you may or may not need to checksum. Depends on the original
location/status of the intended pristine.
>...
> Use case "need": "I want to use this pristine's content, definitely."
> ---------------
> pseudocode:
> pristine_check(&present, checksum, _usable) (3)
> if !present:
> get_pristine_from_repos(checksum, ra) (9)
> pristine_read(&stream, checksum) (6)
Woah. Fetching from the repository can only be done in libsvn_client.
The WC library cannot do that. Your APIs and use cases need to
consider this, depending on the scope (are you trying to define wc_db
/ pristine APIs? or public WC APIs that assist libsvn_client?).
I also don't see a need for "check" in this case. Just try to fetch
it, and have a unique error code for "I don't have it."
> Use case "would use": "I'd use this pristine if it's here already,
> -------------------- if not I still have other options."
> pseudocode:
> pristine_check(&present, checksum, _usable) (3)
> if !present:
> pursue other options
> else:
> pristine_read(&stream, checksum) (6)
Just read, and catch an error.
> Use case "could use": "I have more complex options, and I want to get a
> -------------------- *fast* count on how many of certain pristines I can
> expect to exist. Does this one show?"
> pseudocode:
> pristine_check(&present, checksum, _known) (4)
>
> (4) ### Depending on which turns out to be faster:
> Only check db-presence
> ### OR
> Only check file-presence
Right. This was the intent around the check() API that I designed.
Then later figured out that we'd always have a wc.db open anyways, so
the fine-tuned granularity of check options was premature.
Something like this *is* handy, though. I forget the real-world use
case for the check, but it is there.
> Use case "routine check": "I want to do some effort to validate this pristine"
> ------------------------
> pseudocode:
> pristine_check(NULL, checksum, _valid) (5)
I'm not sure when this would be used explicitly, rather than as a
side-effect to other operations.
I think that any explicit check would also be an explicit repair.
>...
> Use case "repair": "I've seen this pristine is børken. Restore or remove!"
> -----------------
> pseudocode:
> pristine_repair(&present, checksum) (7)
> [if !present:
> get_pristine_from_repos(checksum, ra) (9)
> pristine_read(&stream, checksum) (6)
> ]
I think we'd want to do this as a normal part of "svn cleanup" (and
maybe "svn upgrade").
>...
> Use case "remove": "I have lowered or hit the watermark or failed to repair.
> ----------------- This pristine has to go."
> (or am moving pristines to another pristine store...)
> pseudocode:
> pristine_forget(checksum) (8)
>
> (8) Without bailing on non-existence, remove anything that exists about
> this checksum: PRISTINE table row, pristine file.
Removal during repair is *part* of repair, and not part of the API.
Other removal should be automatic, and part of the wc_db APIs. For
example, when a BASE_NODE is deleted, we should consider removing its
pristine, too.
>...
> CENTRAL PRISTINE STORES
> =======================
> ### TODO: How to configure (per WC and user), how to get write access for
> multiple users while safeguarding against malicious corruption, whether to
> have read-only pristine stores with a per-user writable store, whether to have
> a chain of pristine stores that get asked for presence of a pristine, how to
> record which working copies use which pristine store to get a valid refcount.
IMO, punt to post-1.7.
Cheers,
-g
Received on 2010-02-16 21:10:40 CET