On 01.02.2016 11:11, Stefan Sperling wrote:
> On Mon, Feb 01, 2016 at 10:06:19AM +0000, Philip Martin wrote:
>> Stefan Fuhrmann <stefan2_at_apache.org> writes:
>>
>>> So, all user content is there and merely the deduplication failed
>>> (as already being investigated elsewhere in this thread).
>>
>> I suppose format 7 might allow us to implement a system that fixes
>> missing deduplication during packing.
At least we can scan the file for representation info
in nodesrevs and update the rep-cache.db accordingly.
> And perhaps get rid of sqlite in the repository while at it?
Format 7 assumes that there is a 1:1 relationship between
logical ID and physical locations. So, you can't simply
make two entries in the L2P index point to the same phys.
item without breaking the P2L index.
Since we can't rewrite the references in all future reps
that point to any redundant one, we need to stick with the
same number of logical and physical items. A format 8,
however, could allow for "duplicate" P2L entries where
N-1 items get flagged as "shared". That would be a low-
risk bookkeeping change.
That said, there are limitations to that approach: Cache
contents is logically addressed, i.e. even if the IDs would
point to the same location, they would be cached twice. So,
we would simply save some disk space. Depending on how many
active branches there are, the lacking cache efficiency may
not be an issue.
Another problem with "pack" replacing the rep-cache.db is
that deduplication often happens as a result of merges
and those often cross 1k shard boundaries.
One option would be to e.g. defer deduplication to the
pack phase and use the rep-cache.db exclusively during
that operation.
-- Stefan^2.
Received on 2016-02-01 20:43:47 CET