> What I'm trying to do here is to get a _problem statement_ to which your
> patch is a solution, so we can consider other solutions.
Partially repeating the first letter from this thread, the problem is the
following: if a commit occurs during the rep-cache.db verification, this can
lead to a post-commit error "database is locked" and new representations will
not be added in the rep-cache.db. So a verification of a hot repository leads
to user visible errors. Furthermore, deduplication does not work for new data
committed at this time. And commits work with delays. So running verify for a
hot repository is problematic. But then recommending not to verify hot
repositories is also a problem, because in reality repositories can always be
hot and so it is not clear when to verify them at all.
Speaking about correctness, I think the implementation in the patch is correct
and as far as I understand there are no concerns about this. Regarding the
problem with verification depending on the guarantees provided by SQLite: If we
cannot rely on SQLite guarantees, then I think we should not rely on the
guarantee that the table does not change when it is read using one statement
(which the current implementation relies on). In other words we should either
rely on all SQLite guarantees or not rely on any.
All alternative approaches where an administrator takes additional steps do not
solve the problem in my opinion. Because it turns out that it’s better not to
verify the hot repository at all, because additional steps will be required.
Speaking about other approaches, such as sharding rep-cache.db: It may be a
good approach, but I think there may and probably will be other associated
problems that are currently unknown.
I think the new proposed approach is an improvement of the current one, which
also solves the problem. But if there are still significant concerns about that
approach, I do not have anything else to offer at the moment. I may try to
return to this topic if I have any additional thoughts.
Regards,
Denis Kovalchuk
Received on 2020-05-09 22:37:33 CEST