I'd like to discuss this topic a bit further, to try and understand things more.
On Thu, Oct 18, 2012 at 7:04 PM, Julian Foad <julianfoad_at_btopenworld.com> wrote:
> Philip Martin wrote:
>> Johan Corveleyn <jcorvel_at_gmail.com> writes:
>>
>>> With this new "exclusive" mode it's probably quite clear (no
>>> parallellism possible at all), but because of the fuzzyness of the
>>> non-exclusive behavior this leaves me wondering: "which things does
>>> this exclusive mode make impossible, which were still possible under
>>> non-exclusive mode"? What does one lose by enabling exclusive mode?
>>
>> A write operation like "svn up" opens the working copy and then executes
>> a number of SQLite write transactions. SQLite will block other reads
>> while a write transaction is in progress. With shared, non-exclusive,
>> locking read operations can run between the write transactions. There
>> is a timeout (10s for Subversion) and while the whole write operation
>> may take a long time the individual write transactions are generally
>> shorter that the timeout.
>>
>> So from an SQLite perspective most read operations may run during a
>> write operation but may block before proceeding. From a Subversion
>> perspective the read operation will still fail to run if there is a
>> workqueue item. So in 1.7 read operations only work if they run between
>> write transactions when the workqueue is empty.
>>
>> With 1.8 exclusive locking the SQLite write lock is not dropped after
>> the first write transaction but it help until the whole write operation
>> is over and the database is closed. No read operations can run while
>> the SQLite write lock is held.
>>
> Would it be useful to copy this explanation somewhere more discoverable -- somewhere in the source tree, or linked from the release notes page?
>
> - Julian
+1, some clear documentation around this would be quite welcome I
think, at least for developers of svn (GUI) clients, IDE plugins,
wrappers, scripts, ...
But first, some more questions about the "old" non-exclusive mode:
- This 10 second timeout, for a read operation that is blocked by some
write transaction: is this implemented somewhere in the svn source
code (if so, where?), or is this some configuration setting for the
sqlite connection or something like that?
- If a read operation (with a "shared lock" I presume) squeezes in
between write transactions of a large write operation, and that read
takes a long time, I suppose the next write transaction of that large
operation will also block, with the same timeout?
- Does this also work for two writers? Or do they block out each other
with the working copy lock (wc_locks table)?
- With your remark about the workqueue making the read fail, is this
how a read operation may proceed?
* Try to get a shared sqlite lock for reading. Block for max. 10s if
the db is locked for writing (error out if timed out).
* When db reading is allowed (shared lock acquired), first read the
work_queue. If it's not empty: error out.
* Finally, read what we want to read, and get out.
(and the above sequence may be part of a larger read operation that
e.g. does a query per directory, so this sequence may be repeated
several times, perhaps interleaved with some large write operation
that intermittently takes a write lock for things done per directory
or somesuch)
- All things considered, the "old" non-exclusive mode sounds quite
non-deterministic. I mean, isn't this very hard for large GUI clients,
IDE plugins, ... to handle this?
Then Bert said:
> Note that with the current state we could remove the limitation of being
> able to open the database while there are workqueue operations. In 1.7 we
> still have workqueue operations that change the database which makes this
> unsafe (like the old loggy code was), but for 1.8 these cases are
> removed/resolved.
>
> The remaining problem with enabling that, is that we should introduce a new
> check point for outstanding operations. Maybe at the point where we obtain a
> working copy lock?
If that could be implemented, I guess the non-exclusive mode would
become more stable. If our code takes care of splitting up large write
operations into not-too-large individual transactions, then it sounds
like concurrency (one writer, many readers) should work quite well.
I'm wondering if we can more clearly define the two different modes,
with accurately described behavior: the non-exclusive mode which
maximizes the ability to do things concurrently (at the cost of some
performance), and the exclusive mode which maximizes single-threaded
performance, but allows no concurrency at all.
Are there things that can be done to make the non-exclusive mode more
concurrency-friendly (like Bert's suggestion about the workqueue
limitation)?
--
Johan
Received on 2012-10-25 00:20:49 CEST