On Wed, May 6, 2015 at 1:18 AM, Philip Martin <philip.martin_at_wandisco.com>
wrote:
> fs-test 44 shows a problem when multiple threads access an FSFS txn, the
> txn_dir_cache makes it possible for changes to be lost. Today I have
> been seeing this test XPASS instead of XFAIL about 1 time in 30 when
> running "./fs-test --parallel". I don't understand why. I see this
> with 1.9.x and trunk, I am running the tests on a RAM disk, I still see
> the XPASS if I go back to r1617500 when the XFAIL was introduced.
>
> The test creates a txn, makes a change, then creates an additional
> thread that makes a second change, waits for the additional thread to
> complete, commits the txn. Finally the test looks for the change made
> by the additional thread to be present in the revision and is an XFAIL
> because that change is not usually present. When I see the XPASS the
> change is present.
>
> It is possible to make the test always XPASS by setting
> fs->fsap_data->txn_dir_cache to NULL just before the commit, but if I
> patch the code I see that is not happening when I see the random XPASS.
>
That's good because it makes pool handling issues etc.
much less likely.
> I have tried increasing the size of the FSFS memory cache by a factor of
> 100 and the random XPASS still occurs.
>
Hm. If what I will suggest below actually identifies the problem,
then the cache size should have some impact.
If I apply the patch below to print details of the cache the XFAIL is
> usually:
>
> reads: 0 writes: 0 hits: 0 pretend: 0
> reads: 3 writes: 2 hits: 1 pretend: 0
> reads: 3 writes: 2 hits: 1 pretend: 0
> ../src-1.9/subversion/tests/libsvn_fs/fs-test.c:5705:
> (apr_err=SVN_ERR_TEST_FAILED)
> svn_tests: E200006: assertion 'value && !strcmp(value->data, "value")'
> failed at ../src-1.9/subversion/tests/libsvn_fs/fs-test.c:5705
> XFAIL: lt-fs-test 44: test reopen and modify txn
>
> but occasionally I see
>
> reads: 0 writes: 0 hits: 0 pretend: 0
> reads: 3 writes: 3 hits: 0 pretend: 0
> reads: 3 writes: 3 hits: 0 pretend: 0
> svn_tests: E200006: assertion 'value && !strcmp(value->data, "value")'
> failed at ../src-1.9/subversion/tests/libsvn_fs/fs-test.c:5705
> XFAIL: lt-fs-test 44: test reopen and modify txn
>
IOW, sometimes the data can't be found in the cache, hence it
will be read from disk and the failure condition is not met.
> I don't understand why the cache behaviour varies. Although the test
> uses 2 threads the test is designed so that the threads always do things
> in the same order.
>
Here is what I believe causes the different behaviour between
test runs. To support read function that may block, e.g. wait for
network buffers to flush, writes have been made non-blocking
if they don't update existing cache entries. If the write lock can
not be acquired immediately in that case, the new item will simply
not be added to the cache.
This behaviour is optional and can be disabled via flag:
[[[
Index: subversion/libsvn_subr/cache_config.c
===================================================================
--- subversion/libsvn_subr/cache_config.c (revision 1675401)
+++ subversion/libsvn_subr/cache_config.c (working copy)
@@ -126,7 +126,7 @@
(apr_size_t)(cache_size / 5),
0,
! svn_cache_config_get()->single_threaded,
- FALSE,
+ TRUE,
pool);
/* Some error occurred. Most likely it's an OOM error but we don't
]]]
> Every XPASS has always been the uncommon 3,3,0,0 rather than the common
> 3,2,0,0.
>
That makes sense.
> I only ever see the 3,3,0,0 when running fs-tests with --parallel, in
> sequential mode it is always 3,2,0,0 and I never see the XPASS.
>
(The second one is probably a typo and should read 3,2,1,0.)
Does the above patch fix the non-deterministic behaviour?
-- Stefan^2.
Received on 2015-05-06 04:05:32 CEST