[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Using packet loss to hunt the FSFS corruption

From: Daniel Shahaf <d.s_at_daniel.shahaf.name>
Date: Thu, 9 Sep 2010 22:46:16 +0300

Morning Stefan,

Per IRC today, I've run 'make davautocheck' while subjected to 5% packet loss
(via iptables -A INPUT -i lo -p tcp -m tcp --dport 8088 \
              -m statistic --mode random --probability 0.05 -j DROP),
and while using a patch (attached) to run 'svnadmin setrevprop'
concurrently to the tests.

The patch meant that virtually all svnrdump and svnsync tests failed
(since the dump of r0 differed from the expectation). However, over
serf svnadmin_tests 20 also failed, and over neon three tests also
failed; the fails.log for neon is below. (I didn't save the fails.log
for serf...)

In all three cases, this is the error:

  File "/home/daniel/src/svn/trunk.d/subversion/tests/cmdline/svntest/actions.py", line 147, in guarantee_greek_repository
    main.chmod_tree(path, 0666, 0666)
  File "/home/daniel/src/svn/trunk.d/subversion/tests/cmdline/svntest/main.py", line 648, in chmod_tree
    new_mode = (os.stat(fullname)[stat.ST_MODE] & ~mask) | mode
OSError: [Errno 2] No such file or directory: 'svn-test-work/repositories/depth_tests-9/db/revprops/0/svn-pnTC7M'

Which seems to be uninteresting. (I guess the tempfile used by
'setrevprop' to stage the propchange was removed between the time the
iterator yielded it and the point the iterating loop tried to chmod it)

So, back to the drawing board wrt looking for ways to reproduce the
corruption.

We could repeat the packet-loss exercise with a higher percentage
than 5%. Also, we could have the tests run 'svnadmin verify' and
'fsfsverify.py' on every repository created in the tests, to ensure that
we actually detect the corruption if it occurs :-). Or we could do
something else --- ideas?

Daniel
Received on 2010-09-09 21:55:04 CEST

This is an archived mail posted to the Subversion Dev mailing list.