Re: attn: ghudson; FSFS and multi-threading on Unix broken?

Greg Hudson <ghudson@MIT.EDU> writes:

> On Tue, 2005-04-05 at 05:46, Peter N. Lundblad wrote:
>> As you see, I haven't fleshed out the resolution proposal very much. But I
>> want people to be aware of the possibility that we have a serious FSFS
>> (dataloss?) bug. Please someone (ghudson?), tell me that this is a false
>> alarm! :-)
>
> I don't think it's a false alarm. But as you noted, it's rare to use
> svnserve in threaded mode under Unix. Under Windows, we know you can
> block against a lock you already hold (see recent deadlock issues), so
> locks are effectively per-thread. And I think it's currently unheard of
> for a third-party multithreaded Unix program to be interested in more
> than the network client part of the Subversion libraries.

The problem won't show up on older Linux systems because in the
LinuxThreads library fcntl was a per-thread lock, i.e. it had
non-POSIX behaviour. Newer Linux systems (generally kernel 2.6) use
NPTL and they can show the problem.

I believe I can trigger it by using stress.pl and svnserve in thread
mode. I ran the server on my Linux-2.6 laptop:

$ svnserve -Tdr.

and three clients on my Linux-2.4 desktop:

$ stress.pl -i1 -d -s0 -Usvn://debian1/repostress
$ stress.pl -i2 -d -s0 -Usvn://debian1/repostress
$ stress.pl -i3 -d -s0 -Usvn://debian1/repostress

One of the clients failed with:

Updated to revision 51.
Committing:
Sending wcstress.5742/trunk/bar1/foo1
Sending wcstress.5742/trunk/bar1/foo2
Sending wcstress.5742/trunk/foo1
Sending wcstress.5742/trunk/foo2
Transmitting file data ....../svn/subversion/libsvn_client/commit.c:781: (apr_err=160028)
svn: Commit failed (details follow):
../svn/subversion/libsvn_repos/commit.c:120: (apr_err=160028)
svn: Out of date: '/trunk/bar1/foo1' in transaction '52-1'
Updating:
../svn/subversion/libsvn_wc/update_editor.c:1609: (apr_err=155017)
svn: Checksum mismatch for 'wcstress.5742/trunk/.svn/text-base/foo1.svn-base'; expected: 'eed54c9b53e24e2ff953797ea74d7dca', actual: '9a0a1ec9dd62dfc56be947ba36df0ede'
unexpected update fail: exit status: 256

The wc diff looks as expected:

$ svn diff wcstress.5742/trunk/foo1
Index: wcstress.5742/trunk/foo1
===================================================================
--- wcstress.5742/trunk/foo1 (revision 51)
+++ wcstress.5742/trunk/foo1 (working copy)
@@ -1,7 +1,7 @@
 A0
 0
 A1
-1,1,3,4,5,6,7,9,1,3,4,5,6,7,9
+1,1,3,4,5,6,7,9,1,3,4,5,6,7,9,10
 A2
 2,1,3,4,5,6,7,1,3,4,5,6,7,9,10,12
 A3

But comparing against the repository I get

$ svn diff -r50 wcstress.5742/trunk/foo1
../svn/subversion/libsvn_delta/text_delta.c:594: (apr_err=200003)
svn: Delta source ended unexpectedly

and doing an explicit cat/diff I see a missing modification:

$ svn cat -r51 svn://debian1/repostress/trunk/foo1 > xx
$ diff -u wcstress.5742/trunk/foo1 xx
--- wcstress.5742/trunk/foo1 Tue Apr 5 17:20:08 2005
+++ xx Tue Apr 5 17:31:37 2005
@@ -1,11 +1,11 @@
 A0
 0
 A1
-1,1,3,4,5,6,7,9,1,3,4,5,6,7,9,10
+1,1,3,4,5,6,7,9,1,3,4,5,6,7
 A2
 2,1,3,4,5,6,7,1,3,4,5,6,7,9,10,12
 A3
-3,1,1,3,4,5,6,7,9,10
+3,1,1,3,4,5,6,7,9,10,12
 A4
 4
 A5

I assume Apache using the worker MPM will have the same problem.

-- 
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org