RE: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

Hi Ivan.

I managed to get the debug symbols for the Subversion binaries, we've been
using. It can be downloaded at
http://www.apachehaus.de/subversion-1.9.3-ap24-x64_pdb.zip.

Thank you,
Radek Krotil

-----Original Message-----
From: Radek Krotil [mailto:radek.krotil_at_polarion.com]
Sent: Saturday, May 21, 2016 10:46 AM
To: 'Ivan Zhakov'
Cc: 'dev_at_subversion.apache.org'
Subject: RE: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

Hi Ivan.

Apologies for my late reply. I've been caught in other tasks and events, but
recently I've encountered the behavior again and also similar pattern was
observed in environment of one of our enterprise customer.

All the dumps from svnserve, our application and also the memory dump can be
downloaded at https://drive.google.com/open?id=0B-yalijSzAIULWNuRGNGUW9KeWM.
I've been using the binaries from
http://de.apachehaus.com/downloads/mod_svn-1.9.3-ap24-x64.zip.

So to answer your questions..
1. Do you have debug symbols for Subversion binaries you're using?
A: I'm not an expert in C development, so I will need to consult with one of
my developers to understand, what you actually ask for. Maybe you can figure
the necessary details from the binaries themselves, or maybe you can give me
bit more hint in how to get the information you seek.

2. Other threads should have different stack trace, because AFAIK only one
thread calls accept().
A: Correct. All the stacktraces are captured in the zip file above. Majority
of the threads are in the following state, but our application does not
receive any response for a long time.

ntoskrnl.exe!KeSynchronizeExecution+0x3de6
ntoskrnl.exe!KeWaitForMutexObject+0xc7a
ntoskrnl.exe!KeWaitForMutexObject+0x709
ntoskrnl.exe!KeWaitForMutexObject+0x375
ntoskrnl.exe!IoThreadToProcess+0xff0
ntoskrnl.exe!KeRemoveQueueEx+0x16ba
ntoskrnl.exe!KeWaitForMutexObject+0xe8e
ntoskrnl.exe!KeWaitForMutexObject+0x709
ntoskrnl.exe!KeWaitForMutexObject+0x375
ntoskrnl.exe!NtWaitForSingleObject+0xf2
ntoskrnl.exe!setjmpex+0x3963
ntdll.dll!NtWaitForSingleObject+0x14
MSWSOCK.dll!Tcpip6_WSHSetSocketInformation+0x155
MSWSOCK.dll!WSPStartup+0x3b1c
WS2_32.dll!WSARecv+0x17d
libapr-1.dll!apr_socket_recv+0x4a
svnserve.exe+0x15895
svnserve.exe+0x10a6d
svnserve.exe+0x1175b
svnserve.exe+0x1f8f
svnserve.exe+0xbeb9
libaprutil-1.dll!apr_thread_pool_top+0x797
MSVCR110.dll!beginthreadex+0x107
MSVCR110.dll!endthreadex+0x192
KERNEL32.DLL!BaseThreadInitThunk+0x22
ntdll.dll!RtlUserThreadStart+0x34

3. Could you please create full memory dump of locked process and send it to
me including debug symbols?
Dump is part of the zip file and for the debug symbol, I'll follow up on
that during the next week with my developers.

Thank you,
Radek Krotil

-----Original Message-----
From: Ivan Zhakov [mailto:ivan_at_visualsvn.com]
Sent: Friday, April 15, 2016 9:46 AM
To: Radek Krotil
Cc: dev_at_subversion.apache.org
Subject: Re: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

On 14 April 2016 at 19:14, Radek Krotil <radek.krotil_at_polarion.com> wrote:
> Hi all.
>
> Our application generates lot of concurrent read requests to
> subversion using svn: protocol. When we tested the multithreaded mode
> of svnserve after upgrade to 1.9.3, we noticed strange 'deadlock-like'
> behavior: at some point all the requests are blocked in svnserve and
> wait there for a few minutes (3 to 15 minutes, no CPU activity), after
> which they continue to work. This is making our application significantly
> slower.
>
> Operating system: Windows 10, CentOS 6.6, CentOS 7.2
>
> The release and/or revision of Subversion: 1.9.3
>
> The compiler and configuration options you built Subversion with:
> WANDisco binaries for CentOS, Apache Haus binaries for Windows
>
> The workaround on Linux is to run svnserve without -T switch, i.e. not
> using multithreaded mode. For Windows, there is no workaround as
> svnserve only supports the multi-threaded mode.
>
> Here is a sample of thread dump of svnserve.exe during the 'deadlock'
> obtained on Windows 10 using Process Explorer:
>
> ntoskrnl.exe!KeSynchronizeExecution+0x3de6
> ntoskrnl.exe!KeWaitForMutexObject+0xc7a
> ntoskrnl.exe!KeWaitForMutexObject+0x709
> ntoskrnl.exe!KeWaitForMutexObject+0x375
> ntoskrnl.exe!IoThreadToProcess+0xff0
> ntoskrnl.exe!KeRemoveQueueEx+0x16ba
> ntoskrnl.exe!KeWaitForMutexObject+0xe8e
> ntoskrnl.exe!KeWaitForMutexObject+0x709
> ntoskrnl.exe!KeWaitForMutexObject+0x375
> ntoskrnl.exe!NtWaitForSingleObject+0xf2
> ntoskrnl.exe!setjmpex+0x3963
> ntdll.dll!NtWaitForSingleObject+0x14
> MSWSOCK.dll!Tcpip6_WSHSetSocketInformation+0x155
> MSWSOCK.dll+0x1bf1
> WS2_32.dll!WSAAccept+0xce
>
> WS2_32.dll!accept+0x12
> libapr-1.dll!apr_socket_accept+0x46
> svnserve.exe+0xc11c
> svnserve.exe+0xbae5
> svnserve.exe+0xaf6c
> svnserve.exe+0x13ab
> KERNEL32.DLL!BaseThreadInitThunk+0x22
> ntdll.dll!RtlUserThreadStart+0x34
>
> The similar stack can be seen with other threads too.
>
1. Do you have debug symbols for Subversion binaries you're using?
2. Other threads should have different stack trace, because AFAIK only one
thread calls accept().
3. Could you please create full memory dump of locked process and send it to
me including debug symbols?

--
Ivan Zhakov