Subversion service needs restart after it received to many errors
Date: Wed, 7 Nov 2012 16:47:02 +0100
Dear SVN Buddies,
(directed to this mailing list from:
http://subversion.tigris.org/issue-tracker.html)
At our company we're using Subversion for some years now but are coping
with a strange server instability for some months now. After a random
period, the Subversion server stops responding and users cannot
update/commit/navigate the Subversion repository. Restarting the
Subversion service seems to resolve the problem.
I'd like to explain the problem as much as possible below. Hopefully you
can help me out here, or at least guide me to contacts that can help me
with analyzing the problem.
I did a lot of Google searched on the errors I received in the logfiles,
but wasn't able to find any topics that could really help me out.
Thanks in advance!
Configuration - Subversion Server*
Subversion Distribution CollabNet Subversion Edge (1.2)
Software version 1.2.1-1406.49
Subversion version 1.6.12-1406.49
Operating System Windows Server 2003 SP2, 2Gb
(*This is a virtual server, running on VMWare 8.)
Configuration - Subversion Clients
We're running mixed clients. Our company policy is to use SmartSVN as
Subversion client, however, some people may be using different clients,
and we're aware of at least people running Versions (Mac OS X) and
TortoiseSVN (Windows). As we're also developing on Linux, command-line
Subversion builds may be used as well.
Subversion clients used mostly are 1.6 and 1.7 (we officially introduced
SmartSVN 7 a few months ago after purchasing a bunch of licenses) but I
recently found some project teams still running 1.5.
Problem analysis
Now it's getting interesting.
I studied some logfiles in: \csvn\data\logs (That's where CollabNet
Subverion stores the logfiles)
error_[date].log: After the service failed to respond, I restarted the
Subversion server and started analyzing this file. This file grew
approx. 80Mb in less than 20 minutes. But I also encountered older error
files that exceeded 200Mb, the largest being 3Gb.
[Tue Nov 06 14:36:27 2012] [error] (OS 6)The handle is invalid. : Child
3880: Encountered too many errors accepting client connections. Possible
causes: Unknown. Try using the Win32DisableAcceptEx directive.
[Tue Nov 06 14:36:27 2012] [error] (OS 6)The handle is invalid. : Child
3880: Encountered too many errors accepting client connections. Possible
causes: Unknown. Try using the Win32DisableAcceptEx directive.
Some other logfile (a few days earlier)
[Wed Oct 31 13:34:02 2012] [error] (OS 6)The handle is invalid. : Child
212: Encountered too many errors accepting client connections. Possible
causes: Unknown. Try using the Win32DisableAcceptEx directive.
[Wed Oct 31 13:34:02 2012] [error] (OS 6)The handle is invalid. : Child
212: Encountered too many errors accepting client connections. Possible
causes: Unknown. Try using the Win32DisableAcceptEx directive.
During this unresponsive periode, it writes approx. 200-300 lines per
second to the error log. A lot.
Before the lockup, some other errors can be found:
[Wed Oct 31 09:43:47 2012] [error] [client 172.17.64.54] Provider
encountered an error while streaming a REPORT response. [500, #0]
[Wed Oct 31 09:43:47 2012] [error] [client 172.17.64.54] A failure
occurred while driving the update report editor [500, #620018]
[Wed Oct 31 09:43:47 2012] [error] [client 172.17.64.54] Provider
encountered an error while streaming a REPORT response. [500, #0]
[Wed Oct 31 09:43:47 2012] [error] [client 172.17.64.54] A failure
occurred while driving the update report editor [500, #620018]
These are written to the error log as well, approx. 25-30 lines per
second, and it looks like it keeps writing to the logfile until it has
received too many errors.
From what I know and have read, this may have to do with the DAV SVN
module for Apache. This may be an error in the module, but I also found
articles that say that these errors can still mean something went wrong
within the Subversion part, who knows. I don't. :D
access_[date].log: Interestingly, the access logfile proves that the
client request that generates this error is repeating every second. Two
different access logfiles:
172.17.96.37 - - [06/Nov/2012:10:53:15 +0100] "OPTIONS
/svn/priva_ux/Priva/Projects HTTP/1.1" 401 401
172.17.96.37 - user1 [06/Nov/2012:10:53:15 +0100] "OPTIONS
/svn/priva_ux/Priva/Projects HTTP/1.1" 200 -
172.17.96.37 - user1 [06/Nov/2012:10:53:15 +0100] "PROPFIND
/svn/priva_ux/Priva/Projects HTTP/1.1" 207 712
172.17.96.37 - user1 [06/Nov/2012:10:53:15 +0100] "PROPFIND
/svn/priva_ux/!svn/vcc/default HTTP/1.1" 207 411
172.17.96.37 - user1 [06/Nov/2012:10:53:15 +0100] "PROPFIND
/svn/priva_ux/!svn/bln/230677 HTTP/1.1" 207 428
172.17.96.37 - user1 [06/Nov/2012:10:53:15 +0100] "PROPFIND
/svn/priva_ux/!svn/bc/230677/Priva/Projects HTTP/1.1" 207 346
172.17.96.37 - user1 [06/Nov/2012:10:53:15 +0100] "PROPFIND
/svn/priva_ux/Priva/Projects HTTP/1.1" 207 712
172.17.96.37 - user1 [06/Nov/2012:10:53:15 +0100] "REPORT
/svn/priva_ux/!svn/vcc/default HTTP/1.1" 200 16041
172.17.64.17 - - [06/Nov/2012:17:24:18 +0100] "OPTIONS
/svn/tcx_platform/trunk HTTP/1.1" 401 401
172.17.64.17 - user2 [06/Nov/2012:17:24:18 +0100] "OPTIONS
/svn/tcx_platform/trunk HTTP/1.1" 200 -
172.17.64.17 - user2 [06/Nov/2012:17:24:18 +0100] "PROPFIND
/svn/tcx_platform/trunk HTTP/1.1" 207 702
172.17.64.17 - user2 [06/Nov/2012:17:24:18 +0100] "PROPFIND
/svn/tcx_platform/!svn/vcc/default HTTP/1.1" 207 418
172.17.64.17 - user2 [06/Nov/2012:17:24:18 +0100] "PROPFIND
/svn/tcx_platform/!svn/bln/23230 HTTP/1.1" 207 434
172.17.64.17 - user2 [06/Nov/2012:17:24:18 +0100] "PROPFIND
/svn/tcx_platform/!svn/bc/23230/trunk HTTP/1.1" 207 340
172.17.64.17 - user2 [06/Nov/2012:17:24:18 +0100] "PROPFIND
/svn/tcx_platform/trunk HTTP/1.1" 207 702
172.17.64.17 - user2 [06/Nov/2012:17:24:18 +0100] "REPORT
/svn/tcx_platform/!svn/vcc/default HTTP/1.1" 200 16021
Those requests seem to generate an error on the server. The clients keep
repeating this eternally, until the server can't take it anymore.
What did they do? What are they using ...
I was able to back trace the IP addresses to the corresponding users.
The users I spoke told me that they we're just doing their work,
committing files, or updating their working copy, nothing special. Most
of the users I spoke, but I'm not 100% sure, are using SmartSVN. This
-may- be an indication, however, I'm not really sure that this specific
client is doing something wrong.
Test/spare server - Subversion Server*
We're also having a spare/test server running. I copied a few
repositories to that server, installed the latest version of Collabnet
Subversion Edge, and did some experiments.
Subversion Distribution CollabNet Subversion Edge (3.1.2)
Software version 3.1.2-3272.97
Subversion version 1.7.6-3272.97
Operating System Windows Server 2003 SP2, 2Gb
(*This is a virtual server, running on VMWare 8.)
However, I was able to (randomly) reproduce this error on this server,
using a SmartSVN 7 client. I'm not sure what this means, but the same
errors are logged and even this server eventually locked up and required
a service restart.
Conclusion
I'm almost out of options. Wasn't able to find too many Google hits on
people having same log file errors, which does concern me.
Further options to investigate:
* Clean Windows installation
* Other OS versions (Windows Server 2008, Linux distro's)
* Reproduce errors with different client versions
I'm not really sure what to do. I'd like to receive some advices on how
to isolate this problem and work to a solution.
Thanks in advance, Looking forward to hearing from you, Kind regards,
Remko Seelig
Strategic Professional Configuration Management
Research & Development
Priva
Zijlweg 3
Postbus 18
2678 ZG De Lier
T +31(0)174522 498
M +31(0)6 150 103 97
F +31 (0)174 522 700
www.priva.nl <http://www.priva.nl>
remko.seelig_at_priva.nl <mailto:remko.seelig_at_priva.nl>
______________________________________________________________________
Priva B.V.
A disclaimer is applicable to this email, please refer to www.priva.nl/disclaimer.
Op deze email is een disclaimer van toepassing, ga naar www.priva.nl/disclaimer.
