So, I have a most troubling issue: Intermittent blocking by our
pre-commit hook, with the only output being:
svn: 'pre-commit' hook failed (did not exit cleanly: apr_exit_why_e
was 2, exitcode was -1073741502). with no output.
This hook is very simple: just checks that the log for the commit
contains something (anything),
and succeed if there is content in the log. Otherwise fail.
Straightforward, nothing special, log requirement on commits.
This error only pops up intermittently: a service restart of Apache
will give us about 24-48 hours without any errors.
A server restart usually buys 1-2 weeks. Either way, this error comes
back, and when it happens, there's usually
about a 50-50 chance that when the user tries a second time, they do
not get this error and can commit successfully,
having changed nothing on their end or on the server. When the problem
doesn't somehow fix itself, it usually
persists for a couple of hours until we restart something server-side.
But here's the kicker: The svn binaries that are checking the log for
content are separate from the binaries
used by the Apache service (Why is a good question. The answer is: I
have no clue). Switching which
binaries are used doesn't help at all, and makes it strange that an
Apache restart alleviates things.
I also have found no correlations to the amount of traffic we're
receiving to the server, the amount of disk space
used, disk i/o, or anything other machine metric. Memory is around 30%
used, processor averages 50% load.
I've tried troubleshooting the running httpd process and its spawns,
found noerrors or unusual activity,
and this blocking doesn't seem to affect all repositories, just a
random few at a time.
It's also extremely hard to reproduce intentionally. This server has
ran for years with no issues, and
nothing in our environment has changed recently.
It also affects both Windows and Linux clients, both in and out of the
U.S., and locale settings are universal.
I've run stuck after pursuing this for the last 2 months with no
success. If anyone could offer some more
troubleshooting tips or possibilities, that would be a wonderful blessing.
Some details on the server and SVN:
Server: Windows 2008 R2 (VM) running Apache 2.2.24 (64bit)
Subversion: 1.7.14 and 1.6.2 (Server has1.7.14)
Traffic: Servers 7k+ repos (2k actively used), ~26mil hits a day
(mostly ALM tools like Jenkins)
Disk: 2.5 TB NTFS format
Received on 2016-07-15 23:10:42 CEST