Hi Tim,
On 7/15/2016 21:49, Tim Tim wrote:
> So, I have a most troubling issue: Intermittent blocking by our
> pre-commit hook, with the only output being:
>
> svn: 'pre-commit' hook failed (did not exit cleanly: apr_exit_why_e
> was 2, exitcode was -1073741502). with no output.
>
> This hook is very simple: just checks that the log for the commit
> contains something (anything),
> and succeed if there is content in the log. Otherwise fail.
> Straightforward, nothing special, log requirement on commits.
>
> This error only pops up intermittently: a service restart of Apache
> will give us about 24-48 hours without any errors.
> A server restart usually buys 1-2 weeks. Either way, this error comes
> back, and when it happens, there's usually
> about a 50-50 chance that when the user tries a second time, they do
> not get this error and can commit successfully,
> having changed nothing on their end or on the server. When the problem
> doesn't somehow fix itself, it usually
> persists for a couple of hours until we restart something server-side.
>
> But here's the kicker: The svn binaries that are checking the log for
> content are separate from the binaries
> used by the Apache service (Why is a good question. The answer is: I
> have no clue). Switching which
> binaries are used doesn't help at all, and makes it strange that an
> Apache restart alleviates things.
>
> I also have found no correlations to the amount of traffic we're
> receiving to the server, the amount of disk space
> used, disk i/o, or anything other machine metric. Memory is around 30%
> used, processor averages 50% load.
>
> I've tried troubleshooting the running httpd process and its spawns,
> found noerrors or unusual activity,
> and this blocking doesn't seem to affect all repositories, just a
> random few at a time.
> It's also extremely hard to reproduce intentionally. This server has
> ran for years with no issues, and
> nothing in our environment has changed recently.
>
> It also affects both Windows and Linux clients, both in and out of the
> U.S., and locale settings are universal.
>
> I've run stuck after pursuing this for the last 2 months with no
> success. If anyone could offer some more
> troubleshooting tips or possibilities, that would be a wonderful blessing.
>
> Some details on the server and SVN:
> Server: Windows 2008 R2 (VM) running Apache 2.2.24 (64bit)
> Subversion: 1.7.14 and 1.6.2 (Server has1.7.14)
> Traffic: Servers 7k+ repos (2k actively used), ~26mil hits a day
> (mostly ALM tools like Jenkins)
> Disk: 2.5 TB NTFS format
First of all Apache 2.2.24 as well as SVN 1.7.14/1.6.2 are rather
ancient and SVN 1.6/1.7 is also no longer actively supported.
If at all possible, I'd highly recommend you upgrade at least Apache to
the latest 2.2 version (at the time of writing this is 2.2.31). The same
goes for SVN. At least make sure to have SVN 1.7.20 or 1.7.21 on the
server and no mixture of two different versions. Depending on how builds
were made, you have a hard time determining which SVN version is used
(refering here to the fact that Windows prefers using already loaded
DLLs over loading them separately, if manifest information of the
calling process are missing or not specific enough).
If the issue persists after the upgrade you at least have a stable basis
to start troubleshooting from.
Regards,
Stefan
Received on 2016-07-16 01:48:42 CEST