On 7/16/2016 01:48, Stefan wrote:
> Hi Tim,
>
> On 7/15/2016 21:49, Tim Tim wrote:
>> So, I have a most troubling issue: Intermittent blocking by our
>> pre-commit hook, with the only output being:
>>
>> svn: 'pre-commit' hook failed (did not exit cleanly: apr_exit_why_e
>> was 2, exitcode was -1073741502). with no output.
>>
>> This hook is very simple: just checks that the log for the commit
>> contains something (anything),
>> and succeed if there is content in the log. Otherwise fail.
>> Straightforward, nothing special, log requirement on commits.
>>
>> This error only pops up intermittently: a service restart of Apache
>> will give us about 24-48 hours without any errors.
>> A server restart usually buys 1-2 weeks. Either way, this error comes
>> back, and when it happens, there's usually
>> about a 50-50 chance that when the user tries a second time, they do
>> not get this error and can commit successfully,
>> having changed nothing on their end or on the server. When the problem
>> doesn't somehow fix itself, it usually
>> persists for a couple of hours until we restart something server-side.
>>
>> But here's the kicker: The svn binaries that are checking the log for
>> content are separate from the binaries
>> used by the Apache service (Why is a good question. The answer is: I
>> have no clue). Switching which
>> binaries are used doesn't help at all, and makes it strange that an
>> Apache restart alleviates things.
>>
>> I also have found no correlations to the amount of traffic we're
>> receiving to the server, the amount of disk space
>> used, disk i/o, or anything other machine metric. Memory is around 30%
>> used, processor averages 50% load.
>>
>> I've tried troubleshooting the running httpd process and its spawns,
>> found noerrors or unusual activity,
>> and this blocking doesn't seem to affect all repositories, just a
>> random few at a time.
>> It's also extremely hard to reproduce intentionally. This server has
>> ran for years with no issues, and
>> nothing in our environment has changed recently.
>>
>> It also affects both Windows and Linux clients, both in and out of the
>> U.S., and locale settings are universal.
>>
>> I've run stuck after pursuing this for the last 2 months with no
>> success. If anyone could offer some more
>> troubleshooting tips or possibilities, that would be a wonderful blessing.
>>
>> Some details on the server and SVN:
>> Server: Windows 2008 R2 (VM) running Apache 2.2.24 (64bit)
>> Subversion: 1.7.14 and 1.6.2 (Server has1.7.14)
>> Traffic: Servers 7k+ repos (2k actively used), ~26mil hits a day
>> (mostly ALM tools like Jenkins)
>> Disk: 2.5 TB NTFS format
> First of all Apache 2.2.24 as well as SVN 1.7.14/1.6.2 are rather
> ancient and SVN 1.6/1.7 is also no longer actively supported.
>
> If at all possible, I'd highly recommend you upgrade at least Apache to
> the latest 2.2 version (at the time of writing this is 2.2.31). The same
> goes for SVN. At least make sure to have SVN 1.7.20 or 1.7.21 on the
Slight correction here: I mean SVN 1.7.21 or 1.7.22.
> server and no mixture of two different versions. Depending on how builds
> were made, you have a hard time determining which SVN version is used
> (refering here to the fact that Windows prefers using already loaded
> DLLs over loading them separately, if manifest information of the
> calling process are missing or not specific enough).
>
> If the issue persists after the upgrade you at least have a stable basis
> to start troubleshooting from.
>
> Regards,
> Stefan
>
>
Regards,
Stefan
Received on 2016-07-16 01:49:26 CEST