On 09.01.2014 20:19, Branko Čibej wrote:
> On 09.01.2014 17:09, Mojca Miklavec wrote:
>> I'm unable to reproduce the faulty behaviour if I do a checkout from
>> the same network where the server is located, no matter what I try
>> (upgrading SVN client doesn't "help" triggering the error). Philip
>> also said that he had no problem doing a checkout with client version
>> 1.8.5 or 1.7.
>
> This confirms my suspicion that the error is triggered by some part of
> the network infrastructure between your server and the outside world.
> That's why I asked if there is a load-balancer involved. It could also
> be caused by some kind of transparent proxy, or even a packet
> analyzer. I doubt that your server is open to the world without some
> kind of security measures in place.
>
> To be clear, I'm not saying that any of these things are configured
> incorrectly; only that they may be interacting with Subversion in a
> way that we don't handle well. One of the major differences between
> 1.7 (which works) and 1.8 (which fails) is that we try to work around
> issues with non-standard behaviour of certain "transparent" (sic)
> proxies; and we can't claim to have covered all the possibilities.
>
> I can't see a way to figure out what's going on without help from your
> network admins; we need some insight into why the connection is being
> reset on the server side, and analyzing the TCP stream on the client
> can't tell us that.
>
>
> BTW, if you think it'd help to try a live debugging session, I'm only
> about an hour's drive away from IJS.
So to wrap this up: they managed to fix the problem themselves, and it
was indeed "some part of the network infrastructure;" the specifics are
as follows:
They have a Cisco ASA 5580 running IOS 9.1(4), and they had HTTP
protocol inspection turned on; the configuration was as follows:
policy-map type inspect http HTTP-CONTROL
parameters
protocol-violation action log
policy-map global_policy
class inspection_default
inspect http HTTP-CONTROL
The ASA was closing the connections, and their logs contained one of the
following two reasons:
%ASA-4-415016: policy-map HTTP-CONTROL:Maximum number of unanswered HTTP requests exceeded - Resetting connection from Ext:x.x.x.x/59769 to Int:y.y.y.y/80
%ASA-4-507003: tcp flow from Ext:x.x.x.x/59769 to Int:y.y.y.y/80 terminated by inspection engine, reason - reset unconditionally.
The only reasonable explanation we could come up with was that
moderately low bandwidth and high latency between client and server,
coupled with the fact that some of the files in the repository are
rather large and take a while to transfer, caused the 1.8 client to
queue up enough pipelined GET requests during checkout that the firewall
decided to call quits. A 1.7 client (using serf) did not exhibit this
problem, because it also sends PROPGETS, and this apparently changed the
timing enough that the number of pipelined requests never exceeded the
ASA's configured maximum.
Apparently this is not a new problem, having been reported before:
https://supportforums.cisco.com/thread/2088590
They fixed the issue by switching off HTTP protocol inspection on the
ASA. Interestingly enough, this also fixed a number of intermittent
issues with plain ol' Web browsing that they had on occasion, so this is
not specific to Subversion (as the link above also suggests), but is
rather a bug^Wserious limitation of the ASA and/or IOS.
-- Brane
--
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane_at_wandisco.com
Received on 2014-01-15 19:54:33 CET