Hi Stu,
On Mon, Dec 23, 2013 at 6:11 PM, <stuartm.coding_at_gmail.com> wrote:
> On Tuesday, December 17, 2013 10:32:34 AM UTC-5, Branko Čibej wrote:
>>
>> On 17.12.2013 16:19, Stuart MacDonald wrote:
>> > The Wireshark trace shows clearly and conclusively this is a bug in
>> > the client. The client drops the connection by sending a FIN packet
>> > unexpectedly.
>>
>> It could be waiting for a response from the proxy that never reaches the
>> client, and dropping the connection after a timeout.
>>
>> Regardless, you should still try to reproduce this with the latest
>> client (ideally, both latest 1.7 and 1.8) to determine if the problem
>> has been fixed. Especially in 1.8, there have been a lot of improvements
>> in the HTTP protocol driver.
>
>
> I'm using WANdisco's packages to test with, as it turned out to be easier to
> get those set up than to compile from source with all the optional bits I
> need.
>
> I have seen the exact same issue on 1.7.14:
> svn: E175002: REPORT of '/svn/repos/<name>/!svn/me': Could not read response
> body: connection was closed by server (http://<server>)
> and the Wireshark capture shows the client sending a [FIN, ACK] to close the
> connection, despite it only being a fraction of a second since the last
> server traffic. This always occurs just after the TCP window opening up
> again, not sure if that's significant.
>
> This *also* occurs on 1.8.5, although the error message is more informative:
> svn: E120106: ra_serf: The server sent a truncated HTTP response body.
> The Wireshark capture is *exactly* *identical*; after some server traffic,
> the TCP window opens up and the client sends a [FIN, ACK] to close the
> connection.
to be clear, you've tested:
- svn 1.7.13 with ra_neon, svn 1.7.14 with ra_neon and 1.8.5 with ra_serf.
- to an svn 1.7.7 server
- over http, with nginx 1.0.5. as intermediate proxy
- running 'svn up'
- on an Ubuntu guest (version ______) using Virtualbox (version
__________) on host OS ________.
Can you fill in the blanks?
The error "The server sent a truncated HTTP response body." you're
seeing from ra_serf, means that ra_serf is busy reading a response
from the server but all data isn't arriving.
We have seen this caused by proxies tampering with the response data.
In this case I suppose the connection is shut down unexpectedly first,
which ra_serf interprets as 'the connection closed, but I didn't get
all data for this response yet" => error.
> Having taken a closer look at the network captures; the failure pattern is
> always:
> - 1380 byte packets from the server
> - an ACK from the client
> - a TCP window update from the client (the window hadn't gone to 0 in any
> cases, this is just a regular update)
> - in one case there's another window update from the client
> - a delay of 0 - 4 seconds where there is no traffic in either direction
> - the client sends the [FIN, ACK]
>
>
> Now, I'm not sure what constitutes a truncated HTTP response body, as the
> traffic looks the same in both cases, large blocks of encoded data. So I
> think the 1.8.x error message is also incorrect, but possibly less incorrect
> than the 1.7.x error message.
>
Your setup has many different layers so many possible failing components.
From your transcript of the wireshark trace it looks like the
connection is shut down on the client side (like you said).
You have tested two versions of subversion, each with their own http
client implementation (serf and neon). Both libraries have been
implemented completely independently by different people, without
sharing code. I find it difficult to believe that both libraries have
the same bug, but ok, it's not impossible.
In case of serf I can add that if serf would actively drop the
connection due to a timeout, you'd see a different error message. In
fact, I've searched for error E120106 and it's only raised when a
clean EOF is returned from the network layer, whereas in case of error
(TCP RST) the nw layer returns a ECONNABORTED or ECONNRESET status
code.
So, my first guess would be a problem with either the guest OS, or
with the Virtualbox network code (the latter I think is more likely).
> What's the next step?
1. If you like and are allowed to, you can send me the wireshark .pcap
file privately (replace username/passwd/basic/digest authn headers
with X, but keep the same # of chars!).
Also, the debug logging of both http client libraries would be nice:
- For svn 1.7.x put neon-debug-mask=255 in your servers file.
- For 1.8.5 this won't work anymore, you'll have to compile serf from
source to enable debugging (flags CONN_VERBOSE, SOCK_VERBOSE,
SOCK_MSG_VERBOSE in serf_private.h).
Maybe with this information I can confirm my reasoning or find another
potential cause.
2. I'd also test this same operation on a physical machine, with the
same OS and/or with a different OS.
3. Regardless of this issue, the upgrade of nginx is important for svn
1.8.5, as older versions of nginx have problems with serf's use of
HTTP 1.1 features. There's a fallback mechanism now, but at the cost
of some reduced performance.
>
> ...Stu
>
Lieven
Received on 2013-12-23 21:52:29 CET