Re: Random serf checkout failures

Lieven Govaerts <svnlgo_at_mobsol.be> writes:

> Are you using https or http?

http

> I've done similar tests with success recently, but that was on
> Windows. This was to the European mirror of the apache repository
> though, I don't have a local mirror available.

I suppose you could create a data set by importingd trunk a hundred
times modifying about 200 files each time.

> Are you willing to use serf trunk instead of 1.1.x? I've added logging
> to serf trunk, which has helped me a lot in analyzing serf behavior.
> The relevant flags are in serf_private.h. I'd set SOCK_VERBOSE,
> CONN_VERBOSE, SSL_VERBOSE and one of SSL_MSG_VERBOSE or
> SOCK_MSG_VERBOSE. All output is written to stderr so easy to redirect
> to file.

I can reproduce with trunk. Here's the end of the log for a checksum
failure. It appears to simply stop receiving data in the middle of a
file, hence the checksum failure. The 70014 error is APR_EOF.

HTTP/1.1 200 OK
Date: Fri, 02 Nov 2012 21:56:55 GMT
Server: Apache/2.2.22 (Ubuntu) mod_ssl/2.2.22 OpenSSL/1.0.1 DAV/2 SVN/1.8.0-dev
Last-Modified: Tue, 24 Jul 2001 18:49:22 GMT
ETag: "839773//subversion/tags/0.9.0/subversion/libsvn_repos/hooks.txt"
Cache-Control: max-age=604800
Accept-Ranges: bytes
Content-Length: 9192
Content-Type: text/plain

                           Repository Hooks
                           ================

    ... text elided ...

   The point of sentinels is to provide real-time feedback as a commit
   is being built (or even before the txn is started), or as a
   checkout or update is being produced -- but without the overhead of
   st
-(5268)-
[2012-11-02T21:57:11.198912+00] [l:0.0.0.0:0 r:192.168.0.102:8888] buckets/socket_buckets.c: socket_recv error 70014
../src/subversion/svn/checkout-cmd.c:168: (apr_err=200014)
../src/subversion/libsvn_client/checkout.c:179: (apr_err=200014)
../src/subversion/libsvn_client/update.c:579: (apr_err=200014)
../src/subversion/libsvn_client/update.c:440: (apr_err=200014)
../src/subversion/libsvn_wc/adm_crawler.c:858: (apr_err=200014)
../src/subversion/libsvn_ra_serf/update.c:2515: (apr_err=200014)
../src/subversion/libsvn_ra_serf/util.c:2028: (apr_err=200014)
../src/subversion/libsvn_ra_serf/util.c:2009: (apr_err=200014)
../src/subversion/libsvn_ra_serf/update.c:1065: (apr_err=200014)
../src/subversion/libsvn_ra_serf/update.c:895: (apr_err=200014)
../src/subversion/libsvn_wc/update_editor.c:3921: (apr_err=200014)
../src/subversion/libsvn_subr/checksum.c:469: (apr_err=200014)
svn: E200014: Checksum mismatch for '/home/pm/sw/subversion/obj/wc/0.9.0/subversion/libsvn_repos/hooks.txt':
   expected: f43fbb1f0c8a6ab22e12a540cbda4aca
     actual: 835e68f9025e7f567d68be840bef249c

[2012-11-02T21:57:16.321729+00] [l:0.0.0.0:0 r:192.168.0.102:8888] outgoing.c: cleanup - closed socket, status 9
[2012-11-02T21:57:16.321796+00] [l:0.0.0.0:0 r:192.168.0.102:8888] outgoing.c: cleanup - closed socket, status 9
[2012-11-02T21:57:16.321840+00] [l:0.0.0.0:0 r:192.168.0.102:8888] outgoing.c: cleanup - closed socket, status 9
[2012-11-02T21:57:16.321873+00] outgoing.c: cleaning up connection 0x25e8148
[2012-11-02T21:57:16.322017+00] outgoing.c: closed connection 0x25e8148
[2012-11-02T21:57:16.322044+00] outgoing.c: cleaning up connection 0x2526b58
[2012-11-02T21:57:16.322057+00] outgoing.c: closed connection 0x2526b58
[2012-11-02T21:57:16.322087+00] outgoing.c: cleaning up connection 0x247a768
[2012-11-02T21:57:16.322104+00] outgoing.c: closed connection 0x247a768
[2012-11-02T21:57:16.322125+00] outgoing.c: cleaning up connection 0x23eb838
[2012-11-02T21:57:16.322136+00] outgoing.c: closed connection 0x23eb838

Here's another failure. I was using the default Apache timeouts which
should be 300s and this whole checkout was less than that (from 22:02:54
to 22:04:03). I've upped the timeout to 3000s and it still fails:

[2012-11-02T22:04:01.384267+00] [l:0.0.0.0:0 r:192.168.0.102:8888] buckets/socket_buckets.c: --- socket_recv:
HTTP/1.1 408 Request Time-out
Date: Fri, 02 Nov 2012 22:03:57 GMT
Server: Apache/2.2.22 (Ubuntu) mod_ssl/2.2.22 OpenSSL/1.0.1 DAV/2 SVN/1.8.0-dev
Content-Length: 223
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>408 Request Time-out</title>
</head><body>
<h1>Request Time-out</h1>
<p>Server timeout waiting for the HTTP request from the client.</p>
</body></html>

-(459)-
../src/subversion/svn/checkout-cmd.c:168: (apr_err=20014)
../src/subversion/libsvn_client/checkout.c:179: (apr_err=20014)
../src/subversion/libsvn_client/update.c:579: (apr_err=20014)
../src/subversion/libsvn_client/update.c:440: (apr_err=20014)
../src/subversion/libsvn_wc/adm_crawler.c:858: (apr_err=20014)
../src/subversion/libsvn_ra_serf/update.c:2518: (apr_err=20014)
svn: E020014: Error retrieving REPORT: Internal error
[2012-11-02T22:04:03.023422+00] [l:0.0.0.0:0 r:192.168.0.102:8888] outgoing.c: cleanup - closed socket, status 9
[2012-11-02T22:04:03.023532+00] [l:0.0.0.0:0 r:192.168.0.102:8888] outgoing.c: cleanup - closed socket, status 9
[2012-11-02T22:04:03.023655+00] [l:0.0.0.0:0 r:192.168.0.102:8888] outgoing.c: cleanup - closed socket, status 9
[2012-11-02T22:04:03.023727+00] [l:0.0.0.0:0 r:192.168.0.102:8888] outgoing.c: cleanup - closed socket, status 9
[2012-11-02T22:04:03.023782+00] outgoing.c: cleaning up connection 0x2237288
[2012-11-02T22:04:03.023835+00] outgoing.c: closed connection 0x2237288
[2012-11-02T22:04:03.023860+00] outgoing.c: cleaning up connection 0x2199db8
[2012-11-02T22:04:03.023881+00] outgoing.c: closed connection 0x2199db8
[2012-11-02T22:04:03.023906+00] outgoing.c: cleaning up connection 0x20ecb28
[2012-11-02T22:04:03.023965+00] outgoing.c: closed connection 0x20ecb28
[2012-11-02T22:04:03.024011+00] outgoing.c: cleaning up connection 0x2084838
[2012-11-02T22:04:03.024032+00] outgoing.c: closed connection 0x2084838

-- 
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download