RE: SlikSVN 1.6.6 64-bit stalls on some operations

> From: Johan Corveleyn [mailto:johan.corveleyn_at_uz.kuleuven.ac.be]
> > Van: Jason R. Coombs [mailto:jaraco_at_jaraco.com]
> >
> > Greetings.
> >
> >                 I've run into an issue with the latest release of
> SlikSVN
> > (1.6.6). I'm copying the problem description from an earlier mail to
> the
> > SlikSVN team, as the SlikSVN team has suggested I contact this list
> for
> > suggestions.
> >
> >                 I've been using SlikSVN for all of my 64-bit Windows
> systems.
> > Beginning with 1.6.6, I found that the svn client frequently stalls,
> for 120
> > seconds or more, on certain operations. It seems to stall in
> particular at
> > the following places:
> >
> > 1)            At the beginning of a checkout operation, before
> copying any
> > files.
> > 2)            At the end of an import or checkin operation, after the
> server
> > has completed the transaction, but before the client reports the new
> revision
> > number.
> > 3)            At the end of an list operation, before returning
> control to
> > the parent process.
> > 4)            At the end of some update operations, before returning
> control
> > to the parent process.
> >
> > While the client is stalling, I can check the network
> connections and
> > see that a TCP connection is open to the server, so the problem
> appears to be
> > related to some I/O blocking.
> >
> > I've only observed this problem when working against my
> server, though
> > I spend only a small amount of time working with other servers. I
> haven't
> > been able to replicate the issue with Google or Sourceforge HTTPS
> servers. My
> > server is running SVN 1.6.6 on Apache 2.2.14 behind IIS via ISAPI
> Rewrite.
> >
> > I do not experience the issue with the Tigris.org 1.6.6 svn
> client nor
> > with TortoiseSVN 1.6.6 client. I always experience the problem with
> the 32-
> > bit and 64-bit SlikSVN clients.
> >
> > The problem can be easily replicated by requesting a list
> operation on
> > https://svn.jaraco.com/jaraco/python with the SlikSVN 1.6.6 client.
> > Obviously, the problem seems to be related to my server configuration
> as
> > well, but because the problem emerged with the latest release of the
> client,
> > and since the server works well with other clients, it makes the most
> sense
> > to determine what changed with the client.
> >
> > The SlikSVN team has suggested I write this list to see if you
> have
> > any suggestions as to what might be causing the stalls. I suspect the
> problem
> > has something to do with the way that the I/O is handled. The problem
> emerged
> > with 1.6.6 and goes away if I roll back to SlikSVN 1.6.5.
> >
> > Was there any change in the Subversion 1.6.5-1.6.6 changeset
> that
> > might affect the I/O behavior as described?
> >
> >                 Do you have any other suggestions for troubleshooting
> this
> > issue?
>
> That is ... weird. Especially the fact that you only get the issue with
> SlikSVN 1.6.6 (and not with other windows binaries nor with other
> versions of SlikSVN), and only in combination with your server (and not
> with other (public) https servers).
>
> I don't have time to reproduce or investigate myself, but just some
> ideas:
>
> - Is there anything that you can think of that makes a difference
> between your server and google or sourceforge servers? I'm thinking of
> things like http compression (use of DEFLATE filters in your apache
> config, and of http-compression in your client-side configuration), and
> proxy servers (any of those connections going through a proxy, as
> opposed to the others?). Also, maybe you're accessing your own server
> through another (internal) network route than google or sourceforge?
> Maybe there's a difference in proxies, firewalls, ...?

The SSL is handled by IIS, and the results are "reverse" proxied to Apache
via ISAPI-Rewrite. This is very likely another factor driving the issue. I
have not yet had the luxury to disable SSL and determine if the problem is
related to SSL. I don't have the luxury of configuring Apache as the
front-end server for two reasons: First, I need to maintain the server on
one IP address and have other IIS applications that must also be served from
that address. Second, it's much easier to maintain one front-end SSL server
(IIS) versus having to keep two servers maintained.

The problem does not appear to be related to firewalls or other local
phenomena. The problem occurs across the Internet the same as locally. It
occurs whether firewalls are present or not. No proxies, other than the one
used to reverse-proxy the IIS front-end requests to the back-end Apache
server are used.

> - Any specific configuration in your client-side "servers"
> configuration file?

No. This happens with a clean install.

> - Can you also reproduce this on a 32-bit Windows system?

Yes. But again, only with SlikSVN.

> - Are you using neon or serf for any of those connections (default in
> 1.6.x is still neon, I believe, but you can configure serf in your
> "servers" file, or by command line option)?

It's using the default.

> - Since it only happens with SlikSVN 1.6.6, and not with the tigris.org
> 1.6.6 svn client, it must have something to do with differences in how
> they were built (both clients do pick up the same client-side config,
> don't they?). Dependencies on other libraries come to mind: maybe they
> were built against a different version of neon, serf, zlib, apr-util,
> ... ? I don't know how you can check that, maybe you'll have to ask the
> people who made those binaries ...

Yes, both clients use the same config.

> - There might also be other slight differences in the way SlikSVN is
> built as opposed to the tigris.org client. Maybe they use another
> compiler version, another Microsoft SDK, other compiler options, ...
> but that's very difficult to investigate (there can be lots of such
> small differences).

I'd even be happy if I rebuilt it and the problem went away. Unfortunately,
I don't think SlikSVN is willing to share their build process with me. I may
end up having to make my own build for 64-bit Windows clients.

> If all else fails, I think you'll have to try and troubleshoot this
> with some low-level Windows tools, things like sysinternals
> (http://technet.microsoft.com/en-us/sysinternals/default.aspx), and the
> like ... but then you may be in for a long ride.

Thanks for the suggestions. I think I'll disable SSL on the server (or
enable plaintext HTTP rather) and use wireshark to monitor the traffic.
Perhaps that will elucidate something. I'm familiar with the SysInternals
tools, so I'll give those a go also.

Thanks for Subversion and Regards,
Jason

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2426312

Please start new threads on the <users_at_subversion.apache.org> mailing list.
To subscribe to the new list, send an empty e-mail to <users-subscribe_at_subversion.apache.org>.