On Mon, 7 Oct 2019, 19:47 Doug Robinson, <doug.robinson_at_wandisco.com> wrote:
> Folks:
>
> I spoke with this user late last week. They stated that they can only get
> approximately 400 parallel SVN operations before the "system time" consumes
> all available CPU for an 8-core machine. Adding more cores won't help
> because of the nature of spin locks (it makes things worse). Turns out
> that even with ~100 parallel SVN operations the "system time" starts
> becoming significant/measurable (~10%). Both HTTP (mod_dav_svn) and
> "svnserve" protocols participate in the lock contention.
>
> Your help would be greatly appreciated.
>
Whew. So. Reducing this issue to "use a more efficient lock" is not going
to work, and you provided far too little information to even attempt a
diagnosis. For starters, I recommend gathering as much info as possible
(anonymised of course) about the server configuration, everything from
httpd an svnserve to the repository config and underlying filesystem, if
possible. Getting stack traces of the "stuck" threads would be necessary,
too. Without knowing exactly what is happening, these kinds of problems are
extremely hard to understand, let alone fix.
I'd be surprised if the spinlock is the actual culprit. AFAIK, kernel-level
locks hand off to the scheduler if they spin too long; on multiprocessor
machines, this is usually more efficient than immediately yielding and
causing an expensive context switch. It's possible that you're seeing an
unfortunate timing "resonance" that might go away with either more *or*
less cores being available. The behaviour is really hard to predict.
-- Brane
> On Fri, Oct 4, 2019 at 9:20 AM Doug Robinson <doug.robinson_at_wandisco.com>
> wrote:
>
>> Folks:
>>
>> From a Subversion user:
>>
>> “... we have very high concurrent connections to Subversion that seem to
>> crater Subversion. The SVN Serve process we use to access the Subversion
>> repository is using the “svn” protocol by our “system user”, mostly
>> read-only. Then, we, on behalf of the user make request to Subversion
>> using the “http” protocol to fetch their data. So we have lots of
>> connections to Subversion. But the volume of concurrent requests over the
>> “svn” protocol cause the “svnserve” process to consume CPU cycles in a
>> kernel “mutex-lock” which is implemented using “spin locks”. The “svnserve”
>> process makes the mutex calls using the “apache” (APR) semaphore wait
>> calls, but on Linux this is a “mutext-lock” request.”
>>
>> So is there a better, more scalable, semaphore that can be used?
>>
>
Received on 2019-10-07 20:40:25 CEST