Re: notification output bottleneck

On Wed, Aug 29, 2012 at 2:00 AM, Johan Corveleyn <jcorvel_at_gmail.com> wrote:
> On Mon, Aug 27, 2012 at 10:45 AM, Johan Corveleyn <jcorvel_at_gmail.com> wrote:
>> On Mon, Aug 27, 2012 at 2:09 AM, Bert Huijben <bert_at_qqmail.nl> wrote:
>>>> -----Original Message-----
>>>> From: Johan Corveleyn [mailto:jcorvel_at_gmail.com]
>>>> Sent: maandag 27 augustus 2012 00:39
>>>> To: Subversion Development
>>>> Subject: notification output bottleneck
>>>>
>>>> I've never noticed this before (slow server, but now I'm testing a new,
>>> faster
>>>> server), but it seems that the writing of notification output on stdout is
>>> a
>>>> bottleneck for checkout, update or export on Windows (cmd.exe). With a
>>>> fast server, hot caches, and everything on a Gb LAN, checkout of a large
>>>> working copy is twice as fast (from 79 minutes down to 35 minutes) when I
>>>> redirect output to NUL. I tested with both 1.7.5 (SlikSVN) and trunk, the
>>>> results were about the same.
>>>>
>>>> Is there anything that could be done in svn to remove that bottleneck
>>>> (buffering, ...)?
>>>>
>>>> (there might have been some discussion about this before, but I can't find
>>> it)
>>>
>>> I found this problem too while profiling.
>>>
>>> The console apis on Windows in the Microsoft libraries are slow because they
>>> try to be safe for multithreaded applications. There are some defines that
>>> can be used to speed this up (which will avoid the most costly thread
>>> synchronisations), but we can't just apply these to our code as that would
>>> make libsvn_subr no longer multithread safe.
>>
>> Interesting.
>>
>> Am I correct in saying that this is really only a problem for the
>> commandline 'svn' client (and perhaps 'svnadmin' and 'svnlook' etc
>> ...)? Other clients can do their own fast notification handling,
>> right?
>>
>> And Isn't 'svn' always single-threaded?
>>
>> In that case: couldn't we make the necessary multithread-unsafe, but
>> fast, output functions for 'svn' only?
>
> Is the above idea possible / sane / desirable?
>
> Bert, what are those defines to speed up console access at the expense
> of thread safety? I'd like to give them a quick try just to see how
> much it would impact.
>
> ISTR from before we migrated to SVN that CVS did not have this
> problem: console output would just fly by very quickly. At least
> that's how I remember it :-), I haven't run CVS for some time now
> (it's also possible that I am misremembering, or that it is because I
> was running the client on WinXP back then).
>
One difference that Subversion uses direct unbuffered WriteFile to
write to stdout instead
of buffered printf()/puts(). You may try to tweak
svn_stream_for_stdout() to return buffered stream and check
performance difference.

Other possible reason that Subversion converts all texts from utf8 to
console encoding before writing text to stdout.

-- 
Ivan Zhakov