> -----Original Message-----
> From: Ivan Zhakov [mailto:ivan_at_visualsvn.com]
> Sent: woensdag 29 augustus 2012 00:47
> To: Johan Corveleyn
> Cc: Bert Huijben; Subversion Development
> Subject: Re: notification output bottleneck
>
> On Wed, Aug 29, 2012 at 2:00 AM, Johan Corveleyn <jcorvel_at_gmail.com>
> wrote:
> > On Mon, Aug 27, 2012 at 10:45 AM, Johan Corveleyn <jcorvel_at_gmail.com>
> wrote:
> >> On Mon, Aug 27, 2012 at 2:09 AM, Bert Huijben <bert_at_qqmail.nl> wrote:
> >>>> -----Original Message-----
> >>>> From: Johan Corveleyn [mailto:jcorvel_at_gmail.com]
> >>>> Sent: maandag 27 augustus 2012 00:39
> >>>> To: Subversion Development
> >>>> Subject: notification output bottleneck
> >>>>
> >>>> I've never noticed this before (slow server, but now I'm testing a
> >>>> new,
> >>> faster
> >>>> server), but it seems that the writing of notification output on
> >>>> stdout is
> >>> a
> >>>> bottleneck for checkout, update or export on Windows (cmd.exe).
> >>>> With a fast server, hot caches, and everything on a Gb LAN,
> >>>> checkout of a large working copy is twice as fast (from 79 minutes
> >>>> down to 35 minutes) when I redirect output to NUL. I tested with
> >>>> both 1.7.5 (SlikSVN) and trunk, the results were about the same.
> >>>>
> >>>> Is there anything that could be done in svn to remove that
> >>>> bottleneck (buffering, ...)?
> >>>>
> >>>> (there might have been some discussion about this before, but I
> >>>> can't find
> >>> it)
> >>>
> >>> I found this problem too while profiling.
> >>>
> >>> The console apis on Windows in the Microsoft libraries are slow
> >>> because they try to be safe for multithreaded applications. There
> >>> are some defines that can be used to speed this up (which will avoid
> >>> the most costly thread synchronisations), but we can't just apply
> >>> these to our code as that would make libsvn_subr no longer multithread
> safe.
> >>
> >> Interesting.
> >>
> >> Am I correct in saying that this is really only a problem for the
> >> commandline 'svn' client (and perhaps 'svnadmin' and 'svnlook' etc
> >> ...)? Other clients can do their own fast notification handling,
> >> right?
> >>
> >> And Isn't 'svn' always single-threaded?
> >>
> >> In that case: couldn't we make the necessary multithread-unsafe, but
> >> fast, output functions for 'svn' only?
> >
> > Is the above idea possible / sane / desirable?
> >
> > Bert, what are those defines to speed up console access at the expense
> > of thread safety? I'd like to give them a quick try just to see how
> > much it would impact.
> >
> > ISTR from before we migrated to SVN that CVS did not have this
> > problem: console output would just fly by very quickly. At least
> > that's how I remember it :-), I haven't run CVS for some time now
> > (it's also possible that I am misremembering, or that it is because I
> > was running the client on WinXP back then).
> >
> One difference that Subversion uses direct unbuffered WriteFile to write to
> stdout instead of buffered printf()/puts(). You may try to tweak
> svn_stream_for_stdout() to return buffered stream and check performance
> difference.
>
> Other possible reason that Subversion converts all texts from utf8 to console
> encoding before writing text to stdout.
The time isn't spend in Subversion, but in the thread synchronisation in the CRT library.
I'm not sure when you tested CVS, but mostly everything became much faster over the last few years, while the console subsystem didn't and relatively slowed down a bit. Thread synchronization is much cheaper when you have a single non-hyperthreaded CPU, compared to the current processors.
Microsoft didn't really bother optimizing this as for most Windows applications console IO is never a bottleneck. Outputting to a console much faster than a user can read is fast enough. 10 times faster than unreadable fast has no additional profit.
Visual C++ 6.0 did have a single threaded C++ library as a link time option. This library was removed some 8 or 10 years ago. After that they have the defines as a replacement for some key scenarios. (I don't know a pointer to look for them in the documentation, but they are documented in the IO library on MSDN. You can also check the defines in the header files.)
Bert
>
> --
> Ivan Zhakov
Received on 2012-08-29 00:59:11 CEST