[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: Checkout really slow in Windows with lots of files in one directory

From: Mark Phippard <markphip_at_gmail.com>
Date: Wed, 2 Feb 2011 07:52:23 -0500

On Wed, Feb 2, 2011 at 7:41 AM, Geoff Rowell <geoff.rowell_at_gmail.com> wrote:
> On Wed, Feb 2, 2011 at 4:09 AM, Nick <nospam_at_codesniffer.com> wrote:
>> On Tue, 2011-02-01 at 13:00 -0500, Mark Phippard wrote:
>> On Wed, Jan 26, 2011 at 9:28 AM, Neil Bird <neil_at_jibbyjobby.co.uk> wrote:
>>>  We have a graphics-oriented code-base that's auto-generated and has >5000
>>> source files in one directory.  While I can check this out OK on Linux,
>>> we're seeing an unusable slow-down on Windows XP (NTFS), both using
>>> Tortoise
>>> directly, and as a test on Linux with the Windows drive mapped over CIFS.
>> I created a folder with 5001 files in it ... maybe that is not enough?
>>  I just used small simple text files as I was only checking for the
>> general problem in managing the temp files and the WC metadata.
>> Upon checkout (using 1.6.15 command line client) I did not notice any
>> slowdown.  Windows checked out via HTTP across internet in about 49
>> seconds as opposed to 33 from my Mac (which is a faster system).  The
>> main thing is checkout did not seem to slow down.
>> I did a similar test, using 5100 files in a single directory.  Each file
>> contained only the content "file XXXX" where XXXX was the number of the file
>> (so tiny files).  My linux system took 17 seconds, while Windows took a bit
>> less than 2 min (but Windows is virtualized while linux is on the
>> hardware).  I also did not notice a slow-down as the checkout proceeded.
>> Both systems used 1.6.15 and accessed the repo via https.
>> I did, however, notice that the time to *add* the files (done via svn add
>> *.txt) seemed to progressively slow down.  But this was only observed by
>> watching the files in the console as they were being added (it was
>> relatively easy to see the rate because the each file name had a linear
>> number at the end).  I don't have any timings to back this up, though I'll
>> collect some if anyone's interested.
> I don't know why, but I believe the key thing here is working with
> *binary* files.
> I noticed the same problem with a massive (10K+) amount of audio
> snippets in a single directory.

I was thinking that this was a case where the reading/parsing/writing
of our large entries file was causing a slowdown and moving to SQLite
was going to bring performance gains. Clearly that is not the case as
trunk is much slower.

If I get another batch of free time I will try it with a lot of small PNG's.

Mark Phippard
Received on 2011-02-02 13:53:00 CET

This is an archived mail posted to the Subversion Users mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.