[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: Severe performance issues with large directories

From: Greg Stein <gstein_at_gmail.com>
Date: Fri, 9 Apr 2010 15:59:30 -0400

On Fri, Apr 9, 2010 at 08:27, Bert Huijben <bert_at_qqmail.nl> wrote:
>> -----Original Message-----
>> From: Paul Holden [mailto:paul.holden_at_gmail.com]
>...
>> 2) subversion appears to generate a temporary file in .svn\prop-base\ for
>> every file that's being updated. It's generating filenames sequentially,
>> which means that when 5,800 files are being updated it ends up doing this:
>>
>>
>> file_open tempfile.tmp? Already exists!
>>
>> file_open tempfile.2.tmp? Already exists!
>>
>> file_open tempfile.3.tmp? Already exists!
>>
>> ...some time later
>>
>> file_open tempfile.5800.tmp? Yes!
>
> Wow.

Yeah, Wow. And I'll raise you an Ugh.

> Are you sure that this is in prop-base, not .svn/tmp?

Whatever. Does it matter? :-P

I think we should backport the changes we made in svn_io_open_unique_file3.

> For 1.7 we made the tempfilename generator better in guessing new names, but
> for property handling we won't be using files in 1.7. (Looking at these
> numbers and those that follow later in your mail, we might have to look in
> porting some of this back to 1.6).

Yah.

>...
>> Is there any inherent reason these files need to be generated
> sequentially?
>> From reading the comments in 'svn_io_open_uniquely_named' it sounds like
>> these files are named sequentially for the benefit of people looking at
>> conflicts in their working directory. As these files are being generated
>> within the 'magic' .svn folder, is there any reason to number them
>> sequentially? Just calling rand() until there were no collisions would
>> probably give a huge increase in performance.
>
> In 1.7 we have a new api that uses a smarter algorithm, but we can't add
> public apis to 1.6 now.

Not a new API. Just a revamp of svn_io_open_unique_file3. Compare the
1.6.x version against trunk.

Paul: if you're adventurous, and build your own svn, you could try
lifting the contents of trunk's svn_io_open_unique_file3, and dropping
that into your 1.6.x.

>...

Cheers,
-g
Received on 2010-04-09 22:00:00 CEST

This is an archived mail posted to the Subversion Dev mailing list.