[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: performance issue with large number of files

From: Mark Weaver <mark-clist_at_npsl.co.uk>
Date: 2006-08-04 16:49:17 CEST

Garrett Rooney wrote:
> On 8/4/06, Mark Weaver <mark-clist@npsl.co.uk> wrote:
>> I've got a directory with about 12k files totally about 1gb that is
>> checked into svn. A svn ci in this folder is excruciatingly slow.
>> Partly this is size (~2gb of read, presumably diffing the files), but
>> mostly this is rewriting the entries file (about 3.2Mb written for each
>> file diffed => ~40gb of writes). Client version is reported as svn,
>> version 1.3.2 (r19776).
>>
>> Is there anything that can be done to mitigate this? e.g. get client to
>> check on file modification time rather than diffing; and/or dealing with
>> "entries" in a more scalable fashion? I note it doesn't read the
>> entries file each time, is the write per file really needed?
>
> Uhh, it does use file modification time, if the mtime hasn't changed
> it shouldn't diff the file. Not sure if dealing with the entries file
> can be improved or not, it's always possible though.
>
Yes, you are correct -- my mistake. The directory had been copied onto
a different drive, re-running svn ci is much happier now.

> Honestly though, perhaps having a directory that contains 12k files in
> it isn't the best idea in the world... Splitting it up would almost
> certainly improve things.
>
I agree, but I don't have any control over the file structure in this
instance.

Thanks for the help,

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Received on Fri Aug 4 16:50:52 2006

This is an archived mail posted to the Subversion Users mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.