----- Original Message -----
From: "Garrett Rooney" <rooneg@electricjellyfish.net>
To: "Mark Weaver" <mark-clist@npsl.co.uk>
Cc: <users@subversion.tigris.org>
Sent: Friday, August 04, 2006 10:36 AM
Subject: Re: performance issue with large number of files
> On 8/4/06, Mark Weaver <mark-clist@npsl.co.uk> wrote:
>> I've got a directory with about 12k files totally about 1gb that is
>> checked into svn. A svn ci in this folder is excruciatingly slow.
>> Partly this is size (~2gb of read, presumably diffing the files), but
>> mostly this is rewriting the entries file (about 3.2Mb written for each
>> file diffed => ~40gb of writes). Client version is reported as svn,
>> version 1.3.2 (r19776).
>>
>> Is there anything that can be done to mitigate this? e.g. get client to
>> check on file modification time rather than diffing; and/or dealing with
>> "entries" in a more scalable fashion? I note it doesn't read the
>> entries file each time, is the write per file really needed?
>
> Uhh, it does use file modification time, if the mtime hasn't changed
> it shouldn't diff the file. Not sure if dealing with the entries file
> can be improved or not, it's always possible though.
>
> Honestly though, perhaps having a directory that contains 12k files in
> it isn't the best idea in the world... Splitting it up would almost
> certainly improve things.
Please tell me they're split into subdirectories, or not on a Linux system
using ext2 or ext3 with a 2.4 kernel: there were serious performance issues
for overcrowded directories in that setup, which the 2.6 kernel has pretty
much fixed. But doing file management of anything over a few thousand files
in one directory is usually grounds to think about how to break it down a
bit.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Received on Fri Aug 4 17:14:52 2006