On Fri, Jun 5, 2009 at 15:02, Andy Levy <andy.levy_at_gmail.com> wrote:
> On Fri, Jun 5, 2009 at 14:40, Geoff Rowell <geoff.rowell_at_varolii.com>
wrote:
> > I thought I'd share some difficulties that I encountered when
getting a
> > working copy from a repository with a large number of files.
> >
> >
> >
> > Background
> >
> > --------------
> >
> > The repository is running on a Windows 2003 unit. It's using
CollabNet
> > Subversion Server 1.5.6 (Apache 2.2). It has a small number of
revisions
> > (~130).
> >
> > The repository contains a large number of small audio recordings. In
several
> > cases there are over 5000 audio files in a folder.
> >
> > My laptop is a Windows Vista system using CollabNet Subversion
Client 1.6.2.
> >
> >
> >
> > Working copy details:
> >
> > - Size = 1.86 GB
> >
> > - Size on disk = 3.19 GB
> >
> > - # files = 480,000
> >
> > - # folders = 860
> >
> >
> >
> > Problem
> >
> > ----------
> >
> > I needed to get a working copy on my laptop in order to add/update
some
> > audio files. Since I knew this would be a performance challenge, I
suspended
> > the TSVNCache process and attempted the checkout from the command
line.
> > While it initially flew right along, eventually it slowed down to
the point
> > where every file was taking about 30 seconds. I could tell from the
reported
> > pathnames that the checkout was going to take days (weeks?).
> >
> >
> >
> > I decided to do some experiments to refine where the problem was.
> >
> >
> >
> > I have an Ubuntu Linux box running Subversion 1.5.5. I used that to
get a
> > working copy. It flew along at a steady rate and finished in 40
minutes. I
> > archived the Linux working copy, transferred it to my laptop and
unpacked
> > it. If the Windows file system had been a bottleneck, the unpack
would have
> > shown a performance problem. It didn't. It unpacked it at a steady
rate in a
> > couple of hours.
> >
> >
> >
> > I wanted to determine if the problem would occur when using a
Subversion 1.5
> > client under Windows, but I immediately ran into some problem masked
by
> > issue #3102. That prevented me from investigating this further.
> >
> >
> >
> > Luckily, I have the perfectly usable working copy, transferred from
my Linux
> > box, that can be used for modifications. My typical commits of a few
files
> > at a time don't suffer performance problems.
> >
> >
> >
> > Thought I'd share this experience in hopes that something could be
done to
> > improve the performance.
>
> There probably isn't much that can be done. NTFS and Windows do not
> handle large numbers of files/directories in a single directory very
> well, period. It's not so much a Subversion/library issue as it is
> poor filesystem performance on that scale.
>
> Did you disable your on-access virus scanner while doing this
> checkout? That's usually a real killer.
>
That was my first thought, but when the unpack (7-Zip) of the working
copy
archive didn't experience this problem I considered it to be unlikely.
---
Geoff Rowell | SET Architect & SCC Administrator
Varolii Corporation | 781.482.2125 | FAX 781.482.2199 |
geoff.rowell_at_varolii.com
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender.
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2359808
To unsubscribe from this discussion, e-mail: [users-unsubscribe_at_subversion.tigris.org].
Received on 2009-06-05 21:10:16 CEST