[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: Subversion on a large repository

From: Johnathan Gifford <jgifford_at_wernervas.com>
Date: Fri, 15 Feb 2008 16:05:25 -0600

Vishnu,

At one point the using NFS was considered a big 'no no'. However the Subversion book online (http://svnbook.red-bean.com/) doesn't seem to indicate that is an issue unless your using BerkleyDB. I do know that reasons for not using NFS with multiple daemons was because of the results you mention below. This was due to NFS lacking of low level file locking across multiple connected systems to the drive share.

Now, many Apple hardware users have has extremely successful luck using Xsan with server versions of Mac OS X in the same configuration you mention below with multiple daemons running. This is because of the low lever file locking that takes place with the Xsan protocols. I know that number of other vendors in the *nix world are indicating excellent results with ZFS where their OS support that protocol. So have your tried the same test(s) using ZFS?

Hope this helps,

Johnathan

>>> On Fri, Feb 15, 2008 at 11:34 AM, in message <47B5CD1A.3080906_at_oracle.com>,
Vishnu Venkatesan <vishnu.venkatesan_at_oracle.com> wrote:
> Hi Marc,
> Thanks for your response. By concurrent I meant 100-200 commits per
> second which could be the peak load on the repository. I did some simple
> tests and having concurrency issues. I hosted a repository on a NFS and
> had two svnserve daemons running in two different machines. When two
> concurrent commits were issued using the two different svnserve daemons,
> the repository is in a weird state and doesnt allow any more commits
> with an error as checksum mismatch.
>
> Steps:
> 1. Create a simple svn repo on a network filesystem using FSFS
> 2. Start svnserve daemon on two different machines which has access to
> the network file system repository
> 3. Checkout files from the svn repository using the two different
> svnserve daemons running in different machines.
> 4. commit files simultaneously using the two svnserve daemons
> (surprisingly both the commits resulted in the same revision number)
> 5. When I do update on the checkouts one of them is in an inconsistent
> state and doesnt allow further commits.
>
> Is running two svnserve daemons on a single repository is supported ? I
> am doing this just to make sure that in case if having one svnserve
> daemon's doesn't scale for 100-200 concurrent users per second I can
> have two machines handling the commits.
>
> thanks,
> Vishnu.
>
> Marc Haisenko wrote:
>> On Friday 15 February 2008, Vishnu Venkatesan wrote:
>>
>>> Hi,
>>> I am in the process of evaluating subversion for hosting a repository
>>> of about 10,000+ files with a size of 2 GB that can change at least 10%
>>> everyday.
>>> Can anyone please let me know if there are any references of subversion
>>> deployment with such large repository with FSFS or Berkeley DB.
>>> Some of the things that I am looking for
>>> 1. What would be the advantages of using one over the other (FSFS vs
>>> Berkeley DB)
>>> 2. Any specific limitations
>>> 3. How much scalable the repository would be -- for about 100-200
>>> concurrent commits
>>>
>>> thanks,
>>> Vishnu.
>>>
>>
>>
>> I think 2GB is now considered to be small to medium size... our repository
> is
>> 5GB, and the KDE, GNOME and GCC folks will propably laugh about such small
>> repositories ;-)
>>
>> Look at the testimonial site:
>> http://subversion.tigris.org/testimonials.html
>>
>> You will see that a lot of very large scale projects are listed there (KDE,
>> GNOME, GCC, Python, Samba, Mono), and they're all doing fine.
>>
>> But on to your questions:
>>
>> 1.: Don't know about that... most people now prefer FSFS, it seems.
>>
>> 2.: Maximum revision number is (AFAIK) 2^31 - 1, so you won't run into it for
>
>> next decades even if you commit every second. The next limitation that is of
>
>> importance is your filesystem: I wouldn't store the repository on a FAT16 ;-)
>
>> Choose a modern filesystem that can handle a lot of files and if you're
> going
>> with BDB you'll propably need one that can handle file sizes >2GB. AFAIK all
>> modern UNIX/Linux/Mac filesystems do, I have no idea about Windows but I
>> assume NTFS does as well. If you're using FSFS you'd propably want a FS that
>
>> can handle a lot of small files well (on Linux I'd say ReiserFS but several
>> people here might disagree... I think you better take what you're
> comfortable
>> with, as long as you don't intend to set up a repository that will be under
>> EXTREME load, by which I mean several commits per second).
>>
>> 3.: Don't know what you mean. Do you mean 100-200 concurrent commits per
>> second/minute/hour/day ? Or do you mean 100-200 people working on the same
>> file(s) concurrently ?
>>
>> HTH,
>> Marc
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_subversion.tigris.org
> For additional commands, e-mail: users-help_at_subversion.tigris.org

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe_at_subversion.tigris.org
For additional commands, e-mail: users-help_at_subversion.tigris.org
Received on 2008-02-15 23:06:31 CET

This is an archived mail posted to the Subversion Users mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.