[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

RE: binary file size limit? (4.2GB retry - success)

From: David McBride <dmcbride_at_languageweaver.com>
Date: 2004-03-11 23:52:18 CET

C.A.T.

Thanks for contributing some REAL data, versus the first few responses that
wasted bandwidth espousing the obvious.

I'm trying to understand the differences in response times for adds vs.
commits. I understand that there is a large penalty when adding large
files, but what is the response time for committing relatively small changes
(e.g., < 20%) to that file? For example, I think that you said that it took
10 minutes to add your 700 MB file. How long does it take to commit a small
change to that file?

The reason I ask is that svn commit times are supposed to be relative to the
size of the change and NOT the size of the file. Therefore, committing a
small change should take only a fraction of the 10 minutes it took to add
the file in the first place. Thanks again.

v/r
 
- David
 

-----Original Message-----
From: C.A.T.Magic [mailto:c.a.t.magic@gmx.at]
Sent: Thursday, March 11, 2004 1:43 PM
To: users@subversion.tigris.org
Subject: Re: binary file size limit? (4.2GB retry - success)

Hi,

I just retried my Test with a 4.12GB sized
single file on a completely fresh repos and a 30GB
empty harddisk (using file:/// access to repos)
(svn is 1.0.0 win32)

>svn create X:\SVNSandbox\Repos
>cd X:\SVNSandbox/Work
> [add a few small dummy files]
> [created some "huge4GB.bin" test file with 4,430,187,696 bytes ]
>svn add huge4GB.bin

>X:\SVNSandbox\Work>svn commit -m "huge4"
   Adding (bin) huge4GB.bin
   Transmitting file data .

The file gets duplicated in the WC, then
I can see the db/strings file grow to about 4.4GB.
- About 10 minutes passed until now.
Then a long time period goes by where
"nothing really happens". CPU usage stays around 6%,
memory usage remains 7MB, disk usage remains at
a mimimum. the files in the db/ folder remain unchanged
- log.0000004327 is the last file. by looking at the
db/* modify dates I can see that the whole db data
remains unchanged.

While the svn commit is still busy "doing something
or nothing?", I can already run
X:\>svn ls file:///x:/SVNSandbox/Repos
  huge4GB.bin
- so, the file is already in the db!
I can even browse the repository and checkout small files(!)
while the 'commit' still 'hangs around'...

whew, finally the commit finished! (about 50 minutes now)
   Committed revision 2.
wow. +5 points for svn on handling huge files in a finite
amount of time ( well, committing it .. no merging yet :-)

next, I'm trying svn update
X:\SVNSandbox\Work>svn checkout file:///X:/SVNSandbox/Repos
now, a "huge4GB.bin.tmp.tmp" (.tmp twice? hmm )
gets downloaded into the .svn/tmp/ folder...
gets copied around a few times, but in the end - its where it belongs :-)

I guess most of the time was spent in the DB doing some "cleanup" work?
because the 4GB upload took just about 10 minutes, I could already
ls and checkout files, but it still took about 40 minutes until the
commit command finally completed.

[ I wish there was already a "filesystem" that supports "cheap copies" -
because then svn could copy and duplicate files around all day long
and it would be O(1) independant of the filesize and would not consume
disk space... ( no, I don't mean soft or hardlinks ) ]

svn is at least 'designed' to handle
all the "huge files of tomorrow". good work :-)

:-)
c.a.t.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Received on Thu Mar 11 23:52:50 2004

This is an archived mail posted to the Subversion Users mailing list.