[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

subversion and large files

From: Robert Buels <rmb32_at_cornell.edu>
Date: 2004-11-20 10:16:52 CET

Here where I work, we are looking into options for placing our
mission-critical database of biological data under some sort of version
control. It is all currently housed in MySQL MyISAM tables, the largest
of which has files a bit above 2GB in size, but we are expecting to have
a lot more data coming in as work progresses, making larger files.

I recently played around with importing the MySQL .MYD, .MYI, etc files
for one of our databases into an SVN repository and appeared to hit a
2GB limit somewhere in the SVN code:

---------------------------------------------
rob@toblerone:~$ svn import -m 'initial import' /home/mysql/sgn/
file:///home/rob/svn-db/sgn/
Adding (bin) /home/mysql/sgn/enzymes.MYD
Adding (bin) /home/mysql/sgn/loc_types.frm
Adding (bin) /home/mysql/sgn/enzymes.MYI
...
... <snip>
...
Adding (bin) /home/mysql/sgn/blast_targets.frm
Adding (bin) /home/mysql/sgn/authors.MYI
Adding (bin) /home/mysql/sgn/map_cross.MYI
svn: Can't check path '/home/mysql/sgn/est.MYD': Value too large for
defined data type
rob@toblerone:~$
--------------------------------------------

file sizes:

--------------------------------------------
rob@toblerone:/home/mysql/sgn$ ls -l est.*
-rw-rw-r-- 1 mysql users 2213099276 Jul 27 15:58 est.MYD
-rw-rw-r-- 1 mysql users 62985216 Jul 27 21:58 est.MYI
-rw-rw-r-- 1 mysql users 8826 Dec 4 2003 est.frm
--------------------------------------------

The test system I was running this on is Debian testing, the svn
installation is from Debian's 'subversion' package, with version
information:

svn, version 1.0.9 (r11378)
   compiled Oct 16 2004, 23:29:24

So the questions I have here are:
1.) What's the state of >2GB support in Subversion both now and as
development progresses?
2.) While I have your attention here, do you have any better suggestions
for us to put our (eventually around 50GB) database under version
control, whether just on the level of binary files or even better
keeping SQL diffs as data changes? I guess we'd even consider buying
proprietary software.

Thank you all.

Rob

-- 
Robert Buels
SGN Bioinformatics Analyst 
252A Emerson Hall, Cornell University
Ithaca, NY  14850
Tel: 607-255-2360
rmb32@cornell.edu
http://www.sgn.cornell.edu
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Received on Sat Nov 20 15:20:26 2004

This is an archived mail posted to the Subversion Users mailing list.