[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

RE: large number of large binary files in subversion

From: Bob Archer <Bob.Archer_at_amsi.com>
Date: Mon, 23 May 2011 13:54:59 -0400

> I keep my personal photos and videos in Subversion.
> Total repository size is currently 475 GB. There are
> some large (1-4 GB) video files. Everything seems to
> work fine.
>
> Some details: I'm still running Subversion 1.4.6 on the
> server (FSFS, CentOS 5, x86_64). Running 1.6.16
> cmdline client on CentOS and latest TortoiseSvn on
> Windows XP (32 bit). I should try upgrading the server
> at some point, but it does make me nervous, requires
> a lot of space for backups etc.
>
> The most annoying thing is probably the lack of true
> moves and renames -- it takes a long time to move a
> large file because it doesn't just move it on the client
> side, it deletes it and recopies the whole thing over
> the network (IIUC). I basically never modify any large
> files, I just move them around and rename them.

You should be able to do a server side move... if you look at the help for mv command you will see:

SRC and DST can both be working copy (WC) paths or URLs:
   WC -> WC: move and schedule for addition (with history)
   URL -> URL: complete server-side rename.
All the SRCs must be of the same type.

So you can do something like:

svn mv http://myserver/repo/MyFile.jpg http://myserver/repo/newPath/MyNewFile.jpg

I can't imagine that this would need to move the file over the network since it all happens server side. Although, I guess when you do an update it might bring the file down again rather than doing a local move/rename.

BOb

>
> Overall I'm quite happy using Subversion for this.
>
> (I should try to make an effort to test 1.7, since it's
> an interesting and somewhat unusual workload...)
>
> HTH,
> -Ed
>
> PS TortoiseSvn working copy is 950 GB, 351K files,
> 30K folders, according to Windows "Properties".
>
>
> On Mon, May 23, 2011 at 7:36 AM, Julian Foad
> <julian.foad_at_wandisco.com> wrote:
> > On Fri, 2011-05-20, Winston Smith wrote:
> >> Sorry if this is the wrong list, but I'm curious about one
> thing:
> >> Are the SVN developers aware of any quirks in SVN in regards
> >> to storing a large number (say, 1000) very large binary files
> >> (say, 1GB each)? So, the entire repository would be 1TB of size,
> >> but my concern is not space, but rather whether SVN would have
> >> difficulties (either on the server side or the client side)
> handling
> >> such repositories/workspaces. Thanks for your replies.
> >
> > I can't give a very definite answer, but, assuming your server
> and
> > network and client hardware is adequately sized for the task, I
> would
> > expect:
> >
> >  * No problem handling many thousands of files in total.
> >
> >  * Putting 1000 or more files all in the same directory can give
> poor
> > performance in time and/or space, on both client and server, so
> avoid
> > doing that.
> >
> >  * Subversion can handle "binary" files as large as 1 GB or even
> many
> > GB.  There used to be a 2 GB limit when using old versions of
> Subversion
> > in certain configurations, but that limit is long gone.  I have
> heard
> > that the server can be very slow if a new version of a very large
> file
> > consists of a completely different bit pattern from the previous
> > version.  If your files don't change much, or if you only add new
> files
> > and delete old files instead of checking in modifications to the
> > existing files, that won't be an issue.
> >
> >  * Subversion is designed to process large amounts of data
> "streamily"
> > without trying to read it all into RAM at once, so you shouldn't
> need
> > excessive amounts of RAM.
> >
> > I hope that helps.  Please let us know what results you get.  If
> you do
> > run into any problem we'd like to know about it and try to fix
> it.
> >
> > - Julian
> >
> >
> >
Received on 2011-05-23 19:55:30 CEST

This is an archived mail posted to the Subversion Dev mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.