[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: cvs2svn.py - converting large repository

From: Daniel Berlin <dberlin_at_dberlin.org>
Date: 2002-11-08 20:35:40 CET

Note the ouch's below.

Try adding "set_cachesize 0 64000000 1" to the DB_CONFIG in the db dir of
the repo, and see if it makes it go faster.

On Fri, 8 Nov 2002, Marko Macek wrote:

> Daniel Berlin wrote:
>
> > db_stat -m the resulting database and send the output to me (or the
> > mailing list).
> > This will tell us whether it's subversion, or whether you need to tune
> > the various DB_CONFIG parameters (like cache size).
>
> These are the new statistics with a svn just a few days old and with
> my latest cvs2svn.
>
> $ db_stat -m
> 257KB 768B Total cache size (263936 bytes).
> 1 Number of caches.
> 270336 Pool individual cache size.
> 59M Requested pages found in the cache (74%).

                        ^^^^^
        Ouch!

> 0 Requested pages mapped into the process' address space.
> 21M Requested pages not found in the cache.
                ^^^^^^^^^^^^^
                Ouch

> 161113 Pages created in the cache.
> 21M Pages read into the cache.
> 4150615 Pages written from the cache to the backing file.
> 18M Clean pages forced from the cache.
        ^^^^^^^^^^^^^
                Ouch

> 3923095 Dirty pages forced from the cache.
> 0 Dirty buffers written by trickle-sync thread.
> 64 Current clean buffer count.
> 0 Current dirty buffer count.
> 67 Number of hash buckets used for page location.
> 92M Total number of times hash chains searched for a page.
> 9 The longest hash chain searched for a page.
> 137M Total number of hash buckets examined for page location.
> 236M The number of region locks granted without waiting.
> 2 The number of region locks granted after waiting.
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Pool File: strings
> 4096 Page size.
> 44M Requested pages found in the cache (69%).
                        ^^^^^^^^^^^^
                        Ouch

> 0 Requested pages mapped into the process' address space.
> 20M Requested pages not found in the cache.
> 155254 Pages created in the cache.
> 20M Pages read into the cache.
> 4004768 Pages written from the cache to the backing file.
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Pool File: representations
> 4096 Page size.
> 14M Requested pages found in the cache (95%).
> 0 Requested pages mapped into the process' address space.
> 758317 Requested pages not found in the cache.
> 5859 Pages created in the cache.
> 758316 Pages read into the cache.
> 145847 Pages written from the cache to the backing file.
>
>
> Hmm, there are less pool files than before...
>
> I still get this error after the conversion:
>
> $ svn log file:///repo/svn/emacs
> svn: Berkeley DB error
> svn: Berkeley DB error while reading node revision for filesystem
> /repo/svn/emacs/db:
> Cannot allocate memory
>
> $ svnadmin youngest /repo/svn/emacs
> 49869
>
> And I also got this error at the end conversion during a
> conversion of a tag (only fs.copy is part of the commit):
>
>
> tag gerd_dbe to /branches/gerd_dbe/lisp/ChangeLog from
> /trunk/lisp/ChangeLog revision 39204
> Traceback (most recent call last):
> File "/home/mark/svnwork/final-3/cvs2svn.py", line 947, in ?
> main()
> File "/home/mark/svnwork/final-3/cvs2svn.py", line 944, in main
> util.run_app(convert, ctx, start_pass=start_pass)
> File
> "/root/rpms/tmp/subversion-2002110320-0/usr/lib/python2.2/site-packages/svn/util.py",
> line 38, in run_app
> File "/home/mark/svnwork/final-3/cvs2svn.py", line 874, in convert
> _passes[i](ctx)
> File "/home/mark/svnwork/final-3/cvs2svn.py", line 744, in pass5
> do_copy(ctx, t_fs, 1, branch_copies, copy_from, copies_done,
> branches_done)
> File "/home/mark/svnwork/final-3/cvs2svn.py", line 794, in do_copy
> conflicts, new_rev = fs.commit_txn(txn)
> RuntimeError: Commit succeeded, deltification failed
>
> The conversion took (with fsync disabled):
>
> real 428m18.067s
> user 397m43.871s
> sys 18m39.643s
> Fri Nov 8 14:45:21 CET 2002
>
> Mark
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Fri Nov 8 20:37:14 2002

This is an archived mail posted to the Subversion Dev mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.