rbraswell@connected.com writes:
> Thanks. Is this safe to run on a "hot" repository?
A repository that is in use? Yes, this recipe is meant to be used on
a live, active repository.
That's not the same as saying *this script* is meant to be used on a
live repository. This script's purpose is to document a recipe for
backups and restores of live repositories. However, It has not been
packaged and parameterized in such a way that you can just use it
as-is. You need to write new scripts based on the recipe shown here.
(If I'd had time to do that myself, they'd be in the Subversion tree,
and I wouldn't be posting this script to users@ every few weeks :-) ).
-Karl
> -----Original Message-----
> From: kfogel@collab.net [mailto:kfogel@collab.net]
> Sent: Thursday, May 13, 2004 3:34 PM
> To: Ryan Braswell
> Cc: brane@xbc.nu; sussman@collab.net; users@subversion.tigris.org
> Subject: Re: svnadmin hotcopy failing
>
>
> rbraswell@connected.com writes:
> > So what is the story here. Are the 3 steps Ben provided earlier unusable?
>
> Here is a script that demonstrates how to do Berkeley-safe backups of
> a Subversion repository. I keep hoping to find time to package it
> into something that can be put in contrib/ or tools/, but haven't yet,
> so instead I post it every now and then in threads like this.
>
> You'll have to piece together what's happening; it's commented, but
> you still need to read the code to understand it.
>
> --------------------8-<-------cut-here---------8-<-----------------------
> #!/bin/sh
>
> # Try some backup/restore procedures for Subversion repositories.
>
> # You might need to customize these paths.
> SVN=svn
> SVNADMIN=svnadmin
> SVNLOOK=svnlook
> DB_ARCHIVE=/usr/local/BerkeleyDB.4.2/bin/db_archive
> DB_RECOVER=/usr/local/BerkeleyDB.4.2/bin/db_recover
>
> # Any binary file that's about 64k is fine, doesn't have to be /bin/ls.
> DATA_BLOB=/bin/ls
>
> # You shouldn't need to customize below here.
> SANDBOX=`pwd`/backups-test-tmp
> FULL_BACKUPS=${SANDBOX}/full
> INCREMENTAL_PREFIX=${SANDBOX}/incremental-logs
> RECORDS=${SANDBOX}/records
> PROJ=myproj
> REPOS=${PROJ}-repos
>
> rm -rf ${SANDBOX}
> mkdir ${SANDBOX}
> mkdir ${RECORDS}
>
> cd ${SANDBOX}
>
> ${SVNADMIN} create --bdb-log-keep ${REPOS}
> ${SVN} co file://${SANDBOX}/${REPOS} wc
>
> cd wc
>
> # Put in enough data for us to exercise the logfiles.
> cp ${DATA_BLOB} ./a1
> cp ${DATA_BLOB} ./b1
> cp ${DATA_BLOB} ./c1
> ${SVN} -q add a1 b1 c1
> ${SVN} -q ci -m "Initial add."
>
> echo "Created test data."
>
> cd ..
>
> # Exercise the logfiles by moving data around a lot. Note that we
> # avoid adds-with-history, since those cause much less Berkeley
> # activity than plain adds.
> #
> # Call this from the parent of wc, that is, with $SANDBOX as CWD.
> # Pass one argument, a number, indicating how many cycles of exercise
> # you want. The more cycles, the more logfiles will be generated.
> # The ratio is about two cycles per logfile.
> function exercise
> {
> limit=${1}
>
> saved_cwd=`pwd`
> cd ${SANDBOX}/wc
>
> echo ""
> i=1
> while [ ${i} -le ${limit} ]; do
> mv a1 a2
> mv b1 b2
> mv c1 c2
> ${SVN} -q rm a1 b1 c1
> ${SVN} -q add a2 b2 c2
> ${SVN} -q ci -m "Move 1s to 2s, but not as cheap copies."
>
> mv a2 a1
> mv b2 b1
> mv c2 c1
> ${SVN} -q rm a2 b2 c2
> ${SVN} -q add a1 b1 c1
> ${SVN} -q ci -m "Move 2s back to 1s, same way."
>
> echo "Exercising repository, pass ${i} of ${limit}."
> i=`dc -e "${i} 1 + p"`
> done
> echo ""
>
> cd ${saved_cwd}
> }
>
> # Generate some logfile activity.
> exercise 10
>
> # Do a full backup.
> head=`${SVNLOOK} youngest ${REPOS}`
> echo "Starting full backup (at r${head})..."
> mkdir ${FULL_BACKUPS}
> mkdir ${FULL_BACKUPS}/${PROJ}
> mkdir ${FULL_BACKUPS}/${PROJ}/repos
> mkdir ${FULL_BACKUPS}/${PROJ}/logs
> cd ${REPOS}/db
> ${DB_ARCHIVE} > ${RECORDS}/${PROJ}-full-backup-inactive-logfiles
> cd ../..
> cp -a ${REPOS} ${FULL_BACKUPS}/${PROJ}/repos/
> cd ${REPOS}/db
> for logfile in `${DB_ARCHIVE} -l`; do
> # For maximum paranoia, we want repository activity *while* we're
> # making the full backup.
> exercise 5
> cp ${logfile} ${FULL_BACKUPS}/${PROJ}/logs
> done
> cat ${RECORDS}/${PROJ}-full-backup-inactive-logfiles | xargs rm -f
> cd ../..
> echo "Full backup completed (r${head} was head when started)."
>
> # Do the incremental backups for a nominal week.
> for day in 1 2 3 4 5 6; do
> exercise 5
> head=`${SVNLOOK} youngest ${REPOS}`
> echo "Starting incremental backup ${day} (at r${head})..."
> mkdir ${INCREMENTAL_PREFIX}-${day}
> mkdir ${INCREMENTAL_PREFIX}-${day}/${PROJ}
> cd ${REPOS}/db
> ${DB_ARCHIVE} > ${RECORDS}/${PROJ}-incr-backup-${day}-inactive-logfiles
> for logfile in `${DB_ARCHIVE} -l`; do
> # For maximum paranoia, we want repository activity *while* we're
> # making the incremental backup. But if we did commits with each
> # logfile copy, this script would be quite slow (Fibonacci effect).
> # So we only exercise on the last two "days" of incrementals.
> if [ ${day} -ge 5 ]; then
> exercise 3
> fi
> cp ${logfile} ${INCREMENTAL_PREFIX}-${day}/${PROJ}
> done
> cat ${RECORDS}/${PROJ}-incr-backup-${day}-inactive-logfiles | xargs rm -f
> cd ../..
> echo "Incremental backup ${day} done (r${head} was head when started)."
> done
>
> # The last revision a restoration is guaranteed to contain is whatever
> # was head at the start of the last incremental backup.
> last_guaranteed_rev=${head}
>
> # Make the repository vanish, so we can restore it.
> mv ${REPOS} was_${REPOS}
>
> echo ""
> echo "Oliver Cromwell has destroyed the repository! Restoration coming
> up..."
> echo ""
>
> # Restore.
> #
> # After copying the full repository backup over, we remove the shared
> # memory segments and the dav/* stuff. Recovery recreates the shmem
> # segments, and anything in dav/* is certainly obsolete if we're doing
> # a restore.
> #
> # Note that we use db_recover instead of 'svnadmin recover'. This is
> # because we want to pass the -c ('catastrophic') flag to db_recover.
> # As of Subversion 1.0.x, there is no '--catastrophic' flag to
> # 'svnadmin recover', unfortunately.
> cp -a ${FULL_BACKUPS}/${PROJ}/repos/${REPOS} .
> cp -a ${FULL_BACKUPS}/${PROJ}/logs/* ${REPOS}/db
> rm -rf ${REPOS}/db/__db*
> rm -rf ${REPOS}/dav/*
> cd ${REPOS}/db
> ${DB_RECOVER} -ce
> cd ../..
> head=`${SVNLOOK} youngest ${REPOS}`
> echo ""
> echo "(Restored from full backup to r${head}...)"
> for day in 1 2 3 4 5 6; do
> cd ${REPOS}/db
> cp ${INCREMENTAL_PREFIX}-${day}/${PROJ}/* .
> ${DB_RECOVER} -ce
> cd ../..
> head=`${SVNLOOK} youngest ${REPOS}`
> echo "(Restored from incremental-${day} to r${head}...)"
> done
> echo ""
> echo "Restoration complete. All hail the King."
>
> # Verify the restoration.
> was_head=`${SVNLOOK} youngest was_${REPOS}`
> restored_head=`${SVNLOOK} youngest ${REPOS}`
> echo ""
> echo "Highest revision in original repository: ${was_head}"
> echo "Highest revision restored: ${restored_head}"
> echo ""
> echo "(It's okay if restored is less than original, even much less.)"
>
> if [ ${restored_head} -lt ${last_guaranteed_rev} ]; then
> echo ""
> echo "Restoration failed because r${restored_head} is too low --"
> echo "should have restored to at least r${last_guaranteed_rev}."
> exit 1
> fi
>
> # Looks like we restored at least to the minimum required revision.
> # Let's do some spot checks, though.
>
> echo ""
> echo "Comparing logs up to r${restored_head} for both repositories..."
> ${SVN} log -v -r1:${restored_head} file://`pwd`/was_${REPOS} > a
> ${SVN} log -v -r1:${restored_head} file://`pwd`/${REPOS} > b
> if cmp a b; then
> echo "Done comparing logs."
> else
> echo "Log comparison failed -- restored repository is not right."
> exit 1
> fi
>
> echo ""
> echo "Comparing r${restored_head} exported trees from both repositories..."
> ${SVN} -q export -r${restored_head} file://`pwd`/was_${REPOS} orig-export
> ${SVN} -q export -r${restored_head} file://`pwd`/${REPOS} restored-export
> if diff -q -r orig-export restored-export; then
> echo "Done comparing r${restored_head} exported trees."
> else
> echo "Recursive diff failed -- restored repository is not right."
> fi
>
> echo ""
> echo "Done."
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Received on Fri May 14 17:25:02 2004