[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Repeated "unrecoverable" BDB problems (ver 1.1.3)

From: Zachary Pincus <zpincus_at_stanford.edu>
Date: 2005-06-28 00:29:58 CEST

Hello,

I'm running Subversion server (version 1.1.3 (r12730) compiled Feb 22
2005, 17:54:37 ) on Mac OS X 10.3.9. I use svn+ssh as a client protocol
to communicate with the server; I am the only user of this repository
so permissions issues are properly sorted.

We've had a couple of power failures in my building (!), and each one
broke a subversion repository on the server machine in a similar way.
Specifically, the repository was broken such that svnadmin recover
could not fix it (see step 1 below for output text). However, by
deleting the logfiles from the repository/db directory (a last ditch
measure!), svnadmin recover worked and then svnadmin verify waschecked
everything out as OK, and clients were able to use read-only commands
like update and diff (see steps 2,3 below). However, the next client
commit broke the repository again. This time, svnadmin recover failed
and attempts at running svnadmin verify resulted in a
seemingly-infinite loop of errors (step 4 below).

Fortunately, I had made a dump of the repository when it was working
for read-only purposes, so I was able to restore all the revisions.
Unfortunately, I don't have a copy of the svn repository in it's
original, wedged state, just some copies made after it got further
wedged by some failed runs of svnadnin recover. So I can't try to
re-create what the problem and solution really was, making this
information unfortunately less useful to you guys.

Now, this has happened to me **twice** in as many weeks (again, thanks
to power failures, but still, shouldn't BDB's journaling allow DB
recovery??). I'm getting a little worried aboutsvn and bdb.

Does anyone have any suggestions as to:
(a) What's going on here? Why can't the recovery process deal with this?
(b) Why deleting corrupted logs sort of works? Is this the right thing
to do? The wrong thing to do? What should one do when svnadmin recover
fails like this?

Zach Pincus

Department of Biochemistry and Program in Biomedical Informatics
Stanford University School of Medicine

STEP 1: Failed attempt at svnadmin recover:
----------------
zpincus% svnadmin recover Projects/
Repository lock acquired.
Please wait; recovering the repository may take some time...
svn: DB_RUNRECOVERY: Fatal error, run database recovery
svn: bdb: DB_ENV->log_flush: LSN of 115/802071 past current end-of-log
of 115/731460
svn: bdb: Database environment corrupt; the wrong log files may have
been removed or incompatible database files imported from another
environment
svn: bdb: DB_ENV->log_flush: LSN of 115/802071 past current end-of-log
of 115/731460
svn: bdb: Database environment corrupt; the wrong log files may have
been removed or incompatible database files imported from another
environment
svn: bdb: DB_ENV->log_flush: LSN of 115/802071 past current end-of-log
of 115/731460
svn: bdb: Database environment corrupt; the wrong log files may have
been removed or incompatible database files imported from another
environment
svn: bdb: DB_ENV->log_flush: LSN of 115/802071 past current end-of-log
of 115/731460
svn: bdb: Database environment corrupt; the wrong log files may have
been removed or incompatible database files imported from another
environment
svn: bdb: DB_ENV->log_flush: LSN of 115/802071 past current end-of-log
of 115/731460
svn: bdb: Database environment corrupt; the wrong log files may have
been removed or incompatible database files imported from another
environment
svn: bdb: changes: unable to flush page: 0
svn: bdb: txn_checkpoint: failed to flush the buffer cache Invalid
argument
svn: bdb: PANIC: Invalid argument
svn: bdb: PANIC: fatal region error detected; run recovery
svn: bdb: PANIC: fatal region error detected; run recovery
svn: bdb: PANIC: fatal region error detected; run recovery
svn: bdb: PANIC: fatal region error detected; run recovery
svn: bdb: PANIC: fatal region error detected; run recovery
svn: bdb: PANIC: fatal region error detected; run recovery
---------------------

STEP 2: Then I move log 115 away and attempt to recover again. It fails
---------------------
zpincus% mv Projects/db/log.0000000115 log.0000000115-wedged
zpincus% svnadmin recover Projects/
Repository lock acquired.
Please wait; recovering the repository may take some time...
svn: DB_RUNRECOVERY: Fatal error, run database recovery
svn: bdb: Ignoring log file: Projects/db/log.0000000115: magic number
0, not 40988
svn: bdb: Invalid log file: log.0000000115: Invalid argument
svn: bdb: PANIC: Invalid argument
svn: bdb: PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
---------------------

STEP 3: I do the same thing again (log 115 was re-generated) and it
works!
---------------------
zpincus% mv Projects/db/log.0000000115 log.0000000115-wedged-2
zpincus% svnadmin recover Projects/
Repository lock acquired.
Please wait; recovering the repository may take some time...

Recovery completed.
The latest repos revision is 154.
-----------------------

STEP 4: I ran a verify and a dump. Then I tried a commit, and the
commit failed. Running svnadmin verify then resulted in a
seemingly-endless torrent of the following message repeated over and
over:
-----------------------
svn: bdb: DB_ENV->log_flush: LSN of 115/723808 past current end-of-log
of 1/1066
svn: bdb: Database environment corrupt; the wrong log files may have
been removed or incompatible database files imported from another
environment

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Received on Tue Jun 28 00:54:50 2005

This is an archived mail posted to the Subversion Users mailing list.