Repository corruption; svnadmin recover reports it's fixed, svnadmin verify immediately disagrees

(Apologies if this posts twice; I posted it an hour ago
from an address that isn't subbed to users@ and I think
it got blocked.)

Our main development repository has become corrupted.

Fortunately, I have a full svnsync.d repository that.s only a few days old
(-r114959) and we have all the current development work distributed on
developer workstations and build machines, so we really haven.t lost any
IP, but I.d like to recover everything, rather than punt to a couple day
old backup and re-import the latest since then.

Here.s the short summary of the current situation:

Apache immediately 500s when serving the repository
svnadmin recover says: we.re good. Last rev is 116241
svnadmin verify says: Fatal error; run database recovery when verifying
revision 0 (or 1?)
BDB 4.3 db_recover says: unsupported log version 11
BDB 4.4 db_recover says: <nothing, unless I pass .v, in which case it
seems to succeed>
svnadmin verify still fails, same error
svnlook against the filesystem seems to work (for the options I.ve tried
anyway)

Versions of software (all Windows):

Apache 2.0.59

Apache modules 1.4.3

Svn command line, a mix of 1.4.2 and 1.4.3, mostly 1.4.3

Here's the longish recollection of what led to the current situation:

For a few months, I.d been seeing intermittent (once a week maybe) Apache
crashes.

We were svnsync.ing to a slave repository, but only for about the past 2
months. This was in preparation for the next step:

We moved offices.

The crashes became MUCH more frequent in the past two weeks, so I stopped
the svnsync process, thinking that was contributing to the instability.

Today, the repository went down hard and our usual reset-the-svn-server
script didn.t work. (This script kills apache, svnadmin recovers all the
DBs, starts apache.)

   Instead the svnadmin (1.4.3) recover process hung, and was completely
unkillable, via either pskill or task manager, and the machine had to be
rebooted. It didn.t appear that the .stuck. svnadmin was actually doing
anything on the filesystem, nor consuming CPU. I let it go about 10
minutes and then rebooted the machine.

I then did an svnadmin recover manually, inadvertently using the 1.4.2
svnadmin binary (which was on my path)

Subsequent restarts didn.t improve the situation. All 6 other repos work
fine; it.s just this one that.s affected.

Svnadmin dump doesn.t work, complaining:
SVN-fs-dump-format-version: 2

UUID: 8217dfb6-ca92-db4d-a461-c54c91ed5a1f

Revision-number: 0
Prop-content-length: 56
Content-length: 56

K 8
svn:date
V 27
2003-12-14T18:41:21.530625Z
PROPS-END

* Dumped revision 0.

Revision-number: 1
Prop-content-length: 118
Content-length: 118

K 7
svn:log
V 14
Initial Import
K 10
svn:author
V 9
jsokoloff
K 8
svn:date
V 27
2003-12-14T20:14:33.124375Z
PROPS-END

svnadmin: Berkeley DB error for filesystem 'db' while moving cursor:

DB_RUNRECOVERY: Fatal error, run database recovery

svnadmin: bdb: page 0: illegal page type or format

svnadmin: bdb: PANIC: Invalid argument

svnadmin: bdb: PANIC: fatal region error detected; run recovery

------------------------------------------------------------------------------

svnadmin recover reports:

Repository lock acquired.

Please wait; recovering the repository may take some time...

Recovery completed.

The latest repos revision is 116241.

------------------------------------------------------------------------------

BDB 4.3 db_recover reports:

db_recover: Program version 4.3 doesn't match environment version

db_recover: Unacceptable log file db\log.0000073513: unsupported log
version 11

db_recover: Invalid log file: log.0000073513: Invalid argument

db_recover: PANIC: Invalid argument

db_recover: PANIC: DB_RUNRECOVERY: Fatal error, run database recovery

db_recover: DB_ENV->open: DB_RUNRECOVERY: Fatal error, run database
recovery

BDB 4.4 db_recover reports:

Finding last valid log LSN: file: 73513 offset 444418

Recovery starting from [73513][442146]

Recovery complete at Thu May 17 18:20:31 2007

Maximum transaction ID 8000000a Recovery checkpoint [73513][444418]

I couldn.t figure out how to get anything more useful out of db_verify. It
complained about every DB file I threw it at, and didn.t complain about
any DB file from one of my other non-broken repos:

c:\Program Files\Sleepycat Software\Berkeley DB 4.4.20\bin\db_verify.exe:
PANIC: fatal region error detected; run recovery

As for the logs dir, the only logs from the last week were both from
today, 73512 and 73513.

Anti-virus is on the machine; I.m chasing down whether any config change
was made to that, but it is supposed to be excluding the entire folder
hosting svn. However, in running the various tools above, I occasionally
get:

svnadmin: Berkeley DB error for filesystem '../db' while opening
environment:

svnadmin: Invalid argument

svnadmin: bdb: DB_REGISTER limits each process to a single open DB_ENV
handle

which leads me to wonder if AV is in the way somehow.

Any help would be hugely appreciated. I can give whatever more details
you.d need. I tried to be as comprehensive as I could, but naturally may
have missed something.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Received on Fri May 18 00:58:05 2007