[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: Severe repository database corruption

From: Aaron Wohl <subversion_at_awohl.com>
Date: 2004-07-02 02:46:20 CEST

-I see you doing a lot of stuff as root... be sure when your done to do
chmod -R user:group for who it should be owned by for the webserver or
synserver

-With the same sort of problems your seeing, I was able do svnadmin dump
then svnadmin create and svnadmin load. Ill bet your stuff was
recoverable without going to backup tapes.

-during your work it looks like stuff was left as /usr/local/apache2/svn
During recovery its important that noone else have access for stuff like
this, I do mv /usr/local/apache2/svn /usr/local/apache2/svn.sick .
Then do the repair work and put it back. Double check your
owner/group/permission as that can make it look broken aftewards if root
owns stuff.

----- Original message -----
From: "Frank Freund" <ffreund@memocomp.de>
To: users@subversion.tigris.org
Date: Fri, 02 Jul 2004 00:34:19 +0200
Subject: Severe repository database corruption

Hi all,
today morning, and after my last commit yesterday (and noone accessed
the repository in any way in between) worked fine, I got the following
error message while trying to commit (using TortoiseSVN):
> "DB_RUNRECOVERY: Fatal error, run database recovery."
Checking the repository via the browser, I got: "Could not open the
requested SVN filesystem", so I started reading docs, mailing lists and
searched the web.
To anticipate, none of the things I tried worked :-( and it seems, that
the database is the problem (but maybe not the reason)

First I will give some information about the environment, then the
things I tried and finally some of the sources I found concerning (other
people having) this problem.

Environment
-------------------
The repository was running now for about half a year with nearly 400
commits and 700MB repository size.
The System configuration is:
SuSE 8.2 with
* Subversion 0.35.1 (now upgraded to 1.0.5, but the problem remains
exactly the same)
* BDB 4.2.52.1
* Apache 2.0.48
All above build from source, and the clientside
* Tortoise SVN 0.24.0 build 554

Attempts
-------------------
First I tried
> linux: # svnadmin recover /usr/local/apache2/svn
> Please wait; recovering the repository may take some time...
> Recovery completed.
> The latest repos revision is 394.
which looked fine, but didn't worked - same errors as above. So next was
> linux: # svnadmin verify /usr/local/apache2/svn
which also completed successfully, but didn't helped either. Next step
were the databasetools:
> linux: # ./db_recover -h /usr/local/apache2/svn/db
This worked, but I just could access the repository over the browser for
some tries, before the same error occurred again. (reproducable always)
Accessing the repository using TorsoiseSVN, I just tried checkouts,
failed directly (reproducable always)
Tried again:
> linux: # svnadmin verify /usr/local/apache2/svn
> svn: Berkeley DB error while opening environment for filesystem
/usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery
and
> linux: # ./db_recover -h /usr/local/apache2/svn/db
which worked again for short (sometimes 10 seconds, never longer than
about 2 minutes). Some tries later the verify failed:
> linux: # svnadmin verify /usr/local/apache2/svn
> Verified revision 0.
> svn: Berkeley DB error while beginning Berkeley DB transaction for
filesystem /usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery
> svn: Berkeley DB error while closing 'nodes' database for filesystem
/usr/local/apache2/svn/memocomp/db:
> DB_RUNRECOVERY: Fatal error, run database recovery

Meanwhile updated to SVN1.0.5 and imported the backup (Jesus saves:-)
successfully, but the symptoms remain the same.
Some posts mentioned problems with mixed up libdb's, but as far as I
know a
> linux: # grep libdb /proc/17455/maps
> 4002a000-400e3000 r-xp 00000000 08:02 42429
/usr/local/BerkeleyDB.4.2/lib/libdb-4.2.so
> 400e3000-400e5000 rw-p 000b8000 08:02 42429
/usr/local/BerkeleyDB.4.2/lib/libdb-4.2.so
with 17455 beeing a httpd-PID, confirms me that this isn't the case.

The http-error log shows:
> [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11] Could not
fetch resource information. [500, #0]
> [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11] Could not
determine the proper revision to access [500, #160029]
> [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11]
(13)Permission denied: Berkeley DB error while checkpointing after
Berkeley DB transaction for filesystem /usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery [500, #160029]
> [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11]
(20014)Error string not specified yet: Berkeley DB error while closing
'nodes' database for filesystem /usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery
> [Thu Jul 01 08:17:02 2004] [error] [client 192.168.100.11]
(20014)Error string not specified yet: Berkeley DB error while opening
environment for filesystem /usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery, referer:
http://192.168.100.2/repository.html
and later
> [Thu Jul 01 20:14:46 2004] [error] [client 192.168.100.11] Could not
fetch resource information. [500, #0], referer:
http://192.168.100.2/repository.html
> [Thu Jul 01 20:14:46 2004] [error] [client 192.168.100.11] Could not
open the requested SVN filesystem [500, #160029], referer:
http://192.168.100.2/repository.html
> [Thu Jul 01 20:14:46 2004] [error] [client 192.168.100.11] (17)File
exists: Could not open the requested SVN filesystem [500, #160029],
referer: http://192.168.100.2/repository.html
> [Thu Jul 01 20:14:47 2004] [error] [client 192.168.100.11]
(20014)Error string not specified yet: Berkeley DB error while opening
environment for filesystem /usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery, referer:
http://192.168.100.2/repository.html

Further readings
-------------------
Repository recovery:
http://svnbook.red-bean.com/svnbook/ch05s03.html#svn-ch-5-sect-3.4
Another broken BDB:
http://www.contactor.se/~dast/svnusers/archive-2004-05/0556.shtml
Corrupted repository db:
http://www.contactor.se/~dast/svn/archive-2004-05/0071.shtml
Corrupted repository db:
http://www.contactor.se/~dast/svn/archive-2004-05/0097.shtml
Unrecoverable databases:
http://www.contactor.se/~dast/svnusers/archive-2004-05/1283.shtml
http://subversion.tigris.org/servlets/BrowseList?list=users&by=thread&from=202660

So that's all and I like to thank everyone who has read this &%&$*long
post.

Thanks in advance for any help!
Frank

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Received on Fri Jul 2 02:50:35 2004

This is an archived mail posted to the Subversion Users mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.