Hi Dana
From: Dana Epp
Sent: Saturday, 17 August 2013 4:47 AM
We have an old SVN database corruption we need help with. (SVN 1.1 with BDB 4.2)
Does anyone know of someone who consults on this sort of thing?
We've been dealing with SVN 1.2.3 for a long time, and have developed a process for dealing with corrupted databases. Below is a copy from our (internal) Wiki page. I hope it helps you and others.
Regards,
Geoff
Recovering Corrupted Repositories
Simple Corruption (Log File)
Very occasionally, a file is committed to a repository which causes the repository to be corrupted. This usually manifests itself as access to the repository freezing. See the extensive notes in Mantis Issue 7585 . Here are the steps required to recover:
Warn everybody that SubVersion is about to go down! Remember that some repositories may not be corrupt.
Locate a drive with sufficient spare space for the affected repositories
Log onto the SubVersion server. This requires administrative access, which is limited to a very small number of people.
Stop the Apache service
Backup the affected repositories
Remove the log files which don't have the db version as 0x0A in the 16th byte of the offending log file. For example, log.0000000002030 had a db version of 0x0b.
Run "db_recover" from within the db directory of the repository. This requires the utility to be in the path (C:\Program Files\Sleepycat Software\Berkeley DB 4.3.29\bin)
Run "svnadmin recover .\" from within the repo directory (i.e. one above the db dir)
Re-start the Apache service
Inform everybody that SubVersion is back up.
Note Sometimes db_recover does not work. In this case, the "-c" flag helps for "critical errors"
Serious Corruption (Database)
From http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=1617554
"Nicholas Walker" <NWalker at dialogue-marketing dot com>
Full name "Nicholas Walker" <NWalker at dialogue-marketing dot com>
Date 2009-04-09 13:31:59 PDT
Message I went through a week of trying to resolve various issues with an old
out of control corrupted v 1.4 Berkeley Db subversion repository.
These steps will not work for an fsfs repository.
I put together a process for how I resolved my issues. I hope it helps somebody else
out.
Note: This process does not solve corruption, it just puts the repository into a state
where svnadmin dump will not fail. The steps below will result in specific histories
being lost. If you have the option to go back to recent backup to resolve the issue that
would be a much better strategy. But when the repository has not been maintained for
ears as in my case there is no other way.
You'll need to use the Berkeley db utilities that match the version of Berkeley db that
is being used in your repository in my case those utilities were found in
"/usr/local/bin/db42". The path , and executables on your machine may not match mine.
Copy repository to a temporary location.
You can use svnadmin hotcopy REPOS_PATH NEW_REPOS_PATH
Resolve pure database corruption
This resolves error similar to DB_PAGE_NOT_FOUND error.
There are files in [repository Path]/db
Changes
Copies
Nodes
Representations
Revisions
Strings
Transactions
uuids
Each of these represent a database table
Run "/usr/local/bin/db42/db_dump -r File > File.dump" against each of the files.
This dumps the database table to a dump file, and removes instances where corruption has occurred.
Note that this will remove corrupted file commits from the database table, but they were irrecoverable anyway so it doesn't matter.
If you have a large repository the dump and reload process make take a considerable amount of time, especially for the strings table.
Run "rm [File]" for each of the files.
Delete the file not the .dump file
Run "/usr/local/bin/db42/db_load File < File.dump" for each of the files.
This is wonderful in theory, but I have found that the version of db_load we currently have does not like the format of the file output by db_dump on Windows: db_dump puts CR/LF on the end of each line, while db_load does not like this. To correct this:
Open the file in Notepad++
View the file in Hex (Alt-Ctrl-Shift-H)
Highlight the first 0d 0a.
Select Search->Replace (Ctrl-H)
Enter "0d" (that's number 0, not letter O) in the "Replace with" field
Click on "Replace All"
Save the file.
Since the process takes a long time, you may want to make a backup of your backup in case you mess up in the next steps, you have a half way point to revert to, since step 2 can take a long time.
Resolve repository corruption
These errors occur when a transaction for example references a string/commit that no longer exists because you removed it due to database corruption in step 2, a checksum failure, or some other type of error.
Start a dump of the database
If no errors occur during the dump, you are done, and have resolved all corruption
When you experience an error, you'll have to manually update the database table files to resolve the corruption.
Here are steps for several types of errors that I came across.
Checksum Failure
Error along the lines of checksum mismatch on rep "[rep]" expected: checksum actual: checksum
There is some reason why the checksum failed because some kind of bad commit. We are going to trick svn into thinking that the checksum has passed.
Run "/usr/local/bin/db42/db_dump -kp representations > representations.dump"
The "-kp" option dumps the file into a ascii format, so you can open it in a text editor
Open representations.dump in a text file
Search for the representation id. What is in rep in the error message
You will see lines like the below
11ti
((fulltext 1 1 (md5 16 \e3\035\f0\19\c5V\07}\92h\ab\80\aam\02)) 1 0)
You want to find the line where the rep id matches on the entire line.
You may find the rep id in other lines within the file, you should ignore these. You only want to look at the section where the entire line is equal to the rep id in the error message.
Change the section "md5 16 \e3\035\f0\19\c5V\07}\92h\ab\80\aam\02" to "md5 \00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00". This is 16 serious of strings with "\00". This checksum matches everything.
Run rm representations
Run "/usr/local/bin/db42/db_load representations < representations.dump"
Re-run the dump to verify your changes corrected the problem.
Only dump the single revision "svnadmin dump -r revision -incremental"
Note that the revision that failed is the last revision noted in the output to the dump command +1. The dump outputs the revision when it completes.
Continue with the dump until another error occurs
Representation not found error
Note the rep id.
Run "/usr/local/bin/db42/db_dump -kp representations > representations.dump"
The "-kp" option dumps the file into a ascii format, so you can open it in a text editor
Open representations.dump in a text file
Near the top of the file find a representation that is "fulltext"
Copy the line of text
Insert a representation into the text file that looks like the one you just copies except replace the copied rep id with the rep id in the error message
Run rm representations
Run "/usr/local/bin/db42/db_load representations < representations.dump"
Re-run the dump to verify your changes corrected the problem.
Only dump the single revision "svnadmin dump -r revision -incremental"
Continue with the dump until another error occurs
String not found error
This one is a pain. A corrupted string may cause any depending revisions on it to also fail.
Open representations.dump in a text file
Search for the string id in the text file.
You will find the string in the second line / the data text line. If you find it in the id line then you found a representation that has the same id as the string.
You should find the string in the last few characters in the data line.
"((fulltext 3 2bn (md5 16 \d2\cci\d7\b8\b7\c8\14\b6;St\a0\18\14\b9)) 4 19y4)" in this example 19y4 is the string id
Copy the data line from one of the first few lines of the file that is a fulltext. Remove the data line that shows the string id that was not found, and replace it with the one that was copied.
Run "rm representations"
Run "/usr/local/bin/db42/db_load representations < representations.dump"
Re-run the dump to verify your changes corrected the problem.
Only dump the single revision "svnadmin dump -r revision -incremental"
Continue with the dump until another error occurs
Error similar to svndiff failed inconsistency in content length
This is an error that results from the fix in the step above
Follow the same steps as in the above step to resolve this issue.
A difference is you will be looking for rep id now instead of the string id
The data line will surely be one that is a delta
Error similar to proplist skeleton malformated
Part of the dump file will have been generated. Open the dump file, and go to the last few lines.
The last file noted is the file that we want to note.
Only way I could find to resolve this is to remove all instances of the file from the changes table.
If you found the exact revision instances that was corrupted, you might be able to salvage most of the history of the file, but I had a hard time finding the correct one, so just ended up having to wipe it out completely.
Run "/usr/local/bin/db42/db_dump -kp changes > changes.dump"
The -kp option dumps the file into an ascii format, so you can open it in a text editor
Open changes.dump in a text file
Search for all instances of the file name, and remove them.
Note the id line that is above the line that reads the file name must also be removed.
Run "rm changes"
Run "/usr/local/bin/db42/db_load changes < changes.dump"
Re-run the dump to verify your changes corrected the problem.
Only dump the single revision "svnadmin dump -r revision -incremental"
Continue with the dump until another error occurs
Error similar to file or directory [File name] could not be found
Follow the same steps as in "v" above
Run the entire dump, and restore it to a new repository
Restore any files that had to be removed during the process of fixing the corruption
You will lose the history associated with the file that had to be removed, but you can re-import the last version of the file from the older repository.
Do a svn checkout of both the old, and new repositories, and do a merge between the two to try and ensure that the most recent version of the files are maintained
Run svnadmin verify on a regular basis, and identify the issues before they get out of control.
- The contents of this email, and any attachments, are strictly private
and confidential.
- It may contain legally privileged or sensitive information and is intended
solely for the individual or entity to which it is addressed.
- Only the intended recipient may review, reproduce, retransmit, disclose,
disseminate or otherwise use or take action in reliance upon the information
contained in this email and any attachments, with the permission of
Australian Arrow Pty. Ltd.
- If you have received this communication in error, please reply to the sender
immediately and promptly delete the email and attachments, together with
any copies, from all computers.
- It is your responsibility to scan this communication and any attached files
for computer viruses and other defects and we recommend that it be
subjected to your virus checking procedures prior to use.
- Australian Arrow Pty. Ltd. does not accept liability for any loss or damage
of any nature, howsoever caused, which may result
directly or indirectly from this communication or any attached files.
Received on 2013-08-19 01:31:33 CEST