Fixing subversion corruption - Berkeley DB

I went through a week of trying to resolve various issues with an old
out of control corrupted v 1.4 Berkeley Db subversion repository.

These steps will not work for an fsfs repository.

I put together a process for how I resolved my issues. I hope it helps
somebody else out.

Note: This process does not solve corruption, it just puts the
repository into a state where svnadmin dump will not fail. The steps
below will result in specific histories being lost. If you have the
option to go back to recent backup to resolve the issue that would be a
much better strategy. But when the repository has not been maintained
for years as in my case there is no other way.

You'll need to use the Berkeley db utilities that match the version of
Berkeley db that is being used in your repository in my case those
utilities were found in /usr/local/bin/db42. The path , and executables
on your machine may not match mine.

1) Copy repository to a temporary location.

a. You can use svnadmin hotcopy REPOS_PATH NEW_REPOS_PATH

2) Resolve pure database corruption

a. This resolves error similar to DB_PAGE_NOT_FOUND error.

b. There are files in [repository Path]/db

                                                               i.
Changes

                                                             ii.
Copies

                                                            iii.
Nodes

                                                           iv.
Representations

                                                             v.
Revisions

                                                           vi.
Strings

                                                          vii.
Transactions

                                                        viii. uuids

c. Each of these represent a database table

d. Run /usr/local/bin/db42/db_dump -r [File] > [File].dump against
each of the files.

                                                               i.
This dumps the database table to a dump file, and removes instances
where corruption has occurred.

                                                             ii.
Note that this will remove corrupted file commits from the database
table, but they were irrecovable anyway so it doesn't matter.

                                                            iii. If
you have a large repository the dump and reload process make take a
considerable amount of time, especially for the strings table.

e. Run rm [File] for each of the files.

                                                               i.
Delete the file not the .dump file

f. Run /usr/local/bin/db42/db_load [File] < [File].dump for each
of the files.

3) Since the process takes a long time, you may want to make a
backup of your backup in case you mess up in the next steps, you have a
half way point to revert to, since step 2 can take a long time.

4) Resolve repository corruption

a. These errors occur when a transaction for example references a
string/commit that no longer exists because you removed it due to
database corruption in step 2, a checksum failure, or some other type of
error.

b. Start a dump of the database

c. If no errors occur during the dump, you are done, and have
resolved all corruption

d. When you experience an error, you'll have to manually update the
database table files to resolve the corruption.

e. Here are steps for several types of errors that I came across.

                                                               i.
Checksum Failure

1. Error along the lines of checksum mismatch on rep "[rep]"
expected: [checksum] actual: [checksum]

2. There is some reason why the checksum failed because some kind
of bad commit. We are going to trick svn into thinking that the
checksum has passed.

3. Run /usr/local/bin/db42/db_dump -kp representations >
representations.dump

4. The -kp option dumps the file into a ascii format, so you can
open it in a text editor

5. Open representations.dump in a text file

6. Search for the representation id. What is in [rep] in the
error message

7. You will see lines like the below

a. 11ti

b. ((fulltext 1 1 (md5 16 \e3\035\f0\19\c5V\07}\92h\ab\80\aam\02))
1 0)

8. You want to find the line where the rep id matches on the
entire line.

9. You may find the rep id in other lines within the file, you
should ignore these. You only want to look at the section where the
entire line is equal to the rep id in the error message.

10. Change the section md5 16 \e3\035\f0\19\c5V\07}\92h\ab\80\aam\02
to md5 \00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00. This is 16
serious of strings with "\00". This checksum matches everything.

11. Run rm representations

12. Run /usr/local/bin/db42/db_load representations <
representations.dump

13. Re-run the dump to verify your changes corrected the problem.
Only dump the single revision svnadmin dump -r [revision] -incremental

14. Note that the revision that failed is the last revision noted in
the output to the dump command +1. The dump outputs the revision when
it completes.

15. Continue with the dump until another error occurs

                                                             ii.
Representation not found error

1. Note the rep id.

2. Run /usr/local/bin/db42/db_dump -kp representations >
representations.dump

3. The -kp option dumps the file into a ascii format, so you can
open it in a text editor

4. Open representations.dump in a text file

5. Near the top of the file find a representation that is
"fulltext"

6. Copy the line of text

7. Insert a representation into the text file that looks like the
one you just copies except replace the copied rep id with the rep id in
the error message

8. Run rm representations

9. Run /usr/local/bin/db42/db_load representations <
representations.dump

10. Re-run the dump to verify your changes corrected the problem.
Only dump the single revision svnadmin dump -r [revision] -incremental

11. Continue with the dump until another error occurs

                                                            iii.
String not found error

1. This one is a pain. A corrupted string may cause any depending
revisions on it to also fail.

2. Open representations.dump in a text file

3. Search for the string id in the text file.

4. You will find the string in the second line / the data text
line. If you find it in the id line then you found a representation
that has the same id as the string.

5. You should find the string in the last few characters in the
data line.

6. ((fulltext 3 2bn (md5 16
\d2\cci\d7\b8\b7\c8\14\b6;St\a0\18\14\b9)) 4 19y4) in this example 19y4
is the string id

7. Copy the data line from one of the first few lines of the file
that is a fulltext. Remove the data line that shows the string id that
was not found, and replace it with the one that was copied.

8. Run rm representations

9. Run /usr/local/bin/db42/db_load representations <
representations.dump

10. Re-run the dump to verify your changes corrected the problem.
Only dump the single revision svnadmin dump -r [revision] -incremental

11. Continue with the dump until another error occurs

                                                           iv.
Error similar to svndiff failed inconsistency in content length

1. This is an error that results from the fix in the step above

2. Follow the same steps as in the above step to resolve this
issue.

3. A difference is you will be looking for rep id now instead of
the string id

4. The data line will surely be one that is a delta

                                                             v.
Error similar to proplist skeleton malformated

1. Part of the dump file will have been generated. Open the dump
file, and go to the last few lines.

a. The last file noted is the file that we want to note.

2. Only way I could find to resolve this is to remove all
instances of the file from the changes table.

a. If you found the exact revision instances that was corrupted,
you might be able to salvage most of the history of the file, but I had
a hard time finding the correct one, so just ended up having to wipe it
out completely.

3. Run /usr/local/bin/db42/db_dump -kp changes > changes.dump

4. The -kp option dumps the file into an ascii format, so you can
open it in a text editor

5. Open changes.dump in a text file

6. Search for all instances of the file name, and remove them.
Note the id line that is above the line that reads the file name must
also be removed.

7. Run rm changes

8. Run /usr/local/bin/db42/db_load changes < changes.dump

9. Re-run the dump to verify your changes corrected the problem.
Only dump the single revision svnadmin dump -r [revision] -incremental

10. Continue with the dump until another error occurs

                                                           vi.
Error similar to file or directory [File name] could not be found

1. Follow the same steps as in "v" above

5) Run the entire dump, and restore it to a new repository

6) Restore any files that had to be removed during the process of
fixing the corruption

a. You will lose the history associated with the file that had to
be removed, but you can re-import the last version of the file from the
older repository.

b. Do a svn checkout of both the old, and new repositories, and do
a merge between the two to try and ensure that the most recent version
of the files are maintained

7) Run svnadmin verify on a regular basis, and identify the issues
before they get out of control

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=1617554

To unsubscribe from this discussion, e-mail: [users-unsubscribe_at_subversion.tigris.org].
Received on 2009-04-09 22:36:16 CEST