[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

RE: Are there SVN consultants out there who remember the ol' days?

From: Geoff Field <Geoff_Field_at_aapl.com.au>
Date: Mon, 19 Aug 2013 09:30:53 +1000

Hi Dana
 

        From: Dana Epp
        Sent: Saturday, 17 August 2013 4:47 AM
        
        We have an old SVN database corruption we need help with. (SVN 1.1 with BDB 4.2)
         
        Does anyone know of someone who consults on this sort of thing?

We've been dealing with SVN 1.2.3 for a long time, and have developed a process for dealing with corrupted databases. Below is a copy from our (internal) Wiki page. I hope it helps you and others.

Regards,

Geoff

Recovering Corrupted Repositories
Simple Corruption (Log File)

Very occasionally, a file is committed to a repository which causes the repository to be corrupted. This usually manifests itself as access to the repository freezing. See the extensive notes in Mantis Issue 7585 . Here are the steps required to recover:

    Warn everybody that SubVersion is about to go down! Remember that some repositories may not be corrupt.
    Locate a drive with sufficient spare space for the affected repositories
    Log onto the SubVersion server. This requires administrative access, which is limited to a very small number of people.
    Stop the Apache service
    Backup the affected repositories
    Remove the log files which don't have the db version as 0x0A in the 16th byte of the offending log file. For example, log.0000000002030 had a db version of 0x0b.
    Run "db_recover" from within the db directory of the repository. This requires the utility to be in the path (C:\Program Files\Sleepycat Software\Berkeley DB 4.3.29\bin)
    Run "svnadmin recover .\" from within the repo directory (i.e. one above the db dir)
    Re-start the Apache service
    Inform everybody that SubVersion is back up.

Note Sometimes db_recover does not work. In this case, the "-c" flag helps for "critical errors"

Serious Corruption (Database)
From http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=1617554

 "Nicholas Walker" <NWalker at dialogue-marketing dot com>
 Full name "Nicholas Walker" <NWalker at dialogue-marketing dot com>
 Date 2009-04-09 13:31:59 PDT
 Message I went through a week of trying to resolve various issues with an old
 out of control corrupted v 1.4 Berkeley Db subversion repository.

 These steps will not work for an fsfs repository.

 I put together a process for how I resolved my issues. I hope it helps somebody else
 out.

 Note: This process does not solve corruption, it just puts the repository into a state
 where svnadmin dump will not fail. The steps below will result in specific histories
 being lost. If you have the option to go back to recent backup to resolve the issue that
 would be a much better strategy. But when the repository has not been maintained for
 ears as in my case there is no other way.

 You'll need to use the Berkeley db utilities that match the version of Berkeley db that
 is being used in your repository in my case those utilities were found in
 "/usr/local/bin/db42". The path , and executables on your machine may not match mine.

    Copy repository to a temporary location.
        You can use svnadmin hotcopy REPOS_PATH NEW_REPOS_PATH
    Resolve pure database corruption
        This resolves error similar to DB_PAGE_NOT_FOUND error.
        There are files in [repository Path]/db
            Changes
            Copies
            Nodes
            Representations
            Revisions
            Strings
            Transactions
            uuids
        Each of these represent a database table
        Run "/usr/local/bin/db42/db_dump -r File > File.dump" against each of the files.
            This dumps the database table to a dump file, and removes instances where corruption has occurred.
            Note that this will remove corrupted file commits from the database table, but they were irrecoverable anyway so it doesn't matter.
            If you have a large repository the dump and reload process make take a considerable amount of time, especially for the strings table.
        Run "rm [File]" for each of the files.
            Delete the file not the .dump file
        Run "/usr/local/bin/db42/db_load File < File.dump" for each of the files.
            This is wonderful in theory, but I have found that the version of db_load we currently have does not like the format of the file output by db_dump on Windows: db_dump puts CR/LF on the end of each line, while db_load does not like this. To correct this:
                Open the file in Notepad++
                View the file in Hex (Alt-Ctrl-Shift-H)
                Highlight the first 0d 0a.
                Select Search->Replace (Ctrl-H)
                Enter "0d" (that's number 0, not letter O) in the "Replace with" field
                Click on "Replace All"
                Save the file.
    Since the process takes a long time, you may want to make a backup of your backup in case you mess up in the next steps, you have a half way point to revert to, since step 2 can take a long time.
    Resolve repository corruption
        These errors occur when a transaction for example references a string/commit that no longer exists because you removed it due to database corruption in step 2, a checksum failure, or some other type of error.
        Start a dump of the database
        If no errors occur during the dump, you are done, and have resolved all corruption
        When you experience an error, you'll have to manually update the database table files to resolve the corruption.
        Here are steps for several types of errors that I came across.
            Checksum Failure
                Error along the lines of checksum mismatch on rep "[rep]" expected: checksum actual: checksum
                There is some reason why the checksum failed because some kind of bad commit. We are going to trick svn into thinking that the checksum has passed.
                Run "/usr/local/bin/db42/db_dump -kp representations > representations.dump"
                The "-kp" option dumps the file into a ascii format, so you can open it in a text editor
                Open representations.dump in a text file
                Search for the representation id. What is in rep in the error message
                You will see lines like the below
                    11ti
                    ((fulltext 1 1 (md5 16 \e3\035\f0\19\c​5V\07}\92h\ab\80​\aam\02)) 1 0)
                You want to find the line where the rep id matches on the entire line.
                You may find the rep id in other lines within the file, you should ignore these. You only want to look at the section where the entire line is equal to the rep id in the error message.
                Change the section "md5 16 \e3\035\f0\19\c​5V\07}\92h\ab\80​\aam\02" to "md5 \00\00\00\00\00​\00\00\00\00\00​\00\00\00\00\00​\00". This is 16 serious of strings with "\00". This checksum matches everything.
                Run rm representations
                Run "/usr/local/bin/db42/db_load representations < representations.dump"
                Re-run the dump to verify your changes corrected the problem.
                    Only dump the single revision "svnadmin dump -r revision -incremental"
                Note that the revision that failed is the last revision noted in the output to the dump command +1. The dump outputs the revision when it completes.
                Continue with the dump until another error occurs
            Representation not found error
                Note the rep id.
                Run "/usr/local/bin/db42/db_dump -kp representations > representations.dump"
                The "-kp" option dumps the file into a ascii format, so you can open it in a text editor
                Open representations.dump in a text file
                Near the top of the file find a representation that is "fulltext"
                Copy the line of text
                Insert a representation into the text file that looks like the one you just copies except replace the copied rep id with the rep id in the error message
                Run rm representations
                Run "/usr/local/bin/db42/db_load representations < representations.dump"
                Re-run the dump to verify your changes corrected the problem.
                    Only dump the single revision "svnadmin dump -r revision -incremental"
                Continue with the dump until another error occurs
            String not found error
                This one is a pain. A corrupted string may cause any depending revisions on it to also fail.
                Open representations.dump in a text file
                Search for the string id in the text file.
                You will find the string in the second line / the data text line. If you find it in the id line then you found a representation that has the same id as the string.
                You should find the string in the last few characters in the data line.
                "((fulltext 3 2bn (md5 16 \d2\cci\d7\b8\b​7\c8\14\b6;St\a0​\18\14\b9)) 4 19y4)" in this example 19y4 is the string id
                Copy the data line from one of the first few lines of the file that is a fulltext. Remove the data line that shows the string id that was not found, and replace it with the one that was copied.
                Run "rm representations"
                Run "/usr/local/bin/db42/db_load representations < representations.dump"
                Re-run the dump to verify your changes corrected the problem.
                    Only dump the single revision "svnadmin dump -r revision -incremental"
                Continue with the dump until another error occurs
            Error similar to svndiff failed inconsistency in content length
                This is an error that results from the fix in the step above
                Follow the same steps as in the above step to resolve this issue.
                A difference is you will be looking for rep id now instead of the string id
                The data line will surely be one that is a delta
            Error similar to proplist skeleton malformated
                Part of the dump file will have been generated. Open the dump file, and go to the last few lines.
                    The last file noted is the file that we want to note.
                Only way I could find to resolve this is to remove all instances of the file from the changes table.
                    If you found the exact revision instances that was corrupted, you might be able to salvage most of the history of the file, but I had a hard time finding the correct one, so just ended up having to wipe it out completely.
                Run "/usr/local/bin/db42/db_dump -kp changes > changes.dump"
                The -kp option dumps the file into an ascii format, so you can open it in a text editor
                Open changes.dump in a text file
                Search for all instances of the file name, and remove them.
                    Note the id line that is above the line that reads the file name must also be removed.
                Run "rm changes"
                Run "/usr/local/bin/db42/db_load changes < changes.dump"
                Re-run the dump to verify your changes corrected the problem.
                    Only dump the single revision "svnadmin dump -r revision -incremental"
                Continue with the dump until another error occurs
            Error similar to file or directory [File name] could not be found
                Follow the same steps as in "v" above
                Run the entire dump, and restore it to a new repository
                Restore any files that had to be removed during the process of fixing the corruption
                    You will lose the history associated with the file that had to be removed, but you can re-import the last version of the file from the older repository.
                    Do a svn checkout of both the old, and new repositories, and do a merge between the two to try and ensure that the most recent version of the files are maintained
                Run svnadmin verify on a regular basis, and identify the issues before they get out of control.

- The contents of this email, and any attachments, are strictly private
and confidential.
- It may contain legally privileged or sensitive information and is intended
solely for the individual or entity to which it is addressed.
- Only the intended recipient may review, reproduce, retransmit, disclose,
disseminate or otherwise use or take action in reliance upon the information
contained in this email and any attachments, with the permission of
Australian Arrow Pty. Ltd.
- If you have received this communication in error, please reply to the sender
immediately and promptly delete the email and attachments, together with
any copies, from all computers.
- It is your responsibility to scan this communication and any attached files
for computer viruses and other defects and we recommend that it be
subjected to your virus checking procedures prior to use.
- Australian Arrow Pty. Ltd. does not accept liability for any loss or damage
of any nature, howsoever caused, which may result
directly or indirectly from this communication or any attached files.
Received on 2013-08-19 01:31:33 CEST

This is an archived mail posted to the Subversion Users mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.