Weird "Corrupt representation / Malformed representation header" Errors on fsfs repo.

Hi,

one of our large repositories went corrupt. Maybe someone can help.
On Aug 6th we had a Hardware crash of our svn server (mod_dav_svn) / svn 1.7.11.

This caused erros like:
[Fri Aug 08 14:00:01 2014] [error] [client ip.ip.ip.ip] (20014)Internal error: Couldn't open rep-cache database
[Fri Aug 08 14:00:01 2014] [error] [client ip.ip.ip.ip] (20014)Internal error: -Couldn't perform atomic initialization
[Fri Aug 08 14:00:01 2014] [error] [client ip.ip.ip.ip] (20014)Internal error: -database disk image is malformed, executing statement 'PRAGMA synchronous=OFF;PRAGMA recursive_triggers=ON;'

This led me to:
https://mail-archives.apache.org/mod_mbox/subversion-users/201401.mbox/%3C52D83701.5030201@reser.org%3E

As Restoring a dump of the prod-repo would Take about >=8 hours I found a workaround:
Taking a light older backup, restoring it on another place and replacing the corrupt rep-cache.db on the production repo and restarting apache httpd...
It seems to have worked. None of the error above appears anymore and the rep-cache.db file grows. :)

----------------

Sadly there is a new error where I don't know if the previous issue is linked to this - that's why I explained it. :)

Now the httpd-log presents Errors at Checkout like:
[Wed Aug 13 17:32:27 2014] [error] [client ip.ip.ip.ip] Unable to deliver content. [500, #0]
[Wed Aug 13 17:32:27 2014] [error] [client ip.ip.ip.ip] could not prepare to read the file [500, #160004]
[Wed Aug 13 17:32:27 2014] [error] [client ip.ip.ip.ip] Corrupt representation '76625 15736 24 1736 98aa6ec0f63f0cf7bc71f84cb15decf5 106c014975275e25ea0a7efab549472a8e32917e 77054-1ogo/_4x' [500, #160004]
[Wed Aug 13 17:32:27 2014] [error] [client ip.ip.ip.ip] Malformed representation header at /srv/svn/repo1/db/revs/76/76625:15751 [500, #160004]

Scanning the last revisions throws:
> for i in $(seq 77000 $(svnlook youngest /srv/svn/repo1)); do svnadmin verify /srv/svn/repo1 -r $i; done
[...]
* Verified revision 77014.
svnadmin: E160004: Corrupt representation '75477 1973 1105 7151 a6e0d04828d8d5683c69faf7289c62e0 96f59510929937634de7c7fea27a005db91a31c9 77014-1ofa/_g'
svnadmin: E160004: Malformed representation header at /srv/svn/repo1/db/revs/75/75477:1982
* Verified revision 77016.
...
* Verified revision 77047.
svnadmin: E160004: Corrupt representation '76625 48 29 7011 c2fbbb12f89423213f919c53b926fa82 fc3aba63eaa5672fe664fd47ae5f6db780b9552c 77047-1ogg/_a'
svnadmin: E160004: Malformed representation header at /srv/svn/repo1/db/revs/76/76625:52
* Verified revision 77049.
[...]
* Verified revision 77054.
svnadmin: E160004: Corrupt representation '76625 48 29 7011 c2fbbb12f89423213f919c53b926fa82 fc3aba63eaa5672fe664fd47ae5f6db780b9552c 77054-1ogo/_4w'
svnadmin: E160004: Malformed representation header at /srv/svn/repo1/db/revs/76/76625:52
* Verified revision 77056.
[...]

So revisions: 77015, 77048 and 77055 are corrupt - others are okay - so just commiting seems not to be the problem and thus I think it's not linked to the first one. All corrupt commits come from the same user account (authzn_svn_module) I am going to check his environment.

Researching I found:
https://mail-archives.apache.org/mod_mbox/subversion-dev/201010.mbox/%3C1286360504.2313.132.camel@edith%3E

trying the script sadly throws:
./fixer/fix-rev.py /srv/svn/repo1 77015
Traceback (most recent call last):
  File "./fixer/fix-rev.py", line 237, in <module>
    fix_rev(repo_dir, rev)
  File "./fixer/fix-rev.py", line 222, in fix_rev
    while fix_one_error(repo_dir, rev):
  File "./fixer/fix-rev.py", line 182, in fix_one_error
    if handle_one_error(repo_dir, rev, svnadmin_err):
  File "./fixer/fix-rev.py", line 161, in handle_one_error
    fix_delta_ref(repo_dir, rev, bad_rev, bad_offset, bad_size)
  File "./fixer/fix-rev.py", line 111, in fix_delta_ref
    good_offset = find_good_rep_header(repo_dir, bad_rev, bad_size)
  File "/root/scripts/fixer/find_good_id.py", line 72, in find_good_rep_header
    _, texts = rev_file_indexes(repo_dir, rev)
  File "/root/scripts/fixer/find_good_id.py", line 41, in rev_file_indexes
    for line in open(rev_file_path(repo_dir, rev)):
IOError: [Errno 2] No such file or directory: '/srv/svn/repo1/db/revs/75477

doing a symlink throws following:
]# ln -s /srv/svn/repo1/db/revs/75/75477 /srv/svn/repo1/db/revs/75477
]# ./fixer/fix-rev.py /srv/svn/repo1 77015
Traceback (most recent call last):
  File "./fixer/fix-rev.py", line 237, in <module>
    fix_rev(repo_dir, rev)
  File "./fixer/fix-rev.py", line 222, in fix_rev
    while fix_one_error(repo_dir, rev):
  File "./fixer/fix-rev.py", line 182, in fix_one_error
    if handle_one_error(repo_dir, rev, svnadmin_err):
  File "./fixer/fix-rev.py", line 161, in handle_one_error
    fix_delta_ref(repo_dir, rev, bad_rev, bad_offset, bad_size)
  File "./fixer/fix-rev.py", line 111, in fix_delta_ref
    good_offset = find_good_rep_header(repo_dir, bad_rev, bad_size)
  File "/root/scripts/fixer/find_good_id.py", line 72, in find_good_rep_header
    _, texts = rev_file_indexes(repo_dir, rev)
  File "/root/scripts/fixer/find_good_id.py", line 44, in rev_file_indexes
    id_noderev, id_rev, _ = parse_id(id)
  File "/root/scripts/fixer/find_good_id.py", line 27, in parse_id
    _, rev = noderev.split('.r')
ValueError: too many values to unpack

So script seems to be too old :|

I also tried ( from http://www.szakmeister.net/fsfsverify/ ):
python fsfsverify.py /srv/svn/repo1/db/revs/75/75477
but the output tells a lot of "Can't check <cryptic string>" like: "b-4028.q-73874.r73874/36007" and it does not seem to get fixed, too. (svnadmin verify throws same above)

Now I am at the edge of my competence. Hopefully someone of you can help!? Do you have an Idea how to fix that?
As far as I know the commiting guys use a lot of git-svn - if this is a needful information...

Thank you very much.
Felix

---
T-Systems Multimedia Solutions GmbH
CU BT
Felix Herzog 
Centralized Applicationmanagement
Address: Riesaer Straße 5, 01129 Dresden, Germany  
Postal address: Postfach 10 02 24, 01072 Dresden, Germany
+49 351 2820-2593 (Phone)
+49 351 2820-5111 (Fax)
+49 151 1483-1647 (Mobile)
E-Mail: E-Mail: felix.herzog_at_t-systems.com
Internet: http://www.t-systems-mms.com
You can find the compulsory statement on: www.t-systems-mms.com/en/compulsory-statement