Broken Revision in FSFS Repo

Hello,

After some time it turned out, that there are few revisions in our
repository, which are broken, probably on the filesystem level.
Unfortunetely as time went by, backups we made contain only those
broken revisions so we have no chance in getting stuph working just by
simple recovery. Only hope is to fix the repository in some way. At
this point in time it is impossible to do a full checkout.

% svnadmin verify /storage/svn
[...]
* Verified revision 1025.
* Verified revision 1026.
svnadmin: Decompression of svndiff data failed

This is the most common error, which we get while trying to checkout the stuph.

fsfsverify.py tool points out, that revision 1027 is broken in several
places. Unfortunetely after few fixes the tool crashes on the 1027
rev.

% fsfsverify.py /storage/svn/db/revs/1/1027

NodeRev Id: 2z-1027.0-68.r1027/2303848
 type: file
 text: DELTA 1027 2088431 1684 5317 447a4956b16f81132e54669d33c188f2
 cpath: /_Projekty/Rekl/dd.aspx.resx
 copyroot: 68 /_Projekty
Traceback (most recent call last):
  File "/usr/local/bin/fsfsverify.py", line 1134, in <module>
    options.dumpWindows)
  File "/usr/local/bin/fsfsverify.py", line 571, in verify
    svndiff = Svndiff(f, self.length)
  File "/usr/local/bin/fsfsverify.py", line 461, in __init__
    (self.startingOffset)
__main__.InvalidSvndiffHeader: Invalid svndiff header at offset 2088437

We tried hard to get this thing working, but in the end I have no idea
why it is failing and how to fix it.

Next thing we tried to do, is to create dumps without broken
revisions. During the process it turned out, that out of over 8500
revs 5 is broken. Using svnadmin dump --incremental we managed to take
all the working revs out, but we have no idea how to put them
together.

Creating new repo from scratch and importing dumps doesn't work.
Leading dump is loaded without any problem (0:1026), but the 2nd dump
(1028:6675) fails at 2761 with following Error:

<<< Started new transaction, based on original revision 2763
svnadmin: File not found: revision 2761, path '/_Projekty/Rekl
     * adding path : _Projekty/www/Rekl ...#

No revision over 2761 is inserted.

After a bit more gooling we found out a tool called svndumptool
(0.6.1). Merge was looking very promissing at the start, but it turned
out it fails now and then on certain revisions. Revisions it fails on
are seem to be fine in the original repo, but for some reason it just
wont work. At the start we tried to skip the revisions which aren't
seem to be loaded correctly. After some time it turned out, that
around rev 7000 every 1 in 5 revisions was broken. It makes no sence
to make a recovery, while we have to skip 20% or more of the data.

Revision: 7192 from r7192 all-dump-1:7202.txt
Revision: 7193 from r7193 all-dump-1:7202.txt
Revision: 7194 from r7194 all-dump-1:7202.txt
Revision: 7195 from r7204 7204:7269.txt
Revision: 7196 from r7205 7204:7269.txt
Revision: 7197 from r7206 7204:7269.txt
Revision: 7198 from r7207 7204:7269.txt
Revision: 7199 from r7208 7204:7269.txt
Revision: 7200 from r7209 7204:7269.txt
Revision: 7201 from r7210 7204:7269.txt
Traceback (most recent call last):
  File "/tmp/svndumptool-0.6.1/svndumptool.py", line 119, in <module>
    sys.exit( func( appname, args ) )
  File "/tmp/svndumptool-0.6.1/svndump/merge.py", line 564, in
svndump_merge_cmdline
    merge.merge()
  File "/tmp/svndumptool-0.6.1/svndump/merge.py", line 257, in merge
    self.__copy_revision( oldestIndex )
  File "/tmp/svndumptool-0.6.1/svndump/merge.py", line 291, in __copy_revision
    newNode = self.__change_node( dumpIndex, node )
  File "/tmp/svndumptool-0.6.1/svndump/merge.py", line 330, in __change_node
    newFromRev = self.__in_rev_nr_maps[dumpIndex][fromRev]
KeyError: 7188

I have no idea why it has to do something wih rev 7188, which was
imported earlier. If I now skip rev 7203 it will move on, but it will
fail again somewhere around 7210.

Does any one have any tips on how to recover this repository?

Thanks in advance!

-- 
Mariusz Drozdziel