RE: Issue while loading the SVN Dump SVN version 1.9.7

Hi Johan,

Sorry for the delayed response as I was in long vacation.
Yes, I don’t think we are hitting the real sha-1 collision in our repository and as you said it might be another bug in the sha-1 collision detection code.
I am sure that no one committed the sha-1-colliding files from https://shattered.io
Our repository is very huge with 1 Million + revisions and I am seeing the issue only while loading the specific revision 724413. I am able to load the revisions before and after it.
As you suggested I have now loaded the dump till 724412 which is the revision before the problematic one with rep sharing enabled.
It's not allowing me to load the problematic revision with rep-sharing disabled and getting below message.

FYI,

[root_at_vitech-svn-slave-test-01 svndump]# svnadmin load /u01/svn/repos < incdump724413.txt
svnadmin: E200002: Error while parsing config file: /u01/svn/repos/db/fsfs.conf:
svnadmin: E200002: line 39: Option expected
[root_at_vitech-svn-slave-test-01 svndump]#

If I re-enable the rep-sharing and trying to load the problematic revision, I get the error while loading the specific file"SecurityServiceImpl.java" of that revision number. Please let me know if you need any further details.
Just to reiterate my source SVN is 1.9.5 and target is 1.9.7 version

FYI,

[root_at_vitech-svn-slave-test-01 svndump]# svnadmin load /u01/svn/repos < incdump724413.txt
<<< Started new transaction, based on original revision 724413
     * editing path : v3-web-product/branches/8.3.0.279_IPERS/benefits/Scripts/DML_Exception/web_dml_admin_businessEntityRelationsHistory.sql ... done.
     * editing path : v3-web-product/branches/8.3.0.279_IPERS/benefits/Scripts/DML_Exception/web_dml_admin_securityApplicationUserHistory.sql ... done.
     * editing path : v3-web-product/branches/8.3.0.279_IPERS/benefits/Scripts/DML_Exception/web_dml_admin_securityRoleUserHistory.sql ... done.
     * editing path : v3-web-product/branches/8.3.0.279_IPERS/benefits/Scripts/DML_Exception/web_dml_admin_userMaintenance_audit_Labels.sql ... done.
     * editing path : v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/BusinessEntityRelationHist.hbm ... done.
     * editing path : v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/BusinessEntityRelationHist.java ... done.
     * editing path : v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/SecurityApplicationUsrHist.hbm ... done.
     * editing path : v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/SecurityApplicationUsrHist.java ... done.
     * editing path : v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/SecurityRoleUserHist.hbm ... done.
     * editing path : v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/SecurityRoleUserHist.java ... done.
     * editing path : v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/base/BaseBusinessEntityRelationHist.java ... done.
     * editing path : v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/base/BaseSecurityApplicationUsrHist.java ... done.
     * editing path : v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/model/base/BaseSecurityRoleUserHist.java ... done.
     * editing path : v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/services/SecurityService.java ... done.
     * editing path : v3-web-product/branches/8.3.0.279_IPERS/benefits/src/com/vitechinc/v3/admin/services/SecurityServiceImpl.java ...svnadmin: E160000: SHA1 of reps '669684 7 1143 195972 efbdb058ce857b2860cfa245f014f0b9 9db457be74545c184242e57208bf1d56db1f15b2 724412-fiyk/_16' and '-1 0 929 195972 efbdb058ce857b2860cfa245f014f0b9 9db457be74545c184242e57208bf1d56db1f15b2 724412-fiyk/_16' matches (9db457be74545c184242e57208bf1d56db1f15b2) but contents differ
svnadmin: E160004: Filesystem is corrupt
svnadmin: E200014: Checksum mismatch while reading representation:
   expected: efbdb058ce857b2860cfa245f014f0b9
     actual: 04a53277f405bcbab8a3b1fff9f8c6e0

Thanks,
Santosh.
-----Original Message-----
From: Johan Corveleyn [mailto:jcorvel_at_gmail.com]
Sent: Thursday, January 25, 2018 4:27 PM
To: Santosh Kondapuram <SKondapuram_at_vitechinc.com>
Cc: users_at_subversion.apache.org
Subject: Re: Issue while loading the SVN Dump SVN version 1.9.7

On Thu, Jan 25, 2018 at 10:40 AM, Santosh Kondapuram <SKondapuram_at_vitechinc.com> wrote:
> Hi Johan,
>
> I have tried reloading the dump as you suggested with -M 0 option but still I am running in to the same issue.
> Seems like the svn admin could not load the dump due to sha1 collision files. So the question is how do I identify the sha1 colliding files ? does the error stack trace say something about it ?
> As per the sha1-advisory note if we create a Subversion permission rule(authz) that block access to the one or both of the files, then tools like svnsync will not send the colliding content.
> So If someone can help me in identifying the colliding files then I would like to block them and sync my mirror repository. Below is the error I am hitting.
> Any help much appreciated and I am very much new to the subversion.
>
> https://subversion.apache.org/security/sha1-advisory.txt
>
> FYI,
> svnadmin: E160000: SHA1 of reps '669684 7 1143 195972
> efbdb058ce857b2860cfa245f014f0b9
> 9db457be74545c184242e57208bf1d56db1f15b2 724412-fiyk/_16' and '-1 0
> 929 195972 efbdb058ce857b2860cfa245f014f0b9
> 9db457be74545c184242e57208bf1d56db1f15b2 724412-fiyk/_16' matches
> (9db457be74545c184242e57208bf1d56db1f15b2) but contents differ
> svnadmin: E160004: Filesystem is corrupt
> svnadmin: E200014: Checksum mismatch while reading representation:
> expected: efbdb058ce857b2860cfa245f014f0b9
> actual: 04a53277f405bcbab8a3b1fff9f8c6e0
>
> Thanks,
> Santosh.

Hi Santosh,

Hmmm, I chatted a bit about this on irc with Andreas Stieger. It seems unlikely that you have a real sha-1 collision in your repository.
Unless someone committed those on purpose (for instance by committing files from https://shattered.io/). It's also possible that there is another bug in the "sha-1 collision detection code" (one that doesn't get bypassed with the -M0 trick).

So, are you sure someone committed such sha-1-colliding files?

Also: the revision where it fails, is that the last revision in the dumpstream?

Something you could try to get further, or to get more information about what's going on: if you disable rep-sharing SVN should be able to store the sha-1 collision (you won't be able to check out both colliding files in a working copy though). So you could 'svnadmin load' until the revision right before the problematic one, then disable rep-sharing (putting the option enable-rep-sharing = false in db/fsfs.conf), then load the problematic revision. Then you might be able to find out at least the name of the file that's causing the problem.

HTH,

--
Johan
This e-mail message and any files transmitted with it may contain confidential and proprietary information and are intended solely for the use of the individual or entity to which they are addressed. Any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this e-mail in error please notify the sender by reply email and destroy all copies of the original message. Thank you for your cooperation.