On 02/20/15 10:21, Philip Martin wrote:
> You are making things very hard for yourself by running 1.6.11, it is
> very old and no longer maintained, unless your distribution is
> backporting fixes. A newer version will have fewer bugs and better
> performance.
You'll get no disagreement from me on that. At my last shop we made
sure to upgrade to the next major version any time the one we were
running went out of maintenance.
>
> 1.6 has a hotcopy bug
>
> http://subversion.tigris.org/issues/show_bug.cgi?id=3596
>
> the effects are hard to predict but one possibility is corrupt revisions
> when committing to the hotcopy. You could delete the file
> db/rep-cache.db from the hotcopy and see if this fixes the problem. In
> 1.8 the hotcopy bug is fixed, 1.8 also attempts to detect and avoid
> problems caused by a corrupt rep-cache.db.
This was indeed the issue. We have a cron job that runs every five
minutes that wries a timestamp into a file and checks it in in order to
exercise our CI server. This is pretty much guaranteed to fire at least
once during the hotcopy, ensuring that the rep cache will be
inconsistent with the remainder of the hotcopy. Truncating the hotcopy
cache on the slave before starting the server allowed the svnsync to
proceed as expected.
Thank you very, very much for your help,
Tony.
>
> Tony Sweeney <sweeney_at_addr.com> writes:
>
>> Hi,
>>
>> Mise en scene:
>>
>> We have three Subversion servers, version 1.6.11 running on
>> CentOS6. One is the live master and and the others are hot standbys in
>> different datacenters. The intent is that these are kept in sync with
>> the master using svnsync from a post-commit hook. One of the slaves
>> is already set up this way and has been since it was deployed by one
>> of my predecessors at this company. This is syncing nicely. The
>> other slave instance was also originally set up this way but was then
>> modified to instead use a block level filesystem mirroring technology,
>> again by one
>> of my predecessors. We have since concluded that the
>> mirroringsolution was not sufficiently trustworthy, and so we would
>> like to put it back to the svnsync method that it had run in the past.
>> The live master has in excess of 366,000 revisions and new revisions
>> come in at ~500 commits/day.
>>
>> One solution would be to simply create an empty repo, set the
>> necessary r0 revprops and hook scripts and sync from scratch.
>> However, we estimate that this would take 10-15 days due to the size
>> of the repository, so we would like to short-circuit this by starting
>> from a hotcopy. We take hotcopies nightly, save them to a backup
>> server and 'svnadmin verify' them there. I am trying to bring this
>> slave back on sync using the method described here:
>>
>> http://www.cardinalpath.com/how-to-use-svnsync-to-create-a-mirror-backup-of-your-subversion-repository/
>>
>> But using a hotcopy rather than a dump.
>>
>> The master hotcopy is restored in the correct location on the slave
>> and the hook scripts fixed up using Puppet (master and slave
>> necessarily have different hook script setups). Permissions are fixed
>> up to reflect that the hotcopy is taken as root but the server is
>> running under Apache. I set the svn:sync-last-merged-rev revprop on
>> r0 to reflect the newest revision on the hotcopy. At this point (or
>> so I thought) I should be able to replicate over the changes since
>> last night by running svnsync manually and then enable it in the hook
>> script.
>>
>> The problem is that that the revisions seem to be corrupted on
>> transfer to this slave.
>>
>> From the master:
>>
>> [root_at_uk-svn-1-new yoyodyne]# /usr/bin/svnsync --non-interactive --trust-server-
>> cert sync https://uk-svn-2-new.uk.yoyodyne.lan/svn/yoyodyne
>> --sync-username svnsync --sync-password XXXXXXXX --source-username
>> svnsync --source-password XXXXXXXX --config-dir
>> /repos/svn/yoyodyne/hooks/certs
>> Transmitting file data ..
>> Committed revision 367219.
>> Copied properties for revision 367219.
>> Transmitting file data .
>> Committed revision 367220.
>> Copied properties for revision 367220.
>> Transmitting file data .
>> Committed revision 367221.
>> Copied properties for revision 367221.
>> Transmitting file data .svnsync: Corrupt representation '367220 0 41 29949930e5
>> 5f203ff7027917e765e6ca7d 2654a46e5fff1f98b747a77ebf600c10c895fbcc367219-7wnu/_6'
>> You have new mail in /var/spool/mail/root
>> [root_at_uk-svn-1-new yoyodyne]#
>>
>> In fact, checking on the slave, they have all come across corrupt:
>>
>> [root_at_uk-svn-2-new svn]# svnadmin verify -r 367200:HEAD /repos/svn/yoyodyne
>> * Verified revision 367200.
>> * Verified revision 367201.
>> * Verified revision 367202.
>> * Verified revision 367203.
>> * Verified revision 367204.
>> * Verified revision 367205.
>> * Verified revision 367206.
>> * Verified revision 367207.
>> * Verified revision 367208.
>> * Verified revision 367209.
>> * Verified revision 367210.
>> * Verified revision 367211.
>> * Verified revision 367212.
>> * Verified revision 367213.
>> * Verified revision 367214.
>> * Verified revision 367215.
>> * Verified revision 367216.
>> * Verified revision 367217.
>> * Verified revision 367218.
>> svnadmin: Corrupt representation '367219 0 50
>> 633695f7c627c997b8b4065db2d4cb4db53c
>> 1f15b1d8cea246d731676f43b2df4c74a1710793 367218-7wnt/_d'
>> svnadmin: Malformed representation header
>> You have mail in /var/spool/mail/root
>> [root_at_uk-svn-2-new svn]#
>>
>> Both the master and working slave verify all revisions clean at all
>> times. I've tried multiple hotcopies and every time the first
>> revision that it tries to sync comes across as corrupt.
>>
>> On the master the revision file looks as follows:
>>
>> [root_at_uk-svn-1-new yoyodyne]# head -30 db/revs/367/367219
>> DELTA 366873 14907 1619
>> SVN??@
>>
>> ?=??h?&ng serialVersionUID = 1;
>> ENDREP
>> DELTA 366873 16539 1063
>> SVN?J?7?V?)|???"?Y?? ??private boolean hasMoreResults = trueif(results.size() <
>> requiredResults){
>>
>> hasMoreResults = false;
>> }
>> hasMoreResults;
>> }
>> }
>> ENDREP
>> id: 1j-366873.0-228442.r367219/295
>> type: file
>> pred: 1j-366873.0-228442.r366873/30632
>> count: 1
>> text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c dc0cc36f718b468fc08817
>> 88900f096d5cd4913f 367218-7wnq/_e
>> props: 366873 30580 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
>> cpath:
>> /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
>> copyroot: 228442 /MIME/MADb
>>
>> id: 1h-366873.0-228442.r367219/692
>> type: file
>> pred: 1h-366873.0-228442.r366873/32681
>> count: 1
>> text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c 1f15b1d8cea246d731676f43
>> b2df4c74a1710793 367218-7wnq/_d
>> props: 366873 32629 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
>> cpath:
>> /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/CachedRowList.java
>> copyroot: 228442 /MIME/MADb
>>
>> [root_at_uk-svn-1-new yoyodyne]#
>>
>> The working slave looks as follows:
>>
>> [root_at_us-svn-1 yoyodyne]# head -30 db/revs/367/367219
>> DELTA 366873 14907 1619
>> SVN??@
>>
>> ?=??h?&ng serialVersionUID = 1;
>> ENDREP
>> DELTA 366873 16539 1063
>> SVN?J?7?V?)|???"?Y?? ??private boolean hasMoreResults = trueif(results.size() <
>> requiredResults){
>>
>> hasMoreResults = false;
>> }
>> hasMoreResults;
>> }
>> }
>> ENDREP
>> id: 1j-366873.0-228442.r367219/295
>> type: file
>> pred: 1j-366873.0-228442.r366873/30631
>> count: 1
>> text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c dc0cc36f718b468fc08817
>> 88900f096d5cd4913f 367218-7yyk/_e
>> props: 366873 30579 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
>> cpath:
>> /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
>> copyroot: 228442 /MIME/MADb
>>
>> id: 1h-366873.0-228442.r367219/692
>> type: file
>> pred: 1h-366873.0-228442.r366873/32680
>> count: 1
>> text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c 1f15b1d8cea246d731676f43
>> b2df4c74a1710793 367218-7yyk/_d
>> props: 366873 32628 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
>> cpath: /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/Cach
>> edRowList.java
>> copyroot: 228442 /MIME/MADb
>>
>> [root_at_us-svn-1 yoyodyne]#
>>
>> But on the failing slave:
>> [root_at_uk-svn-2-new yoyodyne]# head -30 db/revs/367/367219
>> id: 1j-366873.0-228442.r367219/0
>> type: file
>> pred: 1j-366873.0-228442.r366873/30632
>> count: 1
>> text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c dc0cc36f718b468fc08817
>> 88900f096d5cd4913f 367218-7wnt/_e
>> props: 366873 30580 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
>> cpath:
>> /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
>> copyroot: 228442 /MIME/MADb
>>
>> id: 1h-366873.0-228442.r367219/395
>> type: file
>> pred: 1h-366873.0-228442.r366873/32681
>> count: 1
>> text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c 1f15b1d8cea246d731676f43
>> b2df4c74a1710793 367218-7wnt/_d
>> props: 366873 32629 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
>> cpath:
>> /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/CachedRowList.java
>> copyroot: 228442 /MIME/MADb
>>
>> PLAIN
>> K 19
>> CSVQueryThread.java
>> V 37
>> file 1e-366873.0-228442.r366873/31862
>> K 18
>> CachedRowList.java
>> V 35
>> file 1h-366873.0-228442.r367219/395
>> K 25
>> DBServiceQueryThread.java
>> V 33
>> [root_at_uk-svn-2-new yoyodyne]#
>>
>> Each time I've tries this the failure mode appears to be the same.
>> The initial 'DELTA <E2><80><A6> ENDREP' sections are missing from the
>> copied over file, it starts with the 'id:' line directory. Is there
>> anything I can do to debug this further?
>>
>> Tony.
>>
>
Received on 2015-02-21 12:56:49 CET