Hi,
Mise en scene:
We have three Subversion servers, version 1.6.11 running on CentOS6.
One is the live master and and the others are hot standbys in different
datacenters. The intent is that these are kept in sync with the master
using svnsync from a post-commit hook. One of the slaves is already set
up this way and has been since it was deployed by one of my predecessors
at this company. This is syncing nicely. The other slave instance was
also originally set up this way but was then modified to instead use a
block level filesystem mirroring technology, again by one
of my predecessors. We have since concluded that the mirroringsolution
was not sufficiently trustworthy, and so we would like to put it back to
the svnsync method that it had run in the past. The live master has in
excess of 366,000 revisions and new revisions come in at ~500 commits/day.
One solution would be to simply create an empty repo, set the necessary
r0 revprops and hook scripts and sync from scratch. However, we
estimate that this would take 10-15 days due to the size of the
repository, so we would like to short-circuit this by starting from a
hotcopy. We take hotcopies nightly, save them to a backup server and
'svnadmin verify' them there. I am trying to bring this slave back on
sync using the method described here:
http://www.cardinalpath.com/how-to-use-svnsync-to-create-a-mirror-backup-of-your-subversion-repository/
But using a hotcopy rather than a dump.
The master hotcopy is restored in the correct location on the slave and
the hook scripts fixed up using Puppet (master and slave necessarily
have different hook script setups). Permissions are fixed up to reflect
that the hotcopy is taken as root but the server is running under
Apache. I set the svn:sync-last-merged-rev revprop on r0 to reflect
the newest revision on the hotcopy. At this point (or so I thought) I
should be able to replicate over the changes since last night by running
svnsync manually and then enable it in the hook script.
The problem is that that the revisions seem to be corrupted on transfer
to this slave.
From the master:
[root_at_uk-svn-1-new yoyodyne]# /usr/bin/svnsync --non-interactive
--trust-server-
cert sync https://uk-svn-2-new.uk.yoyodyne.lan/svn/yoyodyne
--sync-username svnsync --sync-password XXXXXXXX --source-username
svnsync --source-password XXXXXXXX --config-dir
/repos/svn/yoyodyne/hooks/certs
Transmitting file data ..
Committed revision 367219.
Copied properties for revision 367219.
Transmitting file data .
Committed revision 367220.
Copied properties for revision 367220.
Transmitting file data .
Committed revision 367221.
Copied properties for revision 367221.
Transmitting file data .svnsync: Corrupt representation '367220 0 41
29949930e5
5f203ff7027917e765e6ca7d
2654a46e5fff1f98b747a77ebf600c10c895fbcc367219-7wnu/_6'
You have new mail in /var/spool/mail/root
[root_at_uk-svn-1-new yoyodyne]#
In fact, checking on the slave, they have all come across corrupt:
[root_at_uk-svn-2-new svn]# svnadmin verify -r 367200:HEAD /repos/svn/yoyodyne
* Verified revision 367200.
* Verified revision 367201.
* Verified revision 367202.
* Verified revision 367203.
* Verified revision 367204.
* Verified revision 367205.
* Verified revision 367206.
* Verified revision 367207.
* Verified revision 367208.
* Verified revision 367209.
* Verified revision 367210.
* Verified revision 367211.
* Verified revision 367212.
* Verified revision 367213.
* Verified revision 367214.
* Verified revision 367215.
* Verified revision 367216.
* Verified revision 367217.
* Verified revision 367218.
svnadmin: Corrupt representation '367219 0 50
633695f7c627c997b8b4065db2d4cb4db53c
1f15b1d8cea246d731676f43b2df4c74a1710793 367218-7wnt/_d'
svnadmin: Malformed representation header
You have mail in /var/spool/mail/root
[root_at_uk-svn-2-new svn]#
Both the master and working slave verify all revisions clean at all
times. I've tried multiple hotcopies and every time the first revision
that it tries to sync comes across as corrupt.
On the master the revision file looks as follows:
[root_at_uk-svn-1-new yoyodyne]# head -30 db/revs/367/367219
DELTA 366873 14907 1619
SVN??@
?=??h?&ng serialVersionUID = 1;
ENDREP
DELTA 366873 16539 1063
SVN?J?7?V?)|???"?Y?? ??private boolean hasMoreResults =
trueif(results.size() <
requiredResults){
hasMoreResults = false;
}
hasMoreResults;
}
}
ENDREP
id: 1j-366873.0-228442.r367219/295
type: file
pred: 1j-366873.0-228442.r366873/30632
count: 1
text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c
dc0cc36f718b468fc08817
88900f096d5cd4913f 367218-7wnq/_e
props: 366873 30580 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
copyroot: 228442 /MIME/MADb
id: 1h-366873.0-228442.r367219/692
type: file
pred: 1h-366873.0-228442.r366873/32681
count: 1
text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c
1f15b1d8cea246d731676f43
b2df4c74a1710793 367218-7wnq/_d
props: 366873 32629 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/CachedRowList.java
copyroot: 228442 /MIME/MADb
[root_at_uk-svn-1-new yoyodyne]#
The working slave looks as follows:
[root_at_us-svn-1 yoyodyne]# head -30 db/revs/367/367219
DELTA 366873 14907 1619
SVN??@
?=??h?&ng serialVersionUID = 1;
ENDREP
DELTA 366873 16539 1063
SVN?J?7?V?)|???"?Y?? ??private boolean hasMoreResults =
trueif(results.size() <
requiredResults){
hasMoreResults = false;
}
hasMoreResults;
}
}
ENDREP
id: 1j-366873.0-228442.r367219/295
type: file
pred: 1j-366873.0-228442.r366873/30631
count: 1
text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c
dc0cc36f718b468fc08817
88900f096d5cd4913f 367218-7yyk/_e
props: 366873 30579 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
copyroot: 228442 /MIME/MADb
id: 1h-366873.0-228442.r367219/692
type: file
pred: 1h-366873.0-228442.r366873/32680
count: 1
text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c
1f15b1d8cea246d731676f43
b2df4c74a1710793 367218-7yyk/_d
props: 366873 32628 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/Cach
edRowList.java
copyroot: 228442 /MIME/MADb
[root_at_us-svn-1 yoyodyne]#
But on the failing slave:
[root_at_uk-svn-2-new yoyodyne]# head -30 db/revs/367/367219
id: 1j-366873.0-228442.r367219/0
type: file
pred: 1j-366873.0-228442.r366873/30632
count: 1
text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c
dc0cc36f718b468fc08817
88900f096d5cd4913f 367218-7wnt/_e
props: 366873 30580 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
copyroot: 228442 /MIME/MADb
id: 1h-366873.0-228442.r367219/395
type: file
pred: 1h-366873.0-228442.r366873/32681
count: 1
text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c
1f15b1d8cea246d731676f43
b2df4c74a1710793 367218-7wnt/_d
props: 366873 32629 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/CachedRowList.java
copyroot: 228442 /MIME/MADb
PLAIN
K 19
CSVQueryThread.java
V 37
file 1e-366873.0-228442.r366873/31862
K 18
CachedRowList.java
V 35
file 1h-366873.0-228442.r367219/395
K 25
DBServiceQueryThread.java
V 33
[root_at_uk-svn-2-new yoyodyne]#
Each time I've tries this the failure mode appears to be the same. The
initial 'DELTA <E2><80><A6> ENDREP' sections are missing from the copied
over file, it starts with the 'id:' line directory. Is there anything I
can do to debug this further?
Tony.
Received on 2015-02-20 08:55:07 CET