Hi,
> I am assuming that if the commits must start at least one
> second apart - then the sync from the post-commit hook
> would not be able to reach a race condition.
> Is this a reasonable assumption?
No, the bug is worse than that. Suppose there are 3 commits:
- At time 12:00:00, a commit starts sync process #1. The sync
takes 6 seconds.
- At time 12:00:02, a commit starts sync process #2. This blocks
due to sync process #1's lock.
- At time 12:00:04, a commit starts sync process #3. This blocks
due to sync process #1's lock.
- At time 12:00:06, sync process #1 finishes. Sync processes #2 and
#3 both try to take the lock; due to the bug they may _both_
succeed in taking the lock. Chaos ensues.
I suggest you use the "flock(1)" tool. [1]. This is installed as
a standard part of Debian (it's in the util-linux package).
Something like this, in your post-commit hook:
--- cut here - start ---
#!/bin/sh
/usr/bin/flock --wait 1200 \
-x /var/lock/svn_sync_lock \
/usr/local/bin/svnsync sync --non-interactive \
http://mirrorserver.example.com/svn &
--- cut here - end ---
You will need to make the /var/lock/svn_sync_lock file and ensure
it's writable by the user your post-commit hook is running as.
"flock" is a mature, tested piece of code to handle locking.
It will ensure that only one copy of svnsync is running at a
time. That way, the race condition in svnsync is avoided.
Kind regards,
Jon
[1] http://www.google.co.uk/search?q=man+flock%281%29
-----Original Message-----
From: Andersen, Krista [mailto:Krista.Andersen_at_itg.com]
Sent: 15 January 2010 22:29
To: Jon Foster; users_at_subversion.apache.org
Cc: ssi-svn_admin
Subject: RE: sync bug -> corrupted proxy repo
Thank you Jon, for your explanation and workaround.
Are there any "best practices" that we can advise our dev groups to
follow to avoid this problem?
Otherwise, your suggestions seem to indicate I would have to run the
sync on a cronjob and not with the hook script. That is something we
would like to avoid. So I have added a start time comparison and sleep
in a start-commit hook instead. Do you see any reason why this would
cause other problems?
I am assuming that if the commits must start at least one second apart -
then the sync from the post-commit hook would not be able to reach a
race condition. Is this a reasonable assumption?
#!/usr/bin/sh
# START-COMMIT HOOK
# kanderse Jan 13, 2010
# The start-commit hook is invoked before a Subversion txn is created
# in the process of doing a commit.
# This script checks the start time and compares with the start time
# of the previous commit. It will cause a commit to wait one second if
# the last commit was started less than one second earlier.
# The purpose of this wait is to prevent known issue 3546 [1][2].
# a race condition involving multiple sync processes running at
# the same time that result in a corrupted proxy.
REPOS="$1"
USER="$2"
DATE1=`cat /$REPOS/hooks/start-time.txt` # previous start time
DATE2=`/usr/local/bin/date +%s` # record current start time
echo $DATE2 > /$REPOS/hooks/start-time.txt
# echo $DATE2 $DATE1 `expr $DATE2 - $DATE1`
if [ `expr $DATE2 - $DATE1` -lt 1 ]
then sleep 1 # to prevent sync race that results in sync duplication and
corrupted proxy
fi
# All checks passed, so allow the commit.
exit 0
Krista Andersen
Global Development Infrastructure
Investment Technology Group, Inc.
400 Corporate Pointe, 8th Floor
Culver City, CA 90230
Direct: 213.270.7570
-----Original Message-----
From: Jon Foster [mailto:Jon.Foster_at_cabot.co.uk]
Sent: Wednesday, January 13, 2010 5:13 AM
To: Andersen, Krista; users_at_subversion.apache.org
Cc: ssi-svn_admin
Subject: RE: sync bug -> corrupted proxy repo
Hi,
Andersen, Krista [mailto:Krista.Andersen_at_itg.com] wrote:
> Twice I have seen one of my proxy repositories become corrupted due
> to an apparent bug in the svnsync sync process. Has anyone else
> seen this type of behavior from Subversion?
This is probably caused by issue 3546 [1][2]. This is a race
condition - if you have several sync processes running at the same
time then the mirror can get corrupted. You had three commits which
were 1 second apart, so your hook script started 3 copies of svnsync
within 2 seconds. I think this is the first practical report of this
bug; previous discussion was theoretical.
> Here is a comparison the output of the svn log -v for the offending
> revisions (324,325) on both the corrupted and non-corrupted proxy
> repo.
It looks like rev 323 got mirrored twice (as mirror revs 323 and 324),
then rev 324 was mirrored (as mirror rev 325).
> I am a bit concerned about the stability of Subversion since this
> is the second time in two months that I have had to fix this issue.
> Is there a patch or something to prevent this in the future?
Suggested workaround: Change your hook scripts to use the lockf or
lockfile tools[3] to ensure that only one instance of svnsync runs
at once.
Kind regards,
Jon
[1]
http://mail-archives.apache.org/mod_mbox/subversion-dev/200911.mbox/%3C2
0091127115356.GC9302_at_jack.stsp.name%3E
[2] http://subversion.tigris.org/issues/show_bug.cgi?id=3546
[3]
http://mail-archives.apache.org/mod_mbox/subversion-dev/200911.mbox/%3C2
0091127132659.GE9302_at_jack.stsp.name%3E
**********************************************************************
This email and its attachments may be confidential and are intended
solely for the use of the individual to whom it is addressed. Any views
or opinions expressed are solely those of the author and do not
necessarily represent those of Cabot Communications Ltd.
If you are not the intended recipient of this email and its attachments,
you must take no action based upon them, nor must you copy or show them
to anyone.
Cabot Communications Limited
Verona House, Filwood Road, Bristol BS16 3RY, UK
+44 (0) 1179584232
Co. Registered in England number 02817269
Please contact the sender if you believe you have received this email in
error.
**********************************************************************
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
-+-
This message is for the named person's use only. This communication is
for
informational purposes only and has been obtained from sources believed
to
be reliable, but it is not necessarily complete and its accuracy cannot
be
guaranteed. It is not intended as an offer or solicitation for the
purchase
or sale of any financial instrument or as an official confirmation of
any
transaction. Moreover, this material should not be construed to contain
any
recommendation regarding, or opinion concerning, any security. It may
contain confidential, proprietary or legally privileged information. No
confidentiality or privilege is waived or lost by any mistransmission.
If
you receive this message in error, please immediately delete it and all
copies of it from your system, destroy any hard copies of it and notify
the
sender. You must not, directly or indirectly, use, disclose, distribute,
print, or copy any part of this message if you are not the intended
recipient. Any views expressed in this message are those of the
individual
sender, except where the message states otherwise and the sender is
authorized to state them to be the views of any such entity.
Securities products and services provided to Canadian investors are
offered
by ITG Canada Corp. (member CIPF and IIROC - Investment Industry
Regulatory
Organization of Canada), an affiliate of Investment
Technology Group, Inc.
ITG Inc. and/or its affiliates reserves the right to monitor and archive
all electronic communications through its network.
ITG Inc. Member FINRA, SIPC
-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
-+-
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
**********************************************************************
This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Cabot Communications Ltd.
If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone.
Cabot Communications Limited
Verona House, Filwood Road, Bristol BS16 3RY, UK
+44 (0) 1179584232
Co. Registered in England number 02817269
Please contact the sender if you believe you have received this email in error.
**********************************************************************
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
Received on 2010-01-18 14:26:04 CET