Hi Nico,
Can I start by offering to provide a trial copy of Subversion Multisite (or
even a pre-configured virtual environment to save you time) for you to prove
to yourself how we solve these challenges? Many enterprise SVN deployments
use our software and if your assertions were true that certainly wouldn't be
the case.
On Wed, Apr 27, 2011 at 12:59 AM, Nico Kadel-Garcia <nkadel_at_gmail.com>wrote:
<Liberal Snipping for attempted brevity...>
When the link between active-active servers for any
> database is broken, *or phase delayed for any reason*, and each
> database accepts alterations from clients without propagating the
> change to a fundamentally shared repository, mathematics cannot decide
> which changes must be merged, in which order.
WANdisco prevents a split brain scenario by ensuring that no writes are
possible unless an agreement has been reached. The product in fact does make
that decision and while it's probably true that it's not just a function of
pure maths, the agreement process takes care of these cases elegantly and
without any human intervention.
> Single mirrored backend database, synchronizatoin protected some sort
> of locking mechanism to prevent simultaneous commits from the multiple
> "active" front ends.
This statement doesn't sound relevant to WANdisco's technology. We don't
employ mirroring of filesystems and do not have any problems handling as
many nodes or concurrent transactions as you would conceivably want to throw
at us.
> WANdisco provide a well-written White Paper explaining this.
> >
> >
> http://www.wandisco.com/get/?f=documentation/whitepapers/WANdisco_DConE_White_Paper.pdf
>
> Just read it. It confirms my description, implemented as a clever set
> of tools to handle master/slave relationships at high speed on the
> back end.
>
Maybe we need to improve the White Paper. What you described doesn't seem to
reflect how Subversion Multisite operates at all.
In a situation where one node of three becomes unavailable the remaining two
nodes would still be able to gain a majority agreement and users of those
two nodes can continue to read and write normally. The third node where the
VPN had failed would automatically become read only and users would see an
error to that effect if they attempted a write operation. We do offer a
configuration option where that situation can be reversed, for example if
the node in question is the only active one at a particular time of day. See
the section in the Whitepaper on quorum options for more details.
The key again is that WANdisco never allows a situation to occur where there
is risk of a 'split brain'. If a global sequence number can't be generated
using one of our quorum options (Follow the sun or Majority in effect) then
the user's change is prevented before it gets to Subversion.
In your example, as soon as the VPN came back the missed transactions would
be replayed on the third node in the same order as they were on the other
two sites. No admin decisions or effort are needed here whatsoever and this
is where we guarantee that all nodes will maintain identical copies of the
data (assuming the nodes started off with the same data and have been
configured identically).
When, and how, to turn the relevant repos into read-only nodes is left
> as an exercise in resource management and paranoia. But the potential
> for fractures and divergence among them is inherent in any network of
> more than a few nodes, and switching from "active-active" to
> "active-slave" when the link is broken is begging to set up
> "slave-slave" for all sorts of confusing scenaries, and breaking the
> ability to submit code. And cleaning *UP* the mess is horrible if
> they're not set to "slave" behavior.
>
Hopefully this is now answered - There is no potential for any horrible mess
and our customers frequently go through planned and unplanned outages
without them needing to do anything at all in regard to their SVN platform.
If it's the server itself that is unavailable, users can simply svnswitch
and use a different server that can still get a quorum agreement. This is
exactly what a number of our Japanese based customers recently did following
the earthquake and need to shut down local servers to conserve power. We
also offer a third party load balancer which makes that 'failover'
transparent to end users.
> Using a "Paxos" algorithm does not solve the problem of disconnected
> nodes, unless you're reliant on hardcoded lists of active servers and
> are absolutely reliant on a majority of a generous number of
> pre-defined nodes to be available to provide that "vote". But if
> you're detached from more than half the nodes, you can only be
> read-only: nothing else is safe. Wrapping it in a set of equations
> doesn't fix that. And unless you're very careful, som idiot *will*
> rewrite local configuraitons to reduce that "half" requirement, and
> synchronization of the central list of nodes becomes absolutely
> critical.
>
>
To be clear, I said we'd based our original technology on Paxos. WANdisco's
technology (And patent) does go quite a bit further in terms of the
agreement process and again I'd encourage you to get your hand on a copy of
Subversion Multisite and prove this to yourself. Remember this is the
culmination of over 10 years research and development; you can get a lot
done in that time!
> It's workable, but potentially fragile, and it is an *old* distributed
> computing problem.
>
I hope you'll come back to this thread at some point with a changed view on
this. I believe you will find our solution robust and effective when you dig
deeper. It must be, given some of the customers and use cases we see (18
nodes in one instance, 18,000,000 transactions per day in another... I could
go on).
Best Wishes,
Ian
--
Ian Wild
Chief Solutions Architect
WANdisco, Inc.
Received on 2011-04-27 10:26:29 CEST