[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: svn commit: r1628093 - in /subversion/trunk/subversion/libsvn_fs_fs: index.c structure-indexes

From: Stefan Sperling <stsp_at_apache.org>
Date: Tue, 30 Sep 2014 11:15:18 +0200

On Mon, Sep 29, 2014 at 07:18:05PM +0200, Stefan Fuhrmann wrote:
> On Sun, Sep 28, 2014 at 11:17 PM, Stefan Sperling <stsp_at_apache.org> wrote:
>
> > On Sun, Sep 28, 2014 at 05:56:01PM -0000, stefan2_at_apache.org wrote:
> > > Author: stefan2
> > > Date: Sun Sep 28 17:56:01 2014
> > > New Revision: 1628093
> > >
> > > URL: http://svn.apache.org/r1628093
> > > Log:
> > > Support FSFS format 7 commits in load balanced mixed-architecture
> > clusters.
> > >
> > > At least theoretically, machines with different endianess or off_t sizes
> > > might access the same repository on e.g. an iSCSI SAN. Then the machine
> > > performing a commit may have a different architecture from the one
> > building
> > > up the transaction. To allow that, even the intermediate (proto-) index
> > > format within those transactions must be platform-independent.
> >
> > Hi Stefan,
> >
> > If you describe in some more detail how to set it up I can test this
> > scenario for you on OpenBSD with sparc64 (64bit big-endian), amd64
> > (64bit little endian), macppc (32bit big-endian) and i386 (32bit little
> > endian) machines.
> >
>
> Thanks for the offer. So, this is the issue that I addressed
> with the above patch:
>
> At the FS API level, transactions can be opened (and implicitly
> closed again) many times before being committed. If these
> operations are performed from different machines accessing
> the same repository, they might not agree upon C struct size,
> layout or interpretation. So, for pure API consistency alone,
> all data within a txn needs to be portable / architecture independent.
>
> I may be wrong but I vaguely remember that the HTTP client
> sends the "commit" operation over a separate connection that
> the one it used to build up the txn. If there is a load balancer
> in front of the actual servers, that commit may get served by
> a different machine.
>
> Testing the general scenario is simpler, though. Have one machine
> build up a transaction which touches a few of nodes (such that
> different record sizes would result in missing / misaligned data).
> Then e.g. copy the repo to a different machine and let that one
> do the commit. The result must then pass 'svnadmin verify'.
>
> The critical combinations are little / big endian and 32 bit vs.
> 64 bit file offsets (not the same thing as 32 / 64 bit in general).
> So, 64 bit big endian vs. 32 bit little endian with 32 bit off_t
> would probably cover it. Tests with both machines in both
> roles (txn creation and txn commit).
>
> -- Stefan^2.

How do you open a transaction and postpone the commit?
Using some custom code written against the FS API?
Or can some tool such as svnmucc already do this?

I presume you rely on apr_off_t, not off_t, right? I.e. it's
enough to recompile APR and SVN with or without large file
support to switch between 32bit and 64bit apr_off_t?
Received on 2014-09-30 11:15:49 CEST

This is an archived mail posted to the Subversion Dev mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.