Malcolm Rowe <malcolm-svn-dev@farside.org.uk> writes:
> I've been thinking a bit recently about FSFS's scalability and
> performance, and there are two non-backward-compatible things that I'd
> like to be able to implement.
>
> The first is the ability to split your revs/ and revprops/ directories
> into separate 'buckets' or 'shards', so that we don't require a
> repository with a million revs to contain a million files in one
> directory.
Why would this be this non-backward-compatible? I can see why it's
not forward-compatible, but backward-compatibility should be possible
here.
> Okay, that's the first thing I'd like to do. The second is a little
> more specialised.
>
> [...]
>
> (Opening and closing files over a network file system - and in particular,
> NFS - really really sucks, in case you didn't know.)
>
> Now some of those problems I've identified as fixable, and so I hope to
> be able to fix them in due course. The main problems though, are design
> ones, and very much harder to fix - the noderevs need to be marshalled
> to disk and back again several times, for example.
>
> So I've a much simpler way to fix this problem: allow the repository
> administrator to specify an area of the _local_ filesystem to use for
> marshalling transaction data. Only the proto-rev and revprops files
> actually need to be stored on the NAS: the others can just remain on the
> local disk until it's time to do the final commit.
Nice workaround!
> (So we might be writing to, say, /var/tmp/myrepo/1-1.txn/node._1.0 and
> /nfs/repos/myrepo/db/transactions/1-1.txn/rev).
>
> But again, this isn't suitable for everyone. And in this case, there's
> not even a reasonable default location for the local storage.
>
> So, here's the plan:
>
> - Accept FSFS filesystem options at 'svnadmin create' time.
> (perhaps in the cases above we'd name them --fsfs-max-files-per-dir=N
> and --fsfs-local-txn-dir=/foo).
Are you sure max-files-per-dir has to be configurable? If we're going
to have the code in there anyway, would it be possible to just pick
some reasonable values and then make this the new default for FSFS?
(With the old format still supported, of course.) Are you sure it
would be so much less efficient than the current storage mechanism
that we need to retain the option of not sharding?
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Mon Mar 5 09:09:15 2007