Am 19.04.2012 14:03, schrieb Johan Corveleyn:
> On Thu, Apr 19, 2012 at 12:48 PM, Stefan Fuhrmann
> <stefanfuhrmann_at_alice-dsl.de> wrote:
>> Hi all,
>>
>> After having a closer look at merge and discussing it
>> with Julian on IRC, there seems to be no silver bullet.
>> However, we identified a few things that could be changed
>> and set of constellations that make merge harder than
>> it needs to be.
>>
>> For the first, there will be another post soon. The second
>> boils down to policy. Luckily, SVN has a mechanism to
>> enforce policies: server-side hook scripts. My proposal
>> is to develop a small set of scripts that a user can
>> combine to prevent situations that her life harder than
>> necessary. This should give us enough time to improve
>> the merge logic inside the SVN libs.
>>
>> The following pre-commit scripts / policies would be useful.
>>
>> * Common parts [not a policy]
>> We first check whether the commit contains a changed
>> svn:merge-info property. This limits the performance
>> impact on non-merge commits and we need to identify
>> all changed svn:merge-info anyway.
>>
>> Also, the merges that happened on the source branch
>> from a different location than the target branch are
>> of no interest for the policy checkers. E.g.:
>>
>> r20: merge r19 from ^/sub-branch to ^/branch
>> txn: merge r10-20 from ^/branch to ^/trunk
>> Both merges will show up in the merge-info delta but
>> we only need to evaluate the second one.
> I haven't read your mail in detail yet, but just a note in passing: in
> my experience (with a 1.5 server with FSFS-on-NFS backend) propgetting
> ('svnlook propget') every changed file during pre-commit can be very
> expensive. I ran into this with a script for enforcing our autoprops
> (like svn:eol-style=native on some files), with a combination of
> 'svnlook changed' and 'svnlook propget'. On our (admittedly not very
> fast) server, users started running into client-side timeouts because
> of this, when they committed 300 or so changed files (which happens
> easily when syncing branches). The default client-side timeout for
> neon is 30 seconds (which can be increased in the 'servers' file), and
> if the pre-commit hook takes longer than that, you're screwed.
>
> So I've optimized the pre-commit validation script to only 'propget'
> files from 'svnlook changed' which were either Added ('A'), or which
> had property modifications ('*U'). So modified files and deleted files
> don't have to be propgotten. I suppose this set of pre-commit scripts
> can use the same optimization, so it only scales with the amount of
> added and prop-changed files/dirs in the transaction.
>
> But even then, someday a user comes along that wants to add 60.000
> files (really). He's willing to split it up into commits of say 5000
> files each. The pre-commit hook takes approx 2 hours to validate this,
> so he has to set his client-side timeout very high while doing this
> (after first losing a couple of days before he figures out it's time
> to contact an admin).
Thanks for giving some hard performance numbers and
your approach to mitigate bottlenecks. As you know, I'm
constantly working on eliminating those ...
> Maybe some of this is a moot point, because:
>
> - svn+FSFS+NFS might have gotten much faster at getting props??
Random thought: is it possible to link the transactions
folder to /dev/shm even if the repository is on NFS?
> - maybe propgetting through one of the bindings is much faster than
> through svnlook, and you intend for admins to set this up with some
> language bindings? In that case, keep in mind that bindings are not
> always installed. Svnlook usually is always there.
The problem is probably that svnlook is at a hefty
disadvantage over e.g. the server because it has
to open the FSFS repo anew for each access. Don't
know really what to do about that.
> - maybe you just expect the infrastructure to be much faster?
>
>
> The problem is mainly that 'svnlook propget' doesn't support recursive
> propgetting, nor getting all props from the entire transation, or ...
> only the props for a single path. Which means you have to invoke it
> separately for every item that needs to be inspected. Maybe that can
> be improved somehow?
>
> Anyway, these are just some slightly OT random thoughts.
>
Still a good post ;)
-- Stefan^2.
Received on 2012-04-20 22:52:43 CEST