I'd like to weigh in with some perf stats on that ever-changing 15GB file,
but I can't:
With TMPDIR same drive as Svn's root dir: sometime 6mins, sometimes 14
mins for the PUT.
With TMPDIR different drive as Svn's root dir: sometime 6mins, sometimes
14 mins for the PUT.
I can't prove or disprove the benefit of that env var setting.
I have compression-level = 0 at the moment, and have filled the drive to
the 3.4TB level (now full) which isn't pertinent to the question posed, I
guess.
With such oscillation on times for back to back PUTs on that v large
resource, when nothing else is changing, I can't come back with meaningful
stats after a build of Svn rom HEAD or trunk/ :-(
Great work Evgeny. Is the implementation toggled somehow in the build, or
another property? I looked through the diffs but couldn't see anything
that stuck out.
- Paul
On Fri, Jul 14, 2017 at 7:38 AM, Johan Corveleyn <jcorvel_at_gmail.com> wrote:
> On Fri, Jul 14, 2017 at 1:18 PM, Evgeny Kotkov
> <evgeny.kotkov_at_visualsvn.com> wrote:
> > Paul Hammant <paul_at_hammant.org> writes:
> >
> >> 1. compression-exempt-suffixes = mp3,mp4,jpeg
> >>
> >> 2. deltification-exempt-suffixes = mp3,mp4,jpeg
> >>
> >> Regardless of the setting of 'compression-level', #1 above two mean
> certain
> >> things can skip the compression attempt. It must give up at a certain
> point
> >> right?
> >
> > Hi everyone,
> >
> > To improve the situation with slow commits of large binary and, possibly,
> > incompressible files I committed a patch (http://svn.apache.org/r1801940
> )
> > that adds initial support for LZ4 compression in the backend. While
> still
> > providing a decent compression ratio, LZ4 offers much faster compression
> > even than zlib with level=1, and can skip incompressible data chunks.
> > (Presumably, LZ4 is used for on-the-fly compression in different file
> > systems for these reasons.)
> >
> > I have seen significant (up to 3 times) speed improvement for svn import
> > and commit with large binary files. Sometimes, using LZ4 compression can
> > even outperform a configuration with disabled compression, if the file is
> > at least somehow compressible, as it reduces the time required to write
> > the data to disk.
> >
> > Currently, LZ4 compression is enabled if the fsfs.conf file specifies
> > compression-level=1, and all other levels still use zlib for compression.
> > Right now, the support for LZ4 is only implemented for the file://
> protocol,
> > but support/negotiation for other protocols can be added later.
>
> Nice!
>
> But how about deltification? Has anyone tried / benchmarked the effect
> of turning off deltification (with or without compression), to see
> what the effect would be on the commit time? Like I suggested in this
> thread yesterday (i.e. set max-deltification-walk to 0 in fsfs.conf --
> or perhaps play with both max-deltification-walk and
> max-linear-deltification) ...
>
> --
> Johan
>
Received on 2017-07-14 13:59:52 CEST