[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: LZ4 compression

From: Evgeny Kotkov <evgeny.kotkov_at_visualsvn.com>
Date: Mon, 17 Jul 2017 14:54:59 +0200

LZ4 offers much faster decompression than zlib, and read operations should
benefit from this change as well. (Don't have the exact numbers available
here on my phone, sorry for that.)

https://svn.apache.org/r1801974 adds negotiation and support for LZ4 in
mod_dav_svn and in ra_serf. Except for the PUT requests, which require a
couple of tweaks to the negotiation scheme, with this change svndiff2 with
LZ4 will be used in the repository on-disk data and as the wire format for
http://, if the corresponding compression level is set to 1.

Speaking of only using it with compression level 1, that's not a starting
point. While LZ4 offers superior speeds, it is not a substitute for any
zlib compression level > 1, including our current default of 5, as the
latter gives better compression ratio.

I was thinking that it might make sense to make the compression level 1 our
new default, so that new installations would benefit from the increased
speed, while still keeping a decent compression ratio. (If necessary, that
could still be tweaked for better compression.)

Regards,
Evgeny Kotkov

[On vacation, from the mobile; please excuse top-posting and brevity.]

On Monday, July 17, 2017, Julian Foad <julianfoad_at_apache.org> wrote:

> Evgeny Kotkov wrote (in the 'Proposal: new fsfs.conf properties' thread):
>
>> To improve the situation with slow commits of large binary and, possibly,
>> incompressible files I committed a patch (http://svn.apache.org/r1801940)
>> that adds initial support for LZ4 compression in the backend.
>>
>
> This sounds amazing!
>
> and in the log message:
>
>> [...] The interoperability is implemented by bumping the format of >
>> svndiff to 2 and the repository file system format to 8. [...]
>>
>
> Can you state simply what facets of Subversion this benefits so far? It
> looks like it is the compression/deltification internal to FSFS, stored in
> rev files, and not exposed outside FSFS, so it affects server speed (and
> thus commit speed) and server storage space. Any significant effect on
> speed of reading the repo (decompression)?
>
> What parts of Subversion could this usefully be extended to affect?
> (Speed of compression/deltification performed on the client for commits?
> Size reduction of data sent over the wire to/from new enough clients?)
>
> Currently, LZ4 compression is enabled if the fsfs.conf file specifies
>> compression-level=1, and all other levels still use zlib for compression
>>
>
> Is this just a 'safe' starting point for testing and will we likely change
> this to use LZ4 for all 'compression-level' settings?
>
> - Julian
>
>
Received on 2017-07-17 14:55:04 CEST

This is an archived mail posted to the Subversion Dev mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.