----- Original Message -----
> From: Daniel Shahaf <danielsh_at_elego.de>
> To: Ashod Nakashian <ashodnakashian_at_yahoo.com>
> Cc: Markus Schaber <m.schaber_at_3s-software.com>; "julianfoad_at_btopenworld.com" <julianfoad_at_btopenworld.com>; "mtherieau_at_gmail.com" <mtherieau_at_gmail.com>; Subversion Development <dev_at_subversion.apache.org>
> Sent: Wednesday, April 4, 2012 10:29 PM
> Subject: Re: Compressed Pristines (Summary)
>
> Ashod Nakashian wrote on Wed, Apr 04, 2012 at 10:18:47 -0700:
>> >________________________________
>> > From: Daniel Shahaf <danielsh_at_elego.de>
>> >To: Ashod Nakashian <ashodnakashian_at_yahoo.com>
>> >Cc: Markus Schaber <m.schaber_at_3s-software.com>;
> "julianfoad_at_btopenworld.com" <julianfoad_at_btopenworld.com>;
> "mtherieau_at_gmail.com" <mtherieau_at_gmail.com>; Subversion
> Development <dev_at_subversion.apache.org>
>> >Sent: Wednesday, April 4, 2012 9:46 AM
>> >Subject: Re: Compressed Pristines (Summary)
>> >
>> >> >1. Filesystem compression.
>> >> >
>> >> >Would you like to assess the feasibility of compressing the
> pristine
>> >> >store by re-mounting the "pristines" subdirectory as
> a compressed
>> >> >subtree in the operating system's file system?
>> >>
>> >> No :-)
>> >>
>> >> There are two ways to answer this interesting proposition of
>> >> compressed file-systems. The obvious one is that it isn't
> something
>> >> SVN can or should control. The file-system and certainly system
>> >> drivers are up to the user and any requirement or suggestion of
>> >> tempering with them is decidedly unwarranted and unexpected from
>> >> a VCS.
>> >
>> >The suggestion wasn't that you teach svn how to change the OS's
> fs
>> >settinsg, it was that you investigate how solutions at the OS level
>> >compare to the other approaches already suggested (custom format -based
>> >and sqlite-based pristine store).
>>
>> That's an easy question. The answer is that at *best* they'll do as
>> good as in-place compression. However, in practice they'll do much
>> worse. The reason is that the OS level compression works on not only
>
> Is it _necessarily_ that way? What does Subversion about the pristine
> store's contents know that the filesystem doesn't?
No, not unless an OS cares about high performance that is independent of file-size. It's reasonable or an OS to assume random file access for both reading and writing. Recompressing the complete file on each write is not feasible, so whole files aren't compressed by OS compression. Furthermore, they use "real time" compression algorithms that are not very efficient, but are very fast and have low memory footprint.
Subvesion *knows* that pristine files are *never* modified, they are always replaced. (Note for the technically-correct: SVN might get a diff on the wire to update a prisine file, but as far as we care, the file must be completely read/parsed/patched/updated, so in the end, the whole file is streamed back to disk, not an arbitrary chunk, although for binary files this might not be the case. Still, even in that case we can use chunking to limit the largest number of bytes that must be recompressed, so large files can still be partially updated without major overheads.)
>
>> the single file level, but actually at the block level. This is to
>> make modifications reasonably fast (read compressed data, uncompress,
>> modify, write recompressed data). If the complete file is compressed
>> then even changing a single byte (neglecting that no storage works on
>> the byte-level anyway) will yield performance that will at least
>> linearly degrade by the filesize.
>>
>>
>> >
>> >In short: if 'mount -o compress=yes' provides 90% space savings
> then we
>> >would have little reason to implement space-saving solutions in svn
> itself.
>> >But it's the user's, not svn's, responsibility to run that.
>>
>>
>> I should kindly disagree here. The user can do that any time, anyway.
>> SVN is trying to improve its pristine store design,
>
> No we don't. We're trying to provide users with a low-footprint
> pristine store. If we can achieve that by tuning OS settings we
> might not need to write code.
As pointed out already, that's not portable, reasonable to expect of a VCS, achievable for all uses (due to permissions for example) or enforceable (for a unified experience).
>
>> not advise users
>> on how best to organize their storage (in general or when using SVN).
>> In addition, I find discriminating against users who don't have that
>> luxury or knowledge to be a bit unfair. At least if are claiming to
>> support the wide range of systems that we do.
>>
>
> Yes, I edited that out of my previous email: we should ensure that
> "low-footprint pristine store" is achievable on most/all platforms we
> support.
>
> Also, agreed that we should consider the case of users don't have
> permissions to tune their OS settings but have working copies.
I've just repeated this much a few lines above. I think we either agree that SVN should improve its footprint or we don't. If we do, then we need a built-in solution. If we don't, then whether the user is smart enough to use some trick or not becomes irrelevant; we can't force them either way.
>
>> -Ash
>>
>>
>> >
>
Received on 2012-04-04 20:58:18 CEST