Re: Compressed Pristines (Summary)

On Wed, Apr 4, 2012 at 1:18 PM, Ashod Nakashian
<ashodnakashian_at_yahoo.com> wrote:

> That's an easy question. The answer is that at *best* they'll do as good as in-place compression. However, in practice
> they'll do much worse. The reason is that the OS level compression works on not only the single file level, but actually
> at the block level. This is to make modifications reasonably fast (read compressed data, uncompress, modify, write
> recompressed data). If the complete file is compressed then even changing a single byte (neglecting that no storage
> works on the byte-level anyway) will yield performance that will at least linearly degrade by the filesize.

FWIW, that is exactly my concern with your custom file format. I do
not see how you can achieve the benefits you expect without needing to
repack files and I do not see how that can perform reasonably.

That said, you also seem aware that the solution has to perform well
so at worst it is just a question as to whether you want to spend the
cycles to prove it can work and achieve all the goals. I am skeptical
but look forward to being wrong.

The lazy part of me thinks storing files up to 32KB in SQLite and
storing the rest as just single compressed files would give 99% of our
users what they want and would be less likely to have issues.

-- 
Thanks
Mark Phippard
http://markphip.blogspot.com/