Re: in-memory-cache TODO

On Thu, Apr 3, 2008 at 8:19 PM, Blair Zajac <blair_at_orcaware.com> wrote:
>
> David Glasser wrote:
>
> > On Thu, Apr 3, 2008 at 5:22 PM, David Glasser <glasser_at_davidglasser.net>
> wrote:
> >
> > > The in-memory-cache branch is nearing completion. It unifies various
> > > caches found in FSFS behind a simple API, and allows the caches to be
> > > implemented either in process as hashes or with memcached. If you use
> > > memcached (which is enabled in a new fsfs.conf file), caches can live
> > > longer than the lifetime of a single svn_fs_t.
> > >
> > > Remaining TODO:
> > > - Move the code which makes an apr_memcache_t based on an
> > > svn_config_t to libsvn_subr from FSFS, and use it in the C test.
> > > - Build system issues (mostly done by a cache I have from dchristian).
> > > - Add --memcache options to the test suite.
> > > - Actually run a real benchmark.
> > > - Document the fact that all errors communicating with memcached are
> > > fail-stop. This is a problem that should be solved with a better svn
> > > logging infrastructure, which we ought to get before 1.6 anyway (it's
> > > a high priority for my team).
> > >
> >
> > Also make sure that we never pass keys or values that are too big to
> memcache.
> >
>
> The key can be at most 250 bytes long and in the default memcached
> configuration (without recompiling) values 1 Mbyte large.
>
> For keys, if they are longer than 250 bytes, I replace the middle N bytes
> with a md5 hex hash of the removed bytes so the key still fits. I know in
> my app, we will have paths longer then 250 bytes long, so having this would
> be nice.

The hash trick is a good idea. I will do it.

> I would be surprised if there's any data that would go over a megabyte. I
> only ran into this limitation when I serialized the results of
> svn_fs_dir_entries() for a directory with 17,000 entries where each entry
> had 12 svn properties on each it. Using gzip compression got 90%
> compression rates on this data.

Well, I'm considering caching actual representations (either as
fulltexts or deltas too) so perhaps this would be useful. (And this
would be a good use of the memcached flags: noting if the data was
compressed, another argument for not exposing the flags to the user.)

--dave

-- 
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe_at_subversion.tigris.org
For additional commands, e-mail: dev-help_at_subversion.tigris.org