Re: FSFS successor ID design draft

On Fri, Aug 26, 2011 at 1:14 PM, Stefan Sperling <stsp_at_elego.de> wrote:
> Below is an initial draft of a design for successor-IDs in FSFS.
> It is a bit long, but I hope it covers all relevant points in
> sufficient detail.

Nice! Interesting stuff :-).

I'm not an expert in these matters, but I've read it with interest,
for two reasons: (1) successor-IDs seem to be really useful (cf.
cmpilato's response) and (2) this design seems to be generic enough to
be able to create other caches (or indexes if you will) for FSFS in
the future, with the same basic techniques. Caches for other
information that's expensive to calculate, like <wild idea> line-based
diff-info for speeding up diff and blame -- something that was
discussed a bit during the Berlin Hackathon.</wild idea>

Anyway, I'm getting a bit ahead of myself, and don't want to hijack
this thread. But it seems like a nice standard procedure for building
such a cache, in a robust way, and where callers are able to use
whatever they can from the cache, and still do the expensive
calculation (with some upper-bound perhaps) on the part that wasn't
yet cached (and maybe insert that new information in the cache (?)).
Cool.

I had one remark:

[ ... ]
> - Procedure for Writers -
>
> Note that the following describes how successor data for a single
> node_rev_id is updated. This is only a sub-step of the global cache
> synchronisation procedure (which is described in the next section).
>
> A writer operates under the assumption that the entire cache directory
> has been locked for writing.
>
> The writer locates the cache file corresponding to the given node_rev_id.
>
> The writer opens this file and reads the index to learn the offset of the
> last s-block of the node_rev_id.

Somewhere here should be described what the writer does when this file
doesn't exist yet (create it, put an empty index in there, ... I don't
know).

-- 
Johan