Re: Shelve & checkpoint - next steps

Here is a more comprehensive outline of next development steps for
shelve & checkpoint.

I propose now switching completely from 'patch' storage to storing whole
files, and storing both the base and the working version of each shelved
file. This will switch the speed-and-space characteristics, particularly
noticeable when dealing with large files, from small and slow to fast
and big.

To start with, we may well store a new copy of both base and working of
every shelved file, in each version of the shelf. We can evaluate again
after that and see what better compromise to make, adding more
complexity to make a better trade-off, such as de-duplicating when the
same base and/or working version appears in multiple shelf-versions. In
the meantime, if need be, the presentation layer could do something to
help the user avoid storing too many versions.

Functional improvements, in approximate priority order:

   * overcome remaining 'patch' format limitations [1]

     - 'mkdir', 'rmdir'
     - copy and move
     - properties with 'binary' content values
     - files not recognized as 'binary' but that don't diff/patch nicely

   * proper merge

     - 3-way merge
     - conflict handling like in 'svn update' and 'svn merge'

   * make existing read-only 'svn' commands work on a shelf:

     - svn status
     - svn export/cat/propget/proplist (for shelf base & working tree)
     - svn diff

Further functional improvements:

   * integrate 'shelf' and 'changelist' concepts [2]:

     - shelving converts a changelist into a 'shelf',
     - unshelving converts a shelf into a changelist;

   * let 'unshelve' update to the correct base or warn if base mismatch

To overcome remaining 'patch' format limitations:

   * develop the whole-file storage format to support (for binary
     files, at first):

     - base as well as working version,
     - Modify / Add / Delete / Replace status,
     - properties,

   * switch to whole-file storage (base and working versions) for all
     files, not just 'binary' files:

     - when unshelving a text-mod in a non-binary file, and for the
       properties of a 'binary' file, run 'diff' and pipe the result
       into 'patch', so the end result is the same as if we had
       continued storing a patch file, for now

   * remove the now-unused support for storing a patch file

   * develop the storage format to support:
     - 'mkdir' and 'rmdir' (with directory properties)

   * develop the storage format to support:
     - copy and move info

   * send property changes through the existing property merge APIs
     so that binary-valued properties (and probably other oddities)
     will be supported

To implement proper merging:

   * reconstruct the delta on demand as svn_diff_tree_processor_t
     (or store base + delta, reconstruct working version on demand);

   * make a merge-into-WC routine that can merge a given diff (via
     svn_diff_tree_processor_t) into the WC, using 3-way text merge
     for text files and simple theirs-or-mine for binary files, and
     can raise conflicts just like 'update' and 'merge' do;

   * let 'unshelve' feed a text file's diff into that 'merge-into-WC';

   * update the conflict handling to be able to refer to a shelf as
     the merge source;

   * might work better or be easier if we store a reference to the
     WC base layout (revision numbers, etc.).

To implement existing read-only 'svn' commands:

   * implement a way to specify a shelf instead of a regular WC
     operation in the CLI and libsvn_client API; for example:

     - a special revision specifier like '-r SHELFNAME'
       with a new value for svn_opt_revision_t.kind
     - a special changelist specifier like '--cl SHELFNAME'
     - a special extra target argument like '^s/SHELFNAME'
     (special rev-spec is prototyped on 'shelve-checkpoint' branch)

   * for 'svn status':

     - implement an API analogous to svn_wc_walk_status() that
       returns svn_wc_status3_t objects;
     - call it from the existing 'svn status' code;

   * for 'svn export/cat/propget/proplist':

     - implement some kind of API for reading tree data from shelf;
     - augment libsvn_client export/cat/prop routines to use this
       new tree API when a shelf is specified;

   * for 'svn diff':

     - implement an svn_diff_tree_processor_t driver API on the shelf;
     - decouple the existing diff output routines from their driver
       (see the diff_driver_info_t parameter to get_diff_processor());

Take advantage of the new interfaces and routines developed here, to
simplify the rest of svn. This could include:

   * improve the 'status' API to be a better fit for both shelves and
     the main WC storage;

   * implement the new API for reading WC base and working trees, on
     top of the existing WC access APIs, and re-implement existing
     libsvn_client export/cat/prop routines on top of that;

   * rewrite existing 'update' and 'merge' to use the new
     merge-into-WC routine;

- Julian

[1]
https://cwiki.apache.org/confluence/display/SVN/Shelving+and+Checkpointing+Dev#ShelvingandCheckpointingDev-Shelving.2

[2]
https://cwiki.apache.org/confluence/display/SVN/Shelving+and+Checkpointing+Dev#ShelvingandCheckpointingDev-IntegrateShelvingwithChangelists
Received on 2018-05-14 17:27:16 CEST