Re: [RFC] Shelving and Checkpointing

Shelving and Checkpointing Dev.

Local History Operations for Subversion

The Problem
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.gup2kj3jxnow>

Shelving
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.ypbr9w7qsg2m>

Checkpointing
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.5yiep788t1oa>

The Solution
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.tjgsg23l0n40>

Shelving
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.mw1pz02zc8uf>

Checkpointing
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.ne5xm98sqdbw>

Feature/Benefit Matrix; Comparison with Existing Options
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.7bse6jch3wkr>

Role Models
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.eljjsy831sf8>

The Design
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.xgquz085lgrx>

Shelving
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.w7wf2zrpkfkg>

Extensions / Not Supported Initially
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.ysnjkpcjx9ee>

Integrate Shelving with Changelists
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.ay5h5pfumpv8>

Checkpointing
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.e8ir63s9uk9f>

Checkpointing Option 1
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.hprm8omm4h2f>

Checkpointing Option 2
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.rtyl8w8kbci4>

Checkpointing Option 3
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.qn4w4hd6lbaj>

Extensions / Not Supported Initially
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.xuchjgc6xnd0>

Rebasing Checkpoints
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.w6sqpojow3ug>

Integrating Checkpoints with Shelving
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.ilq1gee589ue>

GUI Considerations
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.lnb4e51zn6pb>

Complete WC State Shelving
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.bxf1a4y5kzt>

Existing Issues to Overcome
<https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#heading=h.4sj99cgksge9>

References:

   1.

   Shelving-Checkpointing Dev
   <https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#>
   doc. (J Foad)
   2.

   Shelving-Checkpointing UI
   <https://docs.google.com/document/d/1Z0HZfpWRnU0ke2G7H20V0-my_egV_BY4D_aGlfvKuTk/edit#>
   doc. (J Foad)
   3.

   issue SVN-3625 <https://issues.apache.org/jira/browse/SVN-3625> "Commit
   shelving"
   4.

   issue SVN-3626 <https://issues.apache.org/jira/browse/SVN-3626> "Commit
   checkpointing"
   5.

   Svn Wiki page "Design: SavePoints
   <https://wiki.apache.org/subversion/SavePoints>"

The Problem

This is the problem we are looking to solve. We will use this to determine
the scope of design and implementation.
Shelving

While developing one change, you need to stop and work on an urgent fix or
you want to commit a quick and easy change or try a different approach in
the same code. You need to store the unfinished work and return to it
later. Existing options: server-based branches, patch files, extra Working
Copies.

Priorities: fast, offline.
Checkpointing

While developing a change, you reach a state which you want to save. The
next thing you do might break it, or you want to treat this work as good
and probably ready to publish as a first step and to be looking at your
next work as diff against this first step.

In other words, you want successive local commits, and then be able to:

   -

   discard the entire work, or
   -

   roll back to an intermediate state, or
   -

   commit as a whole, or
   -

   commit all the separate steps.

Priorities: fast, offline.
The SolutionShelving

One-click patch management on top of 'svn diff' and 'svn patch'.

   -

   similar to "git stash"
   -

   backward-compatible extension to WC format
   -

      (old clients still work, just not seeing the shelved changes)

Functions:

   -

   shelve/stash: save a WC diff as a patch and revert it from WC
   -

   unshelve/stash-pop: apply specified patch to WC and delete the patch
   -

   list shelved patches and the files that they affect
   -

   delete shelved patches
   -

   shelving supports exactly the same kinds of change as 'svn patch' and
   'svn diff'

Compared to manual patch management:

   -

   one-click simplicity
   -

      don't need to choose a filename or remember where it is
      -

   still can't save & restore WC base state (in v1)

Checkpointing

Options:

   1.

   further patch management built on a series of shelved changes
   2.

   local commits tightly integrated
   3.

   checkpoints are commits in a local repository

Option 1 looks and feels like an extension -- clunky, new commands,
existing features (such as diff) have minimal support for acting on
checkpoints.

   -

   key benefit: roll back to a checkpoint
   -

   key drawback: limited support for viewing current work against last
   checkpoint
   -

   backward-compatible extension to WC format
   -

      (old clients still work, just not seeing the checkpoints)

Option 2 looks and feels like a Subversion version of DVCS local commits. A
key differentiator is the WC base state is the most recent checkpoint
instead of the original base revision. That affects, for example, which
lines a Subversion-aware editor would highlight as changed.

   -

   key benefit: view the current work against the last checkpoint
   -

   key drawback: not very feasible to implement in current state of
   Subversion
   -

   incompatible change of WC format

Option 3 looks and feels like using native Subversion on a local branch,
with the restriction that the connection between this branch and the
original repository is weak. (For example, "svn log" and "blame" might see
only the checkpoints, not the prior history in the original repository,
unless/until we extend their functionality.)

   -

   key benefit: works exactly like normal svn commits in many ways
   -

   key drawback: a simple implementation will increase WC size by 50% and
   on making the first checkpoint in a series will take time to perform a full
   local commit of the WC base state
   -

   backward-compatible extension to WC format
   -

      (when checkpoints are in use, an old client would see only the
      checkpoints repository and know nothing about the original repository)

Feature/Benefit Matrix; Comparison with Existing Options

Shelving

(& Checkpoints)

server branch

patch files

extra WCs

Shelving (patches)

Chkpt 1

Chkpt 2

Chkpt 3

works offline

no

yes

yes

yes

yes

yes

yes

one-click simplicity

no

no

no

yes

yes

yes

yes

fast

no

yes

medium

yes

yes

yes

yes

low disk space requirement

yes

yes

no

yes

yes

yes

yes

incremental re-build

yes

yes

no

yes

yes

yes

yes

save & restore WC base state

no

no

yes

no

no

yes

yes

save & restore uncommittable state (conflicts, etc.)

no

no

yes

no

n/a

n/a

n/a

can discard the entire work

no

yes

yes

yes

yes

yes

yes

back-compatible WC format

n/a

n/a

n/a

yes

yes

no

yes

Checkpoints

(additional)

server branch

patch files

extra WCs

Checkpt 1 (patches)

Checkpt 2 (special)

Checkpt 3 (local repo)

roll back to any checkpoint

yes

yes

yes

yes

yes

yes

commit as a whole

yes

yes

yes

yes

yes

yes

commit all the separate steps

yes

awkward

awkward

yes

yes

yes

incremental view in WC

yes

no

no

no

yes

yes

automatic checkpoints to enable rolling back a conflicting update

no

no

no

no

can do

maybe

Note: "Automatic checkpoint" is listed as a separate feature, under the
name "Destructive command rollback", in "Design: SavePoints" [5].
Role Models

With Shelving, we are catching up with a feature that most current VCSs
already provide. We should make it easy for users to apply what they have
learnt in those systems, and avoid needless differences. We should also
support Subversion's unique characteristics such as directories being
versioned objects.

SHELVING

git stashing <https://git-scm.com/book/en/v1/Git-Tools-Stashing>

hg shelving <https://www.selenic.com/mercurial/hg.1.html#shelve>

bzr shelving
<http://doc.bazaar.canonical.com/bzr.dev/en/user-guide/shelving_changes.html>

p4 shelving
<https://www.perforce.com/perforce/r17.1/manuals/p4guide/chapter.basic-tasks.html#basic-tasks.recurring-other.changelist.shelve>

IntelliJ
<https://www.jetbrains.com/help/idea/shelving-and-unshelving-changes.html>

NetBeans
<http://docs.oracle.com/netbeans/nb82/netbeans/netbeans/NBDAG/version_control.htm#NBDAG2545>

general

well integrated

simple patch

stored where?

local repo

WC/.hg/

WC/.bzr/

the depot (shareable)

local (project dir by default)

~/.netbeans/

stored how?

special commit

git patch

special commit?

special commit (changelist)

svn+git patch

patch + MIME binary support

changelist integration

n/a

tight: a shelf is a p4 change

cl <==> shelf strong association

no

pop vs. apply

pop or keep

backup or keep

pop or keep

pop

backup or keep

pop or keep

access in other cmds

yes

diff,diff2,files,print

(no)

(no)

merge & conflicts

regular merge

temp. commit + merge + continue

regular merge

The Design

For UI design, see Shelving-Checkpointing UI
<https://docs.google.com/document/d/1Z0HZfpWRnU0ke2G7H20V0-my_egV_BY4D_aGlfvKuTk/edit#>
doc. [2].
Shelving

Implementation in libsvn_client with 'svn' command-line UI.

Main functionality:

   -

   shelve/stash [name] [paths/changelist]: save a WC diff
   -

      if no name given, automatically generate a name
      -

      reverts the successfully shelved changes from the WC
      -

   unshelve/stash-pop: apply specified patch to WC
   -

      optionally removes the patch
      -

      can apply to any revision or branch
      -

   list/delete shelved patches
   -

   stores patches in '.svn' dir, e.g. in '.svn/patches/'
   -

   standard svn patch file format; paths relative to WC root

Extensions / Not Supported Initially

Kinds of change that can be shelved:

WC State or Change

starting point

fair usability

good usability

committable changes

file text, file delete/add, most properties

yes

yes

yes

mergeinfo changes

no

yes

yes

copies and moves

no

yes

yes

directories (mkdir/rmdir/...)

no

yes

yes

binary files & properties

no

yes

yes

uncommittable state

unresolved conflicts

no

no

yes

changelist assignment

no

no

yes

unversioned items (git stash -u/-a)

no

no

no

inconsistent WC states (missing/obstructed)

no

no

no

WC base state (rev, URL, switched, depth)

no

no

no

Other shelving extensions to consider:

   -

   remember the order; 'unshelve' takes newest patch by default
   -

   allow showing a shelf's content -- like 'svn log' options
   -

   keep a backup when unshelving
   -

   allow a description (log message) in shelves and changelists
   -

      show it when listing shelves/changelists
      -

      use it when committing from a changelist that has a log msg
      -

   handle conflicts better, using a proper merge rather than patching

Integrate Shelving with Changelists

I hate to add complexity without simplifying something at the same time.
Considering the relationship between shelving and changelists:

   -

   a changelist is a named set of file-paths in the WC; each path may be
   locally modified
   -

   a shelved change is a named set of file-patches that is not currently
   applied to the WC

Integrate them like this:

   -

   shelved changes and changelists share the same namespace
   -

   shelving means making a separate patch for each changelist (and one for
   the not-in-a-changelist paths)
   -

   unshelving means converting shelved changes to changelists

Benefits:

   -

   clear and simple relationship with no overlap
   -

   changelists remain backwards-compatible

Main issues:

   -

   a changelist can include unmodified paths:
   -

      should a shelved patch also include unmodified paths?
      -

   a path can appear in multiple shelves, but only in one changelist:
   -

      what to do when unshelving if a path clashes?

   -

   any extensions should apply uniformly to both shelving and changelists

Role models:

   -

   changelists in IntelliJ IDEA
   <https://www.jetbrains.com/help/idea/changelist.html> (are unified
   across svn, git, hg, etc.)
   -

   changelists in Perforce
   <https://www.perforce.com/perforce/r17.1/manuals/p4guide/index.html#overview.files.changelists>
   (are new or pending or shelved or submitted)

CheckpointingCheckpointing Option 1

Further patch management built on a series of shelved changes. The working
copy largely does not see the checkpoints, it only see the current working
state against the original base; new commands provide limited operations on
checkpoints.

Take design ideas from Mercurial Queues?

Functionality:

   -

   all existing svn WC commands operate against original base revision
   -

      status, diff, revert, commit, etc.
      -

   some new commands operate against last (or any) checkpoint
   -

      status, diff, etc.
      -

      limited subset and not exactly equivalent functionality
      -

   new commands for new operations such as commit-all-separately and
   rollback
   -

   'svn update' disallowed, as it implies rebasing the checkpoints

Checkpointing Option 2

Local commits tightly integrated into svn WC functionality & command syntax.

Functionality:

   -

   all svn WC commands operate against latest checkpoint by default
   -

      status, diff, revert, etc.
      -

   all svn WC commands can operate against any intermediate checkpoint
   -

      using extended revision-selector syntax
      -

   'svn update' disallowed, as it implies rebasing the checkpoints

Main design tasks:

   -

   pseudo-revision syntax to address local commits, in all WC and higher
   APIs
   -

   WC storage for multiple versions
   -

   WC able to switch the 'current base version' to an arbitrary stored
   version
   -

   internal ability to:
   -

      diff & merge arbitrary versions (local as well as repository versions)
      -

      send merge output to any WC layer (base or working or checkpoint)
      -

   UI and high level implementation of the new checkpointing features

No feasible implementation plan for this option.
Checkpointing Option 3

Checkpoints are commits in a local repository embedded in the WC.

Functionality:

   -

   all svn WC commands operate against latest checkpoint by default
   -

      status, diff, revert, etc.
      -

   all svn WC commands can operate against any intermediate checkpoint
   -

      using standard svn repository-access syntax
      -

   all svn repository operations by default only see the checkpoints
   repository
   -

      e.g. 'svn update' moves only from one checkpoint to another

Design points:

   -

   each series of checkpoints is stored in a new local repository
   -

   committing a first checkpoint creates a new local repository, stores the
   WC base as r1 and the WC working layer as r2 in it, and points the WC at r2
   in this repository, updated to r2.
   -

   convenient access to the original repository is limited to specific new
   functionality -- e.g. 'log' and 'blame' might visit the checkpoints first
   and then the original repository

Extensions / Not Supported Initially

   -

   rebasing checkpoints

Rebasing Checkpoints

It is desirable to be able to rebase a series of checkpoints, when pulling
in new changes from the repository. This involves several steps of merging,
once into each checkpoint, and the possibility of conflicts at each step.

Rebasing checkpoints is more important in trunk-based, less important in
branch-based work flows.
Integrating Checkpoints with Shelving

How are shelving and checkpoints related conceptually?

In other words, what do we expect to happen if we start working, make some
checkpoints, and then shelve "the work" to work on something else? Do we
usually want to shelve the whole caboodle, all the work checkpointed... or
just the latest (uncheckpointed) working state and start new work against
the latest checkpoint?

   1.

   one series of checkpoints, plus a set of stashed working states
   1.

      each stashed working state is relative to a particular checkpoint
      2.

      each stashed working state is relative to the original base
      2.

   multiple shelves, each holding a series of checkpoints plus a working
   state
   1.

      all shelves are based on the same "original" base state

(A checkpoint is a committable change; a working state is a committable
change and/or uncommittable WC state such as conflicts, unversioned files,
etc.)

And is there also a degenerate option where we simply don't track what each
shelved change was relative to, it's just a context-diff patch?

Needs design.
GUI Considerations

GUI priorities tend to be a little different from a CLI.

   -

   APIs that do the job simply and reliably
   -

   progress indication and cancel, for any long-running ops
   -

      hopefully we won't have any long-running ops
      -

   ability to undo
   -

      we don't have to implement undo, just provide enough hooks so caller
      can

Complete WC State Shelving

A key concern will be to have APIs that do the job simply and reliably.
Such as: ability to shelve a WC state whatever state it's in (all the odd
states like unresolved conflicts, etc.) and restore to exactly the same
state.

We should have two complementary APIs:

   -

   a "shelve" API should visit the WC reporting absolutely everything about
   the state, in such a way that the caller can capture that state;
   -

      a "diff" API should report the subset of this that is committable;
      -

   an "unshelve" API should be able to recreate any such state, using input
   entirely provided by the caller
   -

      a "patch" API should accept the subset of this that is committable.

To what extent do such APIs exist?

We should write automatic round-trip testing for diff-&-patch and for
status-&-modify.

For states that we can't shelve, a GUI wants to know in advance that this
is the case (to grey-out the button), rather than the operation to error
out part way through. So we want:

   -

   an API to efficiently tell whether the WC state can be shelved (or why
   not)

Existing Issues to Overcome

Bugs and deficiencies:

   -

   svn patch to handle mergeinfo: SVN-3747
   <https://issues.apache.org/jira/browse/SVN-3747>
   -

   svn diff/patch to handle binary files (is partly done -- in "svn diff
   --git" anyway)
   -

   svn diff/patch to handle copies and moves: SVN-1056
   <https://issues.apache.org/jira/browse/SVN-1056>
   -

   svn diff/patch to handle directories (mkdir, rm, cp, mv)

------------------------------
Julian Foad, Assembla, 2017
Received on 2017-07-10 15:27:12 CEST