RE: Suggestion

Julian,

Thanks for considering my suggestion, and your detailed reply.

Yes, 'sparse branching' does nicely describe the effect I am trying to achieve. I do appreciate that any form of 'variants' is going to cause a management overhead.

The specific problems I'm trying to overcome are:

1. Parameter controlled functionality:
     Fairly easy to achieve,
     Requires only one build (of any 'tag'),
     BUT
     May result in unnecessarily large .EXE or DLL for every customer,
     Checking (at run-time) for a large number of parameters is messy and time-consuming, as is the admin of them,
     All code is present in every customer's copy, so un-licenced features could be enabled if parameter name is learned.

2. Conditional Compilation:
     Advantage of only one check-out (per 'tag'),
     Un-compiled code still visible, but greyed-out,
     BUT
     Suffers from developer resistance ('old school' way of doing things),
     Easy to make mistakes when setting values in MS Visual Studio (other IDEs may handle this better).
     
It is these last two points which are the biggest barrier, and their effect should not be under-estimated.
I have heard developers say that they would much rather suffer the overhead of updating several separate branches (with bug-fixes & enhancements), (along with all the risks that that approach exposes), rather than risk using Conditional Compilation.
 
Certainly, the two existing methods of variant management lend themselves fairly well to the situation of fixes needing applying to common-code (thus all variants), which I accept that a 'sparse' branch' may impact.
 
So, by moving the decision on which variant to compile back to the source-code check-out stage, most of the above failings could possibly be avoided.
I do acknowledge that this still is not a 'silver bullet' to cure the problem of variants - I doubt that there is one - but it might help.

Maybe the question should be put to the SVN community, to see what other users feel about this?

I haven't had a chance to try implementing anything. If I get time, I might try the post-check-out script idea...

Thanks again, Dave Radcliffe

-----Original Message-----
From: Julian Foad [mailto:julianfoad_at_btopenworld.com]
Sent: 09 July 2014 12:06
To: David Radcliffe
Cc: dev_at_subversion.apache.org
Subject: Re: Suggestion

Hi David.

Thank you for writing in with your suggestion. It took me a while to really grasp the essence of it, which I think is

    Sparse branching at the granularity of text hunks

where by "branching" I just mean tracking parallel streams of development in some way.

Thinking about configuration management paradigms rather than the mechanism, the branching paradigm where you have observed the need for a different way of working, and where this probably makes the most sense, is what you describe as "different instances of an application (say for different customers)", which sounds like what I know as "variants". Variant branching is a form of divergent branching, as is release branching, but with variant branching you expect to maintain each of these streams in parallel with ongoing development, while keeping the differences between them. The ongoing development may involve making a development branch sometimes from the whole set of variants, or sometimes from just one variant. In other words, variant branching can be orthogonal to development- and release branching: it's another dimension on the branching graph.

Conditional compilation is one way to implement variants; full branches (as in Subversion) are another way; and you are proposing something that combines aspects of both as being a better way than either.

No matter what method is used to manage the variants, the overall complexity of configuration management is basically the same. The complexity, that is, of managing which bug fixes need to be applied to which configurations, how to share code between two similar configurations, tracking which configurations are in what state of development and testing, and so on. If you have many variants, some of which are perhaps related to each other in sub-groups, and if you have many feature branches and release branches, some of which perhaps branch the whole set of variants and some of which apply only to some specific variants, then the management is complex. That is its own level of complexity.

But let's look at what your suggestion *can* make simpler.

From a read-only point of view, what you propose is something like conditional compilation (e.g. '#if' in C code), except you want the non-compiled text to be entirely omitted from the working file. In that sense it is like branching. It is a form of "sparse" branching as only certain files, and certain hunks of text within each file, are specific to the branch, the rest being common to all branches (in this dimension of branching).

It seems to be this property -- sparse branching -- that could make the management simpler.

When you commit a change to the common code on one branch, that change immediately affects all the other branches as well. Unlike with full branching, the change is not gated through any separate process of merge/test/commit. That immediacy can be useful, but, when changing common code, often a developer is expected to test that all variants will still build and perhaps pass some tests. It would be useful to be able to generate all the variants without having to check in a change and check it out again. Presumably some sort of "svn variant-switch" function would be available to switch to another variant while keeping the local modifications in place -- at least if the local mods are in common code. If there are local mods in branch-specific code, this is harder, requiring another layer of temporary storage.

In comparison with
regular conditional compilation ('#if'), the difference here is that you do not get the code that is specific to other variants. One consequence it that you can't make a set of builds, one per variant, all from the current source code; in some settings that is a disadvantage. Also the programmer can't so easily see how the change they are making to common code may affect the other variants. I suppose the section markers would still be present, so the programmer can see whether she is making changes inside or outside the variant-specific sections. That's an improvement over regular branching.

The developer also has the option of getting a "complete" checkout, but then that is not buildable so it's of limited use. But why not make this "complete" checkout be exactly defined by "#if" (or similar) parseable mark-up so that it can used to build one or many variants? Start thinking that way, and then the "single variant" checkout becomes more like a post-processing on the conditional compilation.

You sound unhappy with the mechanism and/or interface that Visual Studio provides for managing conditional compilation with '#if', describing it as "flaky". I don't really know what you mean by that. Is there something inherently wrong with that model of conditional compilation? What is the essence of the problem and what is needed for a better solution? Or is it just a poor implementation? (One useful thing is if you can grey-out or hide the code for other variants because it's not interesting most of the time. Some source code editors or IDEs will do that automatically for non-active #if clauses.)

What about editing and committing changes and merging fixes into multiple variants? How exactly is this made easier?

Have you ever tried your idea in any form? You could perhaps prototype it using '#if'
 syntax and a wrapper script around 'svn', probably using a separate working copy to interface between the one the user sees and the one Subversion sees.

Anyway, I'd be happy to hear your further thoughts.

Regards,

- Julian

>________________________________
> From: David Radcliffe
>
>
>When creating different instances of an application (say for different customers), the three approaches taken might be:
>1. Only one instance, with feature control enabled via parameters
>2. Separate branches for each customer 3. Conditional
>Compilation of one source base
>
>All three approaches have been used by the author at various companies
>he has worked for, and experience has shown that all three have their own drawbacks:
>1. If the differences between instances are significant, or there
>are a large number of instances, this approach rapidly becomes unwieldy
>2. This creates a maintenance nightmare when applying patches or updates 3. This works, but implementation (e.g. Visual Studio) leaves a lot to be desired.
>
>This suggestion proposes a fourth method: Conditional Check-out.
>This works (as far as the complier is concerned) in much the same way
>as Conditional Compilation, but moves the decision on whether to
>compile specific code back to the point where the codebase is obtained from SVN, rather than relying on the flaky mechanisms provided within the IDE.
>This means that the developer has to very consciously decide which instance she is building before even seeing the code.
>It will also support automated build processes (e.g. Cruise Control) which checks-out code for continuous integration builds.
>
>It works like this.
>Any (existing) code (which is un-adorned by the new SVN decoration) is treated as ‘common’ to all instances.
>This ensures 100% backwards compatibility, with no changes being required to existing source code, or clients.
>
>Sections of code which apply to, or are omitted from, one or more specific instances are decorated by a new SVN decoration (both before & after the relevant section).
>This section is then either included or excluded, as appropriate, when the code base is obtained from SVN repository.
>
>The ‘checkout’ and ‘update’ commands in clients will have the ability
>to be ‘told’ (maybe from a pull-down list of recent values) which specific instance should be obtained, and SVN includes or omits sections of code as required. A special value, over and above the user-named instances is required.
>This is ‘ALL CODE’. This obtains all the codebase, including all instance-specific code, with the decorations for specific sections included.
>This is primarily for documentation and review purposes, but also provides a ‘base’ from which further instance-specific changes can be made.
>
>The decoration referred to is similar in fashion to the keywords already existing in SVN, and can exist as comments in the source code.
>There would be two variants of the decoration to either:
>1. Include this code section only when a specific instance is required. It will not be included for all other instances (except ALL CODE).
>2. To exclude this code section when a specific instance is required. It will be included for all other instances (including ALL CODE).
>Sections of code may have multiple [start of section] decorations; the
>effect is the logical-ORing of those decorations, so that multiple instances can benefit from the same section control.
>
>David W. Radcliffe
>

-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2014.0.4716 / Virus Database: 3986/7821 - Release Date: 07/09/14
Received on 2014-07-09 14:15:13 CEST