Re: SVN Externals 1.6 to 1.7 migration issue

On Thu, May 2, 2013 at 12:50 PM, Johan Corveleyn <jcorvel_at_gmail.com> wrote:
> On Thu, May 2, 2013 at 11:00 AM, Johan Corveleyn <jcorvel_at_gmail.com> wrote:
>> On Thu, May 2, 2013 at 9:55 AM, Johan Corveleyn <jcorvel_at_gmail.com> wrote:
>>> On Wed, May 1, 2013 at 5:01 PM, BRM <bm_witness_at_yahoo.com> wrote:
>>>> While I have not had the issue you are having, assuming the externals are in
>>>> the same repository I would highly recommend changing from using the syntax
>>>> you have to using the carrot (^) operator as it will save you many headaches
>>>> if your original svn location changes.
>>>>
>>>> I.e. use:
>>>>
>>>> ^/FOLDER_A FOLDER_C
>>>> ^/FOLDER_B FOLDER_D
>>>
>>> Indeed, that's far better than using an absolute url with scheme etc.
>>>
>>>> Also, I think the specific issue you are having (looking more closely at the
>>>> error message below) is that AFAIK, SVN does not let you import specific
>>>> files via externals - you have to do name spaces (aka folders) instead.
>>>
>>> That's not correct. File externals are supported (since 1.6 I think).
>>> They have a couple of specific problems though (a lot of which have
>>> been fixed in 1.7), because their implementation is entirely different
>>> from directory externals (file externals use the "switch" mechanism
>>> underneath, while dir externals are essentially an embedded checkout
>>> with some sugar on top).
>>>
>>> So, concerning to the problem of the OP:
>>>
>>>> From: "Hutchinson, Steve (UK)" <steven.hutchinson_at_mbda-systems.com>
>>>> To: "users_at_subversion.apache.org" <users_at_subversion.apache.org>
>>>> Sent: Wednesday, May 1, 2013 7:01 AM
>>>> Subject: SVN Externals 1.6 to 1.7 migration issue
>>>>
>>>> Hi,
>>>>
>>>> We're been using 1.6 svn externals to manage a FW task.
>>>>
>>>> In the repository we have a folder structure (which is pretty much defined
>>>> by the tools we are using) as below :-
>>>>
>>>> FOLDER A -> file_a1.txt, file_a2.txt, file_a3.txt
>>>> FOLDER B -> file_b1.txt, file_b2.txt, file_b3.txt
>>>>
>>>> Then using externals we link to those folder to create a WC PROJECT
>>>> structure that looks like :-
>>>>
>>>> PROJECT -> FOLDER C -> file_a1.txt, file_a2.txt, file_a3.txt AND
>>>> file_b1.txt
>>>> PROJECT -> FOLDER D -> file_b1.txt, file_b2.txt, file_b3.txt
>>>>
>>>> The externals on PROJECT FOLDER we used looked like (not actually using file
>>>> protocol, just created for example purposes) :-
>>>>
>>>> file:///D:/FPGA/SVN_ISSUE/repo/FOLDER_A FOLDER_C
>>>> file:///D:/FPGA/SVN_ISSUE/repo/FOLDER_B FOLDER_D
>>>> file:///D:/FPGA/SVN_ISSUE/repo/FOLDER_B/file_b1.txt FOLDER_C/file_b1.txt
>>>>
>>>> Worked ok with 1.6. We get the below error in 1.7 when updating of :-
>>>>
>>>> External failed: D:\FPGA\SVN_ISSUE\PROJECT\FOLDER_C\file_b1.txt
>>>> Error: Cannot insert a file external defined on
>>>> 'D:\FPGA\SVN_ISSUE\PROJECT' into the
>>>> Error: working copy 'D:\FPGA\SVN_ISSUE\PROJECT\FOLDER_C'.
>>>>
>>>> We have tried a few things (will not share all at this point due to info
>>>> overload), but wondered if there were any others that achieve something like
>>>> this with 1.7 or could offer some advise ?
>>>
>>> So essentially you are "injecting" a file external (file_b1.txt)
>>> inside a directory external (FOLDER_C (an external of FOLDER_A)).
>>>
>>> Hm, I remember some past discussions about edge cases like these
>>> (during the working copy rewrite of 1.7), but I don't remember exactly
>>> if this is actually a supported use case, or if it just happened to
>>> work by accident in 1.6.
>>>
>>> Perhaps someone else on this list can shed some light on this?
>>
>> I had the following conversation with Philip Martin, on IRC (#svn-dev):
>>
>> <@jcorvel> philipm, perhaps you know the answer to this externals
>> question (works in 1.6, but no longer in 1.7):
>> http://svn.haxx.se/users/archive-2013-05/0006.shtml
>> <philipm> A file external defined in one working copy but located
>> inside another working copy is not going to work in 1.7 or 1.8.
>> <philipm> The directory external is a separate working copy with its
>> own .svn/wc.db and that knows nothing about the external defined in a
>> different .svn/wc.db
>> <@jcorvel> Ok. Sooo, is this more in the "it worked by accident in
>> 1.6, but was never supposed to be supported" category? Or could this
>> be considered a regression?
>> <@jcorvel> I guess it doesn't matter ... semantics ...
>> <philipm> I don't suppose we ever supported it deliberately.
>> <@jcorvel> perhaps the more interesting question then is: is this at
>> all supportable within the externals design (of wc-ng)? Or would this
>> require "externals 2.0"?
>> <philipm> If we change directory externals to be more like switch then
>> the directory externals become part of a single working copy.
>> <@jcorvel> ack
>> <philipm> It may be relatively simple to do that. At present if you
>> try to "svn sw URL path" and path does not exist then the switch
>> fails.
>> <philipm> But it only fails because of a high level check on path, in
>> the past it was possible to switch a non-existant path.
>> <philipm> (In the past being during 1.7 development)
>> <philipm> If you switch a path that does not exist that looks very
>> much like an external (it is how file externals are implemented)
>> <philipm> One of the difficulties about externals is that the
>> specification is so relaxed that there are so many different cases.
>> <philipm> Almost any change is going to break some untested corner
>> cases, while perhaps fixing other corner cases.
>> <@jcorvel> philipm: so, realistically, this will probably not be fixed
>> very soon (as in: in a 1.8.x release). At the very least, it needs (a
>> lot of) work for coming up with a detailed spec etc ... to try and get
>> a grip on the edge cases etc ...
>> <philipm> I don't think it will be fixed in the near future. I don't
>> know what we are going to do about externals.
>>
>>
>> So I guess that answers it. It's not going to work in the near future.
>> Short term I guess you have to find a workaround by structuring things
>> differently. Longer term, someone needs to "champion" an effort to
>> rework externals (starting by driving a discussion about it's
>> specification etc ...). Volunteers welcome :-).
>>
>
> And some additional context was given by Bert Huijben:
>
> <@Bert> jcorvel: Injecting file externals into a separate working copy
> was deemed to be a never intended feature and a potential security
> problem. It was explicitly disabled during 1.7 development.
> <@Bert> jcorvel: The proper fix is part of the externals redesign to
> something like viewspecs... where the file external just becomes a
> working copy containing one file, but properly defined by some
> ancestor
> <@Bert> A problem with this 'feature' was that it was possible to
> inject such externals, but not to remove them. The working copy that
> contained the external didn't even contain the definition.
>

FWIW, "injecting" externals inside other externals seems to "work" in
1.7 when you do it with "directory externals" (for some definition of
"work" ... I haven't tried all use cases with it). However, I wouldn't
recommend it (it feels pretty wonky to me -- not sure if there aren't
any side effects -- for instance, 'svn status' output looks a bit
strange, see below).

Here is a transcript of a quick experiment with SVN 1.7.7:

[[[
C:\Temp\svntest2>svnadmin create repos

C:\Temp\svntest2>svn co file:///C:/Temp/svntest2/repos wc
Checked out revision 0.

C:\Temp\svntest2>cd wc

C:\Temp\svntest2\wc>svn mkdir dir1 dir2
A dir1
A dir2

C:\Temp\svntest2\wc>svn ci -mm
Adding dir1
Adding dir2

Committed revision 1.

C:\Temp\svntest2\wc>svn up
Updating '.':
At revision 1.

C:\Temp\svntest2\wc>svn pe svn:externals . # edit externals in text editor
Set new value for property 'svn:externals' on '.'

C:\Temp\svntest2\wc>svn pg svn:externals
^/dir1 ext_dir1
^/dir2 ext_dir1/ext_dir2

C:\Temp\svntest2\wc>svn ci -mm
Sending .

Committed revision 2.

C:\Temp\svntest2\wc>svn up
Updating '.':

Fetching external item into 'ext_dir1':
External at revision 2.

Fetching external item into 'ext_dir1\ext_dir2':
External at revision 2.

At revision 2.

C:\Temp\svntest2\wc>svn st
X ext_dir1

Performing status on external item at 'ext_dir1':
? C:\Temp\svntest2\wc\ext_dir1\ext_dir2

Performing status on external item at 'ext_dir1\ext_dir2':

C:\Temp\svntest2\wc>echo test > dir1\file.txt

C:\Temp\svntest2\wc>svn add dir1\file.txt
A dir1\file.txt

C:\Temp\svntest2\wc>svn ci -mm
Adding dir1\file.txt
Transmitting file data .
Committed revision 3.

C:\Temp\svntest2\wc>svn up
Updating '.':

Fetching external item into 'ext_dir1':
A ext_dir1\file.txt
Updated external to revision 3.

Fetching external item into 'ext_dir1\ext_dir2':
External at revision 3.

At revision 3.

C:\Temp\svntest2\wc>echo blah > dir2\file.txt

C:\Temp\svntest2\wc>svn add dir2\file.txt
A dir2\file.txt

C:\Temp\svntest2\wc>svn ci -mm
Adding dir2\file.txt
Transmitting file data .
Committed revision 4.

C:\Temp\svntest2\wc>svn up
Updating '.':

Fetching external item into 'ext_dir1':
External at revision 4.

Fetching external item into 'ext_dir1\ext_dir2':
A ext_dir1\ext_dir2\file.txt
Updated external to revision 4.

At revision 4.

C:\Temp\svntest2\wc>svn st
X ext_dir1

Performing status on external item at 'ext_dir1':
? C:\Temp\svntest2\wc\ext_dir1\ext_dir2

Performing status on external item at 'ext_dir1\ext_dir2':

C:\Temp\svntest2\wc>cat ext_dir1\file.txt
test

C:\Temp\svntest2\wc>cat ext_dir1\ext_dir2\file.txt
blah
]]]

--
Johan