[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: svndumpfilter and svnsync?

From: Chris <devnullaccount_at_yahoo.se>
Date: Tue, 30 Oct 2018 13:36:09 +0000 (UTC)

Hi,

I just wanted to say that I finally managed to get the dump-filter-load cycle done and deploy the filtered repo. It did get rid of about 90% of the repository size so that's good for us. A big thanks to all who helped out with information in this mail thread! I would definitely have stranded somewhere without you.

One thing that was a bit annoying was when the dumpfilter threw an error because of a source of a file was missing when I filtered out a certain path and it turned out it had been copied to another location. The error message only prints out the missing source and not the destination, so I had to go into the repo to check the revision it crashed on to find the copy destination and add it to my filter list. Would have been nice if the error message could list both the source and the destination.

/Chris

--------------------------------------------
On Wed, 10/10/18, Johan Corveleyn <jcorvel_at_gmail.com> wrote:

 Subject: Re: svndumpfilter and svnsync?
 To: "Chris" <devnullaccount_at_yahoo.se>
 Cc: "Daniel Shahaf" <d.s_at_daniel.shahaf.name>, "Ryan Schmidt" <subversion-2018_at_ryandesign.com>, "Subversion" <users_at_subversion.apache.org>
 Date: Wednesday, October 10, 2018, 12:11 PM
 
 On Wed, Oct 10, 2018 at 11:18 AM
 Chris <devnullaccount_at_yahoo.se>
 wrote:
 ...
> >>>
 The syntax I used: svnadmin dump -q MYREPO | svndumpfilter
 exclude
> >>> --targets
 filterfile filterdump svnadmin load -q --no-flush-to-disk
> >>> --force-uuid -M 2048
 --bypass- prop-validation ./NEWREPO < filterdump
> >>>
>
>>> (I had to use the bypass-prop-validation due to
 some newline issues
> > in old log
 message, similar to this one
> >
 https://groups.google.com/forum/#!topic/subversion_users/P3ohZ-hKhCA,
> > don't know why they have wrong
 newlines, but the repo works as it is
>
> now...)
> >>
> >> Instead of ignoring wrong
 newlines, you could fix them using
>
>> svndumptool (using its eolfix-revprop command),
 originally at:
> >>
> >> http://svn.borg.ch/svndumptool/
> >>
> >>
 Newer fork at:
> >>
> >> https://github.com/jwiegley/svndumptool
> >
> > Also, as of
 version 1.10, svnadmin finally has an option to normalize
> > these on-the-fly during
 'load':
> > http://subversion.apache.org/docs/release-notes/1.10.html#normalize-
> > props
> >
> > It's a lot better to normalize
 these (either with the
> >
 --normalize-props option for 'svnadmin load' or by
 using svndumptool)
> > than to
 "bypass" them. Otherwise you'll run into this
 again later (if
> > you would
 dump+load again sometime in the future).
>
> I tried
 --normalize-props and I still got the same error which is
 why I
> switched over to bypass. Maybe
 I've run into some bug with --normalize-props.
> Unfortunately, I don't think I'll
 be able to create a script for reproducing
> the error since it happens far into a
 monster dump load.
> So I'll stick
 with the bypass for now or try the tool that Ryan
 suggested.
 
 In that case the
 culprit might be another property than svn:log (or it
 might be something like "non UTF-8
 encoded" but not EOL-related in
 svn:log). Possibly a "versioned"
 property like svn:ignore or some
 other
 property in the svn: namespace. This is more difficult to
 fix,
 but still it might be best to get rid
 of it or you'll run into it
 again in the
 future.
 
 See the very last
 bullet in:
 http://subversion.apache.org/faq.html#dumpload
 
 If that's indeed the
 problem, then you'll have to use that svndumptool
 that Ryan pointed you to.
 Quoting from that last bullet in the FAQ entry
 above:
 
 "This is more
 difficult to repair, because 'svn:ignore' is not
 a
 revision property (unlike svn:log, which
 can be manipulated with
 svnadmin
 setrevprop), but a versioned property (so it's part
 of
 history). Again, you can ignore this with
 --bypass-prop-validation.
 But since this is
 a corruption "in history", this can only be
 repaired
 with a dump+load, so this might be
 a good time to try and fix this (or
 you'll run into this again in the future).
 To repair it you can use a
 tool like
 svndumptool. But it only works on dump files, not as part
 of
 a pipe. So a possible way to go about it
 is: dump that single
 (corrupt) revision to a
 file, repair it ('svndumptool.py eolfix-prop
 svn:ignore svn.dump svn.dump.repaired'),
 load that single dumpfile,
 and then continue
 with a new "piped" command (like step (6) above).
 "
 
 I should note here
 that svnsync is more powerful in this regard: it
 does have the ability to normalize all of these
 on the fly. It's a
 real pity that
 'svnadmin load' doesn't (except for the svn:log
 EOL
 fixing). Doesn't *yet* that is,
 until a volunteer comes along that
 submits a
 patch for it ;-).
 
 Anyway, I
 hope you succeed in cleaning this up eventually :-).
 --
 
 Johan
 
Received on 2018-10-30 14:36:29 CET

This is an archived mail posted to the Subversion Users mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.