On Fri, Mar 2, 2012 at 2:58 AM, Daniel Shahaf <danielsh_at_elego.de> wrote:
> Jason Wong wrote on Thu, Mar 01, 2012 at 10:01:26 -0800:
>> I have had a developer here create a build of the latest SVN code
>> with your changes you mentioned in r1294470 for the svnadmin verify
>
> Okay, that's great news, for two reasons:
>
> 1. It means building svn on windows isn't as painful as it used to be :)
Actually, it did take some work to get it going as we did not have
another system available to us and also did not have VC++ 6. We had
to use VS 2010 in order to do this. Also, for the other components
required (python,perl etc), the files after the install were copied
to the workstation to see if it would work as we did not want to
change the current workstation configuration by running the
installers. All in all, it did seem to work.
>
> 2. It means I can ask you to build a custom server with the 'inprocess'
> cache disabled, or (if all else fails) to bisect, per my previous email.
>
> One of the things you could try is to disable caching: simply modify
> the function create_cache() in libsvn_fs_fs/caching.c to always return
> NULL in *CACHE_P. See below for another suggestion.
>
>> command. We have run 'svnadmin verify' against every revision of our
>> hotcopy of our repository taken when we first brought this issue to
>> the forums and are now tracking down each of the revisions to see
>> what actions were being done at those times.
>>
>
> Thanks! I do hope this work enables us to pinpoint and fix the bug.
I will be going through the list to see what else was happening at the
same time on the apache server since it was alluded to that there may
be concurrency issues. I know the last two times that this error has
popped up, we had two svn operations starting at around the same time
according to the Apache logs. I will go through the previous apache
history to see if this was always the case or not.
>
>> From the results, we see 25 error messages for predecessor count is
>> wrong and the first one appeared on January 26, 2011. Near that time
>> the following events occurred:
>> Jan. 14, 2011 - svn upgraded from 1.6.6 to 1.6.15
>> Jan. 14, 2011 - Apache HTTP server upgraded from 2.2.15 to 2.2.17
>> Jan. 21, 2011 - repository was pruned to delete some binary files.
>>
>> Between January and our upgrade in Dec. to 1.7.1, we have had about
>> 14,000 revisions and seen only 25 instances of this node revision
>> issue. During the times we had these errors, we were using svn
>> versions 1.6.15 and 1.6.16.
>>
>
> Thanks, very valuable information.
>
> I've reviewed the 1.6.6->1.6.15 diff, and I have the following
> suggestions:
>
> - Change subversion/libsvn_fs_fs/fs.h such that
> SVN_FS_FS__USE_LOCK_MUTEX is set to 1. It was set to 1 in 1.6.6
> but to 0 in 1.6.15.
>
> (This wouldn't explain why ASF saw it, but it might explain why you're
> seeing it.)
>
>> Fail2ban from what I could find does not look like it has a Windows
>> port which I currently have my production environment hosted on.
>>
>
> Yeah, sorry. But you can write a cron job -- I mean, a Scheduled Task
> -- that greps your error logs for "160004" every night and mails you it
> it found anything, right?
>
> That's the error code to watch for for many FS error conditions:
> % ./tools/dev/which-error.py E160004
> 00160004 SVN_ERR_FS_CORRUPT
>
I will look into it. We did ask developers to note any error messages
that they see from tortoisesvn now as the last time we saw the error
message pop up, we asked the developer what happened and he said that
an error message popped up and he just tried to check in again and it
worked. We will note the exact message next time.
>> Thanks.
>>
>> Jason
>
> For convenience I'm attaching a patch that implements both of my
> suggestions. Let us know please if it has any effect.
>
I will forward this to the developer to look at.
> Cheers,
>
> Daniel
>
Hi.
See replies above. I will post what we find.
Thanks.
Jason
Received on 2012-03-02 16:33:12 CET