Re: predecessor count for the root node-revision is wrong message

Jason Wong wrote on Thu, Mar 01, 2012 at 10:01:26 -0800:
> I have had a developer here create a build of the latest SVN code
> with your changes you mentioned in r1294470 for the svnadmin verify

Okay, that's great news, for two reasons:

1. It means building svn on windows isn't as painful as it used to be :)

2. It means I can ask you to build a custom server with the 'inprocess'
cache disabled, or (if all else fails) to bisect, per my previous email.

One of the things you could try is to disable caching: simply modify
the function create_cache() in libsvn_fs_fs/caching.c to always return
NULL in *CACHE_P. See below for another suggestion.

> command. We have run 'svnadmin verify' against every revision of our
> hotcopy of our repository taken when we first brought this issue to
> the forums and are now tracking down each of the revisions to see
> what actions were being done at those times.
>

Thanks! I do hope this work enables us to pinpoint and fix the bug.

> From the results, we see 25 error messages for predecessor count is
> wrong and the first one appeared on January 26, 2011. Near that time
> the following events occurred:
> Jan. 14, 2011 - svn upgraded from 1.6.6 to 1.6.15
> Jan. 14, 2011 - Apache HTTP server upgraded from 2.2.15 to 2.2.17
> Jan. 21, 2011 - repository was pruned to delete some binary files.
>
> Between January and our upgrade in Dec. to 1.7.1, we have had about
> 14,000 revisions and seen only 25 instances of this node revision
> issue. During the times we had these errors, we were using svn
> versions 1.6.15 and 1.6.16.
>

Thanks, very valuable information.

I've reviewed the 1.6.6->1.6.15 diff, and I have the following
suggestions:

- Change subversion/libsvn_fs_fs/fs.h such that
  SVN_FS_FS__USE_LOCK_MUTEX is set to 1. It was set to 1 in 1.6.6
  but to 0 in 1.6.15.

  (This wouldn't explain why ASF saw it, but it might explain why you're
  seeing it.)

> Fail2ban from what I could find does not look like it has a Windows
> port which I currently have my production environment hosted on.
>

Yeah, sorry. But you can write a cron job -- I mean, a Scheduled Task
-- that greps your error logs for "160004" every night and mails you it
it found anything, right?

That's the error code to watch for for many FS error conditions:
% ./tools/dev/which-error.py E160004
00160004 SVN_ERR_FS_CORRUPT

> Thanks.
>
> Jason

For convenience I'm attaching a patch that implements both of my
suggestions. Let us know please if it has any effect.

Cheers,

Daniel