Re: svn: bdb: PANIC / Reliability?

On Monday 09 May 2005 11:22, John Szakmeister wrote:
> On Monday 09 May 2005 04:13, Georg Wittenburg wrote:
> > Good morning,
> >
> > I switched over from CVS to SVN 1.1.4 about a month ago and have been
> > accessing my repository using file:/// and svn+ssh:// as one of a group
> > of two people that have write access. No fancy stuff, imho. While my
> > last commit before this weekend went in without trouble, now my
> > repository seems to be corrupt.
> >
> > Actually, I already recovered once this Saturday using a procedure I
> > found online (sorry, no URL) that included rm __db.00*, rm
> > log.000000001*, db_recover -cv in the db directory. Things worked fine
> > after that and I was able to commit and update. I do have a backup from
> > that time, so my data is safe and I'm not in trouble. However, I DO
> > worry about the reliability of SVN because this morning I got the
> > following errors (and I don't think my first recovery attempt was
> > flawed, because things were working fine just afterwards):
> >
> > $svn up
> > svn: Berkeley DB error while checkpointing after Berkeley DB
> > transaction for filesystem /home/bude/wittenbu/svn/db:
> > Invalid argument
> > svn: bdb: DB_ENV->log_flush: LSN of 19/323156 past current end-of-log
> > of 1/38823
> > svn: bdb: Database environment corrupt; the wrong log files may have
> > been removed or incompatible database files imported from another
> > environment svn: bdb: representations: unable to flush page: 0
> > svn: bdb: txn_checkpoint: failed to flush the buffer cache Invalid
> > argument
>
> What version of BDB are you using, and have you recently upgraded it at
> all? Also, does the repository lie on a local filesystem, or are you
> using it on an remote share of some sort?

BDB is version 4.2, to be exact the Debian package is 4.2.52-18. The
filesystem is mounted over the network, I assume using NFS but autofs won't
tell me.

> > $ svnadmin recover .
> > Repository lock acquired.
> > Please wait; recovering the repository may take some time...
> > svn: DB_RUNRECOVERY: Fatal error, run database recovery
> > svn: bdb: Log sequence error: page LSN 1 7802; previous LSN 19 316241
> > svn: bdb: Recovery function for LSN 1 7802 failed on forward pass
> > svn: bdb: PANIC: Invalid argument
> > svn: bdb: PANIC: fatal region error detected; run recovery
> > svn: bdb: PANIC: fatal region error detected; run recovery
> > svn: bdb: PANIC: fatal region error detected; run recovery
> > svn: bdb: PANIC: fatal region error detected; run recovery
> > svn: bdb: PANIC: fatal region error detected; run recovery
> > svn: bdb: PANIC: fatal region error detected; run recovery
> > svn: bdb: PANIC: fatal region error detected; run recovery
> > svn: bdb: PANIC: fatal region error detected; run recovery
> > svn: bdb: PANIC: fatal region error detected; run recovery
> > svn: bdb: PANIC: fatal region error detected; run recovery
> > svn: bdb: PANIC: fatal region error detected; run recovery
> > svn: bdb: PANIC: fatal region error detected; run recovery
> > svn: bdb: PANIC: fatal region error detected; run recovery
> > svn: bdb: PANIC: fatal region error detected; run recovery
> > svn: bdb: PANIC: fatal region error detected; run recovery
> > svn: bdb: PANIC: fatal region error detected; run recovery
> >
> > Usually I wouldn't mind helping the developers to iron this out, except
> > for the fact the I'm currently writing my master thesis and from an
> > users perspective just need a SCM system that is ROCK SOLID!
>
> I understand the frustration, but there is something wrong with your set
> up. I've been using the BDB backend for quite some time (over 2 years
> now), and have had exactly 1 problem... it was my fault.
>
> > I'm sorry to say that SVN just doesn't seem to be there yet. Further,
> > the entry in the FAQ about DB_RUNRECOVERY isn't particularly
> > encouraging, as it tells me that my data not only gets corrupted by
> > processes crashing but also by incorrect permissions. I'm sorry for the
> > people who put a lot of effort into SVN, but my perception is that this
> > is somewhat too fragile for production use, or rather at least for me.
>
> Again, it's fine if you set things up correctly. Admittedly, that's
> easier if you use the FSFS backend.
>
> > Now, I fixed my repository AGAIN by repeating the three steps pointed
> > out above. However, I'm getting quite close to switching back to CVS
> > because from my perspective as an user, this trouble is just not worth
> > it.
> >
> > While I guess that this mailing list gets a lot of noise about these
> > kind of problems (Google gives me 318 hits for the first part of the
> > subject of this mail), I'd appreciate if you could share some insight,
> > particularly whether my problems are just unique events that'll never
> > happen to anyone and I was just being unlucky, or whether I'd just have
> > to expect these kind of problems when using SVN. If there's interest,
> > I've also tarballs of the corrupted and fixed repositories.
>
> There is one of several problems. More than likely the repository is
> being stored on a share (don't do that with BDB), or you have a buggy
> version of BDB (we seem to bring out several bugs in the 4.1.x line).
> The only other item that tends to catch people is upgrades. If BDB was
> upgraded, you need to do a few things to your repository to use the new
> version of BDB.
>
> Honestly, if you're using svn+ssh://, and FSFS repository is probably the
> best way to go. Using svn+ssh:// correctly generally requires a script
> to ensure that the permissions are correct. FSFS can do it without said
> script.

Thanks for your encouragement and for all the replies on this list. :)

I've migrated the repository over to FSFS and things seem to work now (link to
the HOWTO in my previous post).

I have to admit that in retrospect the critical bit of information isn't that
hard to find: FAQ -> "My repository seems to get stuck..." -> "How do I set
repository permissions..." -> SVN Book, chapter 6, bottom. Three clicks if
you know what you're locking for.

As word has it that the problem with BDB will be addressed in the next major
version, mind if I make a suggestions for the next minor version to allow
users (who like me prefer not to read the whole 933KB SVN Book, blame me) to
sort out this stuff by themselves: Make the SVN's error messages a bit more
descriptive:
- With "svn up" instead of "svn: Berkeley DB error while checkpointing after
Berkeley DB transaction for filesystem /home/bude/wittenbu/svn/db: Invalid
argument" add something like "An error occurred probably because of incorrect
setup. Please have a look at the documentation and run svnadmin
recover /path/to/repository.".
- With "svnadmin recover" instead of "svn: DB_RUNRECOVERY: Fatal error, run
database recovery" add something like "...BUT YOUR DATA IS PROBABLY OK (see
doc XY for ways to recover manually)." where XY would be a pointer to the
steps I described in my first mail.
As my kind of misconfiguration seems to be somewhat common, there's really no
reason to leave the users out in the dark.

Further, one could argue about whether it would be nice for svnadmin to check
for proper permission, umask, etc. when working on a repository and display
warnings even in non-verbose mode.

The above is just my two cents to keep other users in the future from worrying
about their projects early on Monday mornings. :)

Thanks again for your help,
   Georg

-- 
Georg Wittenburg
http://page.mi.fu-berlin.de/~wittenbu/