[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: I lost 7 bdb repositories yesterday!

From: Dave Camp <dave_at_thinbits.com>
Date: 2005-07-01 20:49:57 CEST

I'd like to thank everyone for their input. It sounds like the
universal suggestion is to get off of bdb as soon as possible and
upgrade to fsfs.

Is there anything I need to look out for when migrating from svn/bdb
1.1.2 to svn/fsfs 1.2 (other than following the instructions in the
release notes or whatever)?

Thanks,
Dave

On Jul 1, 2005, at 11:01 AM, Karan, Cem (Civ, ARL/CISD) wrote:

> I'm going to second what one of the other people on the list said;
> switch to fsfs ASAP. When I was running BDB based repositories, I
> was getting corruptions at the rate of 1 every 2-3 days (this was
> under an OS X 10.3 system). I switched to FSFS and couldn't be
> happier. No problems, no corruptions, no ANYTHING (except what I
> want and expect).
>
> I don't know if you're building your own binaries, but if you want
> to and need help, mail me off list.
>
> Good luck,
> Cem Karan
>
>
> -----Original Message-----
> From: Dave Camp [mailto:dave@thinbits.com]
> Sent: Fri 01-Jul-05 11:17 AM
> To: Subversion Users
> Subject: I lost 7 bdb repositories yesterday!
>
> This was terrible (but I guess that goes without saying). The server
> "unexpectedly" restarted and when it cam back up, 10 of our 16
> repositories were broken (a variety of errors when a svn co was
> performed). 2-3 were fixable with svnadmin recover. The remainder
> were not fixable with either svnadmin recover or db_recover -c. In
> each case, they reported problems with the "magic number" of one of
> the log files, and list-unused-dblogs always reported that the
> affected files were needed.
>
> Since we've got 10 engineers waiting to get back to work, the
> immediate solution was to restore the broken repositories from the
> nightly backups. Not pleasant, but at least no work was lost this
> time. One of the repositories is 24 Gigs of source and data for a
> massive project.
>
> So, after getting my co-workers to switch to subversion, I'm now
> being questioned as to how a catastrophe like this is possible with a
> version control system (we've had other problems in the past too).
> I'm questioning it as well. How can subversion/bdb be this brittle?
> It seems as though you can't rely on it at all if something goes
> wrong! We realize that when a machine goes down, any files that were
> being written at the time are obviously suspect, but we lost
> repositories that have not been used in weeks or months. We have
> checked drive and data corruption on the rest of the server and there
> is none. Only the log.xxx files report issues.
>
> Our setup:
> Xserve, Mac OS X 10.3, 250 Gig mirrored raid, HFS+ (Journaled)
> filesystem
> svnserve 1.1.2, db-4.2.52
> All access is via ssh. No local access or Apache.
> Clients are a mixture of Linux, Mac, and Windows.
>
> Questions I was hoping someone could answer:
>
> - Some of the damaged repositories were not in use at the time
> (idle for days or months). How is that possible without drive or
> filesystem damage? There were no svnserve processes running on those
> files at the time.
>
> - Is this expected behavior for bdb repositories? This is not the
> first time we've had unrecoverable damage to them (previous times
> seemed to be from normal use).
>
> - Does fsfs solve the fragility issues that bdb seems to have? If
> moving the repositories to fsfs will make these problems a thing of
> the past, I'm there.
>
> Management (and my fellow engineers) are going to be wanting some
> answers from me in the next few days. If I can't explain how this can
> be avoided in the future It's a good bet there will be a call to move
> to a more stable platform. I really don't want that, but I'm at a
> point where I don't trust subversion either right now (one of the
> projects I had to restore was mine, and I'm supposed to be Beta
> tomorrow).
>
> Can anyone help me figure out what happened and how to prevent this
> from happening again.
>
> Thanks,
> Dave
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Received on Fri Jul 1 21:11:52 2005

This is an archived mail posted to the Subversion Users mailing list.