[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: Inexcusable BDB upgrade triple blunder

From: Max Bowsher <maxb1_at_ukf.net>
Date: 2006-02-19 15:57:25 CET

Hash: SHA1

Eric S. Raymond wrote:
> Once before I flamed the Subversion dev group over an unbelievable
> sequence of blunders, slipshod programming, and culpable
> obliviousness to the situation of actual users. Looks like it's time
> I have to do it again, and once again the botch combines a broken
> administrative procedure with pathetically inadequate documentation
> and an alleged recovery path that's a mirage.


I have a couple of questions for you:

First: What is the point of self-declaring a message as a flame? It
seems to me that the only thing this accomplishes is to deliberately
incite anger in the recipients.

Second: What happened to "don't claim you have found a bug unless you
are very, very sure of your ground." and "Courtesy never hurts, and
sometimes helps", as asserted in
http://www.catb.org/~esr/faqs/smart-questions.html ?

> Yes, this means I've tripped over the 4.2->4.3 BDB upgrade problem.
> My symptoms:
> svn: Unable to open an ra_local session to URL
> svn: Unable to open repository 'file:///home/esr/WWW/shipper/SVN/trunk/README'
> svn: Berkeley DB error for filesystem /home/esr/WWW/shipper/SVN/db while opening environment:
> DB_VERSION_MISMATCH: Database environment version mismatch
> svn: bdb: Program version 4.3 doesn't match environment version
> First blunder: screwing up db_recover() so a current svnadmin can't
> be used to upgrade the repository. But that would have been OK,
> except for...

Oops. I see we really have screwed up the FAQ entry's background info.
There is no such function db_recover(), as it mentions.
- From what I can remember, the issue was actually that the response of
BDB 4.3 to the invocation that Subversion has always used was different
to the response of BDB 4.0/1/2. I will go and investigate the situation,
and fix that misinformation.

> Second blunder: the FAQ answer. Fix it with an older version of
> svnadmin? Sounds like a peachy idea -- until the svnadmin run fails
> because it doesn't see the right version of the SVN repo shared
> libraries to dynamic-link to. A 100% predictable roadblock about
> which the FAQ entry says *nothing*.

No, this is neither 100% predictable, nor is it a significant roadblock.
If your older version of svnadmin uses shared libraries, then you should
install the appropriate versions. Presumably, they are available through
exactly the same means as the older version of svnadmin itself was.

> Oh, and by the way, there's that marvelously vague instruction to
> 'make a backup'. Why?

In case something unexpected goes wrong.

> With what?

Anything you like. 'cp' or 'tar', for instance.

> And how does one tell if one needs to *use* the backup?

The whole point of having a backup is to recover from _un_expected

> In general, this FAQ entry is so dizzyingly useless to anyone who
> doesn't inhabit the skull of a Subversion developer that one is left
> to wonder that kind of drugs the dolt who wrote it was on.

Vastly over-the-top ad-hominem attack, containing the grain of truth
that the author wasn't as thoroughly familiar with the issues as
would ideally be the case.

> Here, too, the lossage could easily have been contained -- by
> carrying links to downloadable statically linked svnadmin42 binaries
> for major platforms in the FAQ entry. But no. That would have made
> too much sense.

I'd like to point out that the Subversion project has _always_ made only
source-code releases, and has left the production of binaries up to
third parties. We have no control over what kinds of binaries they
produce. Certainly, we would be happy to append distro-specific notes to
the end of that FAQ entry if asked.

> Third blunder: Instead, diligent Googling turned up a set of recovery
> instructions bypassing svnadmin and using the db* utilities. Except
> that when applied on the most commonly deployed Linux distribution
> (Fedora Core) the instructions don't actually, you know, *work* or
> anything. I'm referring to this:
> http://lists.alioth.debian.org/pipermail/pkg-subversion-maintainers/2005-November/000268.html
> Buoyed up by the naive confidence that a reply I found on the
> Subversion dev list itself might embody something resembling a clue,
> I did as it suggested and wrote a script, intending to donate it
> back to the project. Here it is:
> ------------------------------------------------------------------------------
> #!/bin/sh
> #
> # Upgrade a local SVN repo from BDB 4.2. Follows the instructions in
> # http://lists.alioth.debian.org/pipermail/pkg-subversion-maintainers/2005-November/000268.html
> #
> repo=$1
> db42_checkpoint -1 -h ${repo}/db
> db42_recover -h ${repo}/db
> db42_archive -d -h ${repo}/db
> rm ${repo}/db/__db.???
> ------------------------------------------------------------------------------
> Notice that I couldn't actually follow the instructions, because the
> db utility names weren't right.

Yes, Fedora and Debian use different ways of disambiguating versions of
db_ utilities. Given the recipe was presented in a Debian context,
naturally the Debian names were used.

> I fixed them, and ran the script, and got this:
> db_checkpoint: open: No such file or directory
> db_recover: Ignoring log file: SVN/db/log.0000000004: unsupported log version 10
> db_recover: Invalid log file: log.0000000004: Invalid argument
> db_recover: PANIC: Invalid argument
> db_recover: PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
> db_recover: DB_ENV->open: DB_RUNRECOVERY: Fatal error, run database recovery
> db_archive: Ignoring log file: SVN/db/log.0000000004: unsupported log version 10
> db_archive: Invalid log file: log.0000000004: Invalid argument
> db_archive: PANIC: Invalid argument
> db_archive: PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
> db_archive: open: DB_RUNRECOVERY: Fatal error, run database recovery
> rm: cannot remove `SVN/db/__db.???': No such file or directory
> And yes, I did check that the repo path was correct. (It pretty much
> has to be for the second error line to have been generated.)

It seems that previous efforts to use BDB 4.3 on this repository had
created a 4.3-version log file.

> But I should not have had to try writing a recovery script iun the
> first place. I can see by the date of the aforementioned posting
> that this bug has been a known problem for *three* *months*. That
> tells me there is zero excuse for there not to have been a fix script
> downloadable through the Subversion FAQ page and dated at least two
> and a half months ago.

And what exactly do you take the list of steps in the FAQ entry to be,
other than a script written out in English to encourage care in working
through it?

To the best of our knowledge that sequence of instructions *works*.
Something caused additional problems in your case, but in general, it works.

> What in the fleeping frack were you guys *thinking*? Or, more likely,
> not-thinking.

We're thinking that:
(1) We provide source releases only, and have _always_ recommended the
use of 'svnadmin dump' and 'svnadmin load' as the safest way to change
BDB versions.
(2) We provided a recipe that many people have had success with to
short-circuit the admittedly lengthy 'svnadmin dump/load' technique.

> It's guaranteed that this same problem is going to mug anyone who
> updates from the 1.1.4 stock on FC4 to 1.2.3. You've now left
> anybody who relied on your upgrade path with invisibly broken
> repositories and no way out for at least *eight* *weeks*.

If the Fedora Core packagers have allowed a silent upgrade across BDB
versions without any sort of notification or warning, then they have not
acted in the best interests of their users. Please redirect your rant.
We've provided a way out, which apparently didn't work smoothly for you.

I guess I'll extract the grain of bug-report from the inferno, and see
if I can reproduce the specific failure mode you've reported.

> That's inexcusable. Whoever is responsible for this triple-play of
> incompetence should be deeply ashamed. It reflects badly in the
> entire dev group.

To recap:

1a: We failed to spot that the *background info* about the 4.2/4.3
issues in the FAQ entry was incorrect.

1b: Regarding the underlying problem that _should_ have been described
there - when the behaviour of a library we use changed, we attempted to
document the problems caused.

2: We used the words "make a backup" in our FAQ where we might have left
less room for confusion by saying "make a copy, just in case anything
unexpected goes wrong".

3: Some instructions you found on the dev list failed to work for you,
possibly because of other attempts to upgrade you tried before trying them.

Well, point 1a certainly warrants an exclamation of "Oops!" from the dev
team, but since it was a fault in background info only, it's relatively

As for 1b, in hindsight, given the amount of people running into
problems, we would have been better off modifying the code to smooth
over the issue, rather that only documenting the issue.

Point 2 - Maybe we could rephrase that.

Point 3 - Commiserations, but not our fault.

Max Bowsher.

Version: GnuPG v1.4.1 (Cygwin)


To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Sun Feb 19 15:58:06 2006

This is an archived mail posted to the Subversion Dev mailing list.