On Sun, Feb 05, 2006 at 10:55:42AM -0500, Marc Sherman wrote:
> John, do you or Malcome have a tool (or instructions) for repairing
> these corrupted repositories that's publicly available? I've been
> living in fear of this bug hitting the repositories I manage at work for
> months now. :)
>
I think 'living in fear' is a bit strong: we've had, what, five/six
reports? ;) Maybe I should explain what I think is happening:
We start off by writing the rev-file for a particular file that's
changed. We hit an error, and so part of the file's contents is left
in an in-memory buffer. The file looks like this:
Disk: 1=================
Mem: ...
For whatever reason, we don't abort the transaction, but instead write
out the file's contents again. This succeeds, so now we have this:
Disk: 1=================2########################>
Mem: ...
Note that we still have the in-memory buffer from the first file open.
The revfile now has some crud at the front, but it's otherwise perfectly
valid. The problem comes when we finally destroy the pool that the
first filehandle was in: it flushes its buffered contents to the revfile
(this might even be after we've committed the transaction).
Now we have this:
Disk: 1=================...######################>
The start of the second noderev (the successfully written one) has been
overwritten by the middle of the second, corrupting it. Fortunately,
the fix is simple: since the two noderevs are identical, we can copy&paste
the start of the second noderev from earlier:
Disk: 1=================1==######################>
And now the noderev is valid again. This is what John's tool does, IIRC.
With that in mind, some mitigating factors/guesses:
* We think this can only happen if you get something like a write error
while writing the file. In about half(?) of the cases, users subsequently
reported disk problems - so if your disk is okay, you're probably less
likely to hit this.
* Possibly coupled with the previous, versions of APR prior to 0.9.7
didn't report write errors on buffered files at all - if you're using
APR 0.9.7, you might be in a better position.
* I think every report has been for a server using mod_dav_svn.
It's possible that svnserve isn't affected (or then again, that it's
just less commonly used).
* Finally, if we are overwriting the revfile after we commit the
transaction, it's possible that this is only a problem on non-Windows
servers, since I don't think you can rename an open file on Windows.
The biggest problem is that we've just not been able to reproduce this,
so we can't see exactly what's going on.
> Also, what's the current timeline for fixing this problem in subversion?
> Is it expected to go into a 1.3.x release, or will it have to wait for
> 1.4.0?
>
It's not currently planned for a release, since we're not entirely sure
how best to fix it. There's a couple of approaches:
1. Fix the problem at source (i.e. mod_dav_svn, if it's that).
Trouble is, we aren't really sure how to proceed with tracking down the
source, so this is a bit of a dead end. This also doesn't fix any other
badly-behaved clients.
2. Kill the FSFS transaction if we spot a fatal error. Except that we
have to distinguish 'fatal' from 'non-fatal' errors, and as previously
mentioned, APR 0.9.6 and below don't report errors writing buffered
files at all. (and the error-handling mechanics would likely be quite
intrusive).
3. Make sure FSFS uses only one file handle for the transaction's revfile.
This wouldn't be a bad idea anyway, since we currently do a lot of
open/seek-to-end/close throughout a transaction. The problem here is
that there's no unique 'transaction' object that a client has to use to
modify a transaction (it can legitimately use two 'transaction handle'
objects opened from different 'filesystem handle' objects to update the
same transaction), so we'd have to maintain some kind of synchronised
reference-counted hash of {filesystem UUID, txn id} => {transaction
object}, which is quite an overhead just for this.
4. Finally, the idea I'm currently favouring: Make sure that everywhere
we open a file for write in FSFS, we ensure that it's closed before we
return. This is simply a matter of calling apr_file_close() everywhere,
including in our error paths. It's likely to be fairly intrusive (for
libsvn_fs_fs/fs_fs.c), but not too complex, I hope.
Regards,
Malcolm
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Wed Feb 8 13:06:19 2006