[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: Subversion FSFS corruption

From: Ted Gould <ted_at_gould.cx>
Date: 2005-08-23 23:24:37 CEST

kfogel@collab.net wrote:
>>I believe that one of the numbers on the 'text:' line is probably the
>>diff size? Which one? I'd like to determine whether the diff is too
>>small or if the number is.
>
> Well, the fact that it says "PLAIN" at the beginning makes me think
> it's not a delta at all, but a fulltext representation. I'm getting
> this from reading

I wouldn't be surprised by this. I believe that this is the original
import of the picture -- so it wouldn't be a very significant delta if
it is one at all.

> Does any of this help you debug it further? (I'm not saying we
> shouldn't look at your repository too, of course, but if you can make
> some progress on it, that's a valuable parallelization -- so please
> understand that I'm not saying "talk to the hand"! :-). I hope the
> above link helps.

Yes, I'll try this stuff recommend by you and Greg tonight and see what
I can come up with. I realize that this will be hard to debug as I
don't have the original file, and the revision files are so large.

> How big is your total repository?

Approximately 12 GB. If you're bored and want to read how I'm using
subversion for my photos, I've written about my complete work flow here:
http://gould.cx/ted/blog/Photography_Workflow.html The larger revisions
tend to be about 500 MB as I have a 512 MB flash card for my camera :)

Greg Hudson wrote:
> On Mon, 2005-08-22 at 23:51 -0700, Ted Gould wrote:
>>I sent this message the user list on Friday, but I believe that may
>>have been the wrong list.
>
> That's the right list for this sort of problem, but your problem is deep
> enough that only a couple of people are qualified to look into it
> without studying up, and one of them (me) doesn't read the user list.

Okay. If you want me to move this thread, just say so.

> The problem seems to be within the delta itself. Chunk 30 or so, at
> offset 3073127 into the delta, contains instructions which produce
> 100675 bytes when they are supposed to produce 102400 bytes.

This is odd to me, as I believe that this is a direct import of the
picture. I don't know anything about the Delta format, but it seems
that there should be very few instructions at head of each chunk. More
of a "just use this data" type command. Could this command be changed?
  Loosing one macroblock in the JPEG would be better than loosing the
whole image.

>>2) Assuming I give up on ever seeing img_5633.jpg again, how do I get
>>my repository back online?
>
> If there are no derivatives of img_5633.jpg, then you can resolve this
> problem through very delicate surgery. The following seems to work:
>
> * At offset 125184612 in the file, change the word "DELTA" to "PLAIN".
>
> * In the metadata, where it says
>
> text: 671 125184612 3924753 3922699 9112d223ba3e88b19e846e429f0ef361
>
> change that to:
>
> text: 671 125184612 0 0 d41d8cd98f00b204e9800998ecf8427e
>
> where the spaces are important. The length of the line cannot
> change, since FSFS relies heavily on offsets into files.
>
> That will morph img_5633.jpg into an empty file.

I will definitely try this. Loosing one picture is much better than
having a non-working repository :) I couldn't find any other references
to this file in later revisions, so I think this will work out.

>>3) Is this a bug? Any other data that you need?
>
> Well, I can see two possibilities:
>
> * There is a bug in our binary delta code. This would be pretty
> surprising, but it's a possibility. Unfortunately, the way to look into
> this would be to take the original img_5633.jpg and see if the problem
> can be reproduced by checking it in. If that file is gone, it's hard to
> see a way forward.

Unfortunately this seems to be the case. I thought I might have the
file somewhere, but it seems that I don't. I'm worried though, that
this might be the case of where the problem is coming from.

> * The rev file was subtly corrupted during or after its creation by
> something other than Subversion. Usually such corruption manifests
> itself as a large swath of 0 bytes, which is not the case here.

I don't believe this is the case as I have checked 3 backups that are
identical. Ofcourse, there is an amount of time where the file existed,
but wasn't backed up, so I can't prove it.

Where is the documentation of the delta format? I know it must exist,
but I can't seem to find it. This is a WebDAV thing, right?

        Thanks for all your help,
                Ted

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Tue Aug 23 23:32:34 2005

This is an archived mail posted to the Subversion Dev mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.