Mirror_metadata corruption on OS X 10.4

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Mirror_metadata corruption on OS X 10.4

Kevin Horton
I made a fresh backup with rdiff-backup 1.1.0, then did an  
incremental backup.  It failed, complaining about not being able to  
read a 13M xml file.  The file was quite readable, and I couldn't see  
any reason for the failure, so I tried again.  This time the backup  
failed complaining about corrupt mirror_metadata.

Processing changed file rdiff-backup.patch
Regular copying ('rdiff-backup.patch',) to /Volumes/Ext_BU/Users/kwh/
rdiffbu/rdiff-backup.tmp.790
Writing file object to /Volumes/Ext_BU/Users/kwh/rdiffbu/rdiff-
backup.tmp.790
Copying attributes from ('rdiff-backup.patch',) to /Volumes/Ext_BU/
Users/kwh/rdiffbu/rdiff-backup.tmp.790
Writing resource fork to ('rdiff-backup.tmp.790',)
Setting time of /Volumes/Ext_BU/Users/kwh/rdiffbu/rdiff-backup.tmp.
790 to 1130462329
Incrementing mirror file /Volumes/Ext_BU/Users/kwh/rdiffbu/rdiff-
backup.patch
Touching /Volumes/Ext_BU/Users/kwh/rdiffbu/rdiff-backup-data/
increments/rdiff-backup.patch.2005-10-31T20;05810;05849-05;05800.missing
Renaming /Volumes/Ext_BU/Users/kwh/rdiffbu/rdiff-backup.tmp.790 to /
Volumes/Ext_BU/Users/kwh/rdiffbu/rdiff-backup.patch
Traceback (most recent call last):
   File "/sw/bin/rdiff-backup", line 23, in ?
     rdiff_backup.Main.Main(sys.argv[1:])
   File "/sw/lib/python2.4/site-packages/rdiff_backup/Main.py", line  
293, in Main
     take_action(rps)
   File "/sw/lib/python2.4/site-packages/rdiff_backup/Main.py", line  
263, in take_action
     elif action == "backup": Backup(rps[0], rps[1])
   File "/sw/lib/python2.4/site-packages/rdiff_backup/Main.py", line  
313, in Backup
     backup.Mirror_and_increment(rpin, rpout, incdir)
   File "/sw/lib/python2.4/site-packages/rdiff_backup/backup.py",  
line 51, in Mirror_and_increment
     DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
   File "/sw/lib/python2.4/site-packages/rdiff_backup/backup.py",  
line 227, in patch_and_increment
     for diff in rorpiter.FillInIter(source_diffiter, dest_rpath):
   File "/sw/lib/python2.4/site-packages/rdiff_backup/rorpiter.py",  
line 177, in FillInIter
     for rp in rpiter:
   File "/sw/lib/python2.4/site-packages/rdiff_backup/backup.py",  
line 103, in get_diffs
     for dest_sig in dest_sigiter:
   File "/sw/lib/python2.4/site-packages/rdiff_backup/backup.py",  
line 167, in get_sigs
     for src_rorp, dest_rorp in cls.CCPP:
   File "/sw/lib/python2.4/site-packages/rdiff_backup/backup.py",  
line 301, in next
     source_rorp, dest_rorp = self.iter.next()
   File "/sw/lib/python2.4/site-packages/rdiff_backup/rorpiter.py",  
line 100, in Collate2Iters
     try: relem2 = riter2.next()
   File "/sw/lib/python2.4/site-packages/rdiff_backup/metadata.py",  
line 256, in iterate
     next_pos = self.get_next_pos()
   File "/sw/lib/python2.4/site-packages/rdiff_backup/metadata.py",  
line 247, in get_next_pos
     newbuf = self.fileobj.read(self.blocksize)
   File "/sw/lib/python2.4/gzip.py", line 225, in read
     self._read(readsize)
   File "/sw/lib/python2.4/gzip.py", line 290, in _read
     self._read_eof()
   File "/sw/lib/python2.4/gzip.py", line 309, in _read_eof
     raise IOError, "CRC check failed"
IOError: CRC check failed
Exception exceptions.TypeError: "'NoneType' object is not callable"  
in <bound method GzipFile.__del__ of <gzip open file '/Volumes/Ext_BU/
Users/kwh/rdiffbu/rdiff-backup-data/file_statistics.
2005-11-01T21;05811;05811-05;05800.data.gz', mode 'wb' at 0x3892a8  
0x7a7530>> ignored
Exception exceptions.TypeError: "'NoneType' object is not callable"  
in <bound method GzipFile.__del__ of <gzip open file '/Volumes/Ext_BU/
mirror_metadata.2005-10-31T20\;05810\;05849-05\;05800.snapshot.gz  
mirror_metadata.2005-11-01T21\;05811\;05811-05\;05800.snapshot.gz

Users/kwh/rdiffbu/rdiff-backup-data/error_log.
2005-11-01T21;05811;05811-05;05800.data.gz', mode 'wb' at 0x783f50  
0x76b378>> ignored
mirror_metadata.2005-10-31T20\;05810\;05849-05\;05800.snapshot.gz  
mirror_metadata.2005-11-01T21\;05811\;05811-05\;05800.snapshot.gz
Exception exceptions.TypeError: "'NoneType' object is not callable"  
in <bound method GzipFile.__del__ of <gzip open file '/Volumes/Ext_BU/
Users/kwh/rdiffbu/rdiff-backup-data/mirror_metadata.
2005-11-01T21;05811;05811-05;05800.snapshot.gz', mode 'wb' at  
0x7ae140 0x79efa8>> ignored

I checked the last mirror_metadata file, and it was corrupt.

1. Is this corruption a result of the earlier failure to backup?  
Should I need to do anything special to recover from an error, or can  
I just try again.

2. I should have kept a better log of the first error.  I tried to  
gunzip the error log, and that failed with:

error_log.2005-11-01T21;05811;05811-05;05800.data.gz: unexpected end  
of file

3. I note that many of the file names in the rdiff-backup-data  
directory have strange encoding in the file names.  Why?  See:

rw-------    1 root     staff        226M Nov  1 22:17 backup.log
-rw-------    1 root     staff          14 Oct 31 20:10 chars_to_quote
-rw-------    1 root     staff          10 Oct 31 21:51  
current_mirror.2005-10-31T20;05810;05849-05;05800.data
-rw-------    1 root     staff           9 Nov  1 21:42  
current_mirror.2005-11-01T21;05811;05811-05;05800.data
-rw-------    1 root     staff         127 Oct 31 21:51 error_log.
2005-10-31T20;05810;05849-05;05800.data.gz
-rw-------    1 root     staff         114 Nov  1 22:17 error_log.
2005-11-01T21;05811;05811-05;05800.data.gz
-rw-------    1 root     staff        1.4M Oct 31 21:51  
file_statistics.2005-10-31T20;05810;05849-05;05800.data.gz
-rw-------    1 root     staff        1.6M Nov  1 22:17  
file_statistics.2005-11-01T21;05811;05811-05;05800.data.gz
drwx------   25 root     staff         850 Nov  1 22:17 increments/
-rwxr-xr-x    1 kwh      staff           0 Oct 31 20:10 increments.
2005-10-31T20;05810;05849-05;05800.dir*
-rw-------    1 root     staff        215M Oct 31 21:51  
mirror_metadata.2005-10-31T20;05810;05849-05;05800.snapshot.gz
-rw-------    1 root     staff        215M Nov  1 22:17  
mirror_metadata.2005-11-01T21;05811;05811-05;05800.snapshot.gz
-rw-------    1 root     staff         524 Oct 31 21:51  
session_statistics.2005-10-31T20;05810;05849-05;05800.data

Kevin Horton
Ottawa, Canada




_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Mirror_metadata corruption on OS X 10.4

Ben Escoto
>>>>> Kevin Horton <[hidden email]>
>>>>> wrote the following on Wed, 2 Nov 2005 06:01:12 -0500
> I made a fresh backup with rdiff-backup 1.1.0, then did an  
> incremental backup.  It failed, complaining about not being able to  
> read a 13M xml file.

Not sure what would cause that.

> The file was quite readable, and I couldn't see any reason for the
> failure, so I tried again.  This time the backup failed complaining
> about corrupt mirror_metadata.

Did it try to regress at the beginning of the session after the failed
one?

> I checked the last mirror_metadata file, and it was corrupt.
>
> 1. Is this corruption a result of the earlier failure to backup?  
> Should I need to do anything special to recover from an error, or can  
> I just try again.

It's normal for the mirror_metadata and other things to be corrupt
after a failed session (since they're only partially written) but they
should all be cleaned up during the beginning "regress" period of the
next session.

> 3. I note that many of the file names in the rdiff-backup-data  
> directory have strange encoding in the file names.  Why?  See:

If the repository is already quoted then rdiff-backup will maintain
the quoting for backwards compatibility.  But if you start a new
repository with a new version (maybe just 1.1.1/ maybe 1.1.0) then it
should be (and stay) unquoted.


--
Ben Escoto

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Mirror_metadata corruption on OS X 10.4

Kevin Horton
On 5 Nov 2005, at 23:58, Ben Escoto wrote:

>>>>>> Kevin Horton <[hidden email]>
>>>>>> wrote the following on Wed, 2 Nov 2005 06:01:12 -0500
>> I made a fresh backup with rdiff-backup 1.1.0, then did an
>> incremental backup.  It failed, complaining about not being able to
>> read a 13M xml file.
>
> Not sure what would cause that.
>
>> The file was quite readable, and I couldn't see any reason for the
>> failure, so I tried again.  This time the backup failed complaining
>> about corrupt mirror_metadata.
>
> Did it try to regress at the beginning of the session after the failed
> one?

I'm not sure to be honest.  I didn't note any strange messages at the  
start to say it was regressing, but I may not have been paying close  
enough attention.  I've already deleted that destination, so I can't  
try again.

I'll build v1.1.2 today and start trying it out.

Kevin



_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Mirror_metadata corruption on OS X 10.4

Kevin Horton

On 6 Nov 2005, at 09:12, Ben Escoto wrote:

>>>>>> Kevin Horton <[hidden email]>
>>>>>> wrote the following on Sun, 6 Nov 2005 07:13:45 -0500
>>
>> I'll build v1.1.2 today and start trying it out.
>
> Thanks, you'll also need the carbonfile patch that I mentioned to  
> Blair
> Zajac:
>
> http://savannah.nongnu.org/cgi-bin/viewcvs/rdiff-backup/rdiff- 
> backup/rdiff_backup/rpath.py.diff?r2=1.95&r1=1.94&diff_format=u
>
> Sorry for all these Mac OS X problems..  I usually depend heavily on
> automatic testing---there are a lot of tests (200ish?) that get run on
> rdiff-backup before a release.  But clearly those don't help find
> errors in the Mac specific code.

I grabbed the patch.  Thanks.

I'm not upset about the OS X specific problems.  It is completely  
understandable that you can't test for this yourself.

How do you do your automatic testing?  I wonder if I can use my  
machine to do some automatic tests on OS X?  I'm willing to try, but  
I am not a developer, so there are no guarantees that I will be able  
to make it work.

Kevin Horton
Ottawa, Canada




_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Mirror_metadata corruption on OS X 10.4

Ben Escoto
>>>>> Kevin Horton <[hidden email]>
>>>>> wrote the following on Sun, 6 Nov 2005 10:12:00 -0500
>
> How do you do your automatic testing?  I wonder if I can use my
> machine to do some automatic tests on OS X?  I'm willing to try, but
> I am not a developer, so there are no guarantees that I will be able
> to make it work.

I use python unittesting framework.  The tests are in CVS in the
testing/ directory.  (About 40% of rdiff-backup's total code is test
code that isn't included in the package.)

Unfortunately, the tests are setup for developers and not end-users,
and require some python knowledge.  Also they aren't very robust and
probably need tweaking to run on other platforms.

Daniel Hazelbaker added some Mac OS X specific tests when he patched
in resource fork support.  But they are out-of-date I doubt anyone but
he has run them.


--
Ben Escoto

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

attachment0 (196 bytes) Download Attachment