Reporting update bug for large files in rdiff-backup (>4Gb)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Reporting update bug for large files in rdiff-backup (>4Gb)

gerard van Dijnsen
Hi all,

a while ago I reported seeing librsync error 107 when updating large
files. The answer was to use the command rdiff to see if this was indeed
rdiff-backup related. At the time I was mighty busy, but now I have
taken the time to look into this more thorough and found out that it is
indeed a problem in rdiff-backup. I would like to flag this as a bug.
The following was carried out with rdiff-backup 1.0.3 with Debian. I
know this is not the latest and greatest, but according to the
changelist nothing even remotely resembling this problem has been fixed.
Here goes:

To start I created some files in a temporary directory:

$ mkdir /bigdisk/tmp
$ cd /bigdisk/tmp
$ for i in 512 1024 2048 4096 5120; do echo $i; dd if=/dev/urandom
of=largefile_${i}_MB.dat bs=1024K count=$i; done

This creates files with random data sized 512Mb, 1, 2 , 4 and 5 Gb. I
proceeded by making a backup of the temporary directory:

$ cd ..
$ rdiff-backup tmp/ tmp2

This takes a while and succeeds. Everything is ok up to this point. The
mirror is created, no errors reported, the files are identical in both
directories. Now I changed the big files somewhat by adding some bytes:

$ cd tmp
$ for i in *.dat; do echo bladiebla >> $i; done

Now restart the backup process:

$ cd ..
$ rdiff-backup tmp/ tmp2

This again takes a while and then produces the following output:
python: ERROR: (rs_job_iter) internal error: job made no progress
[orig_in=23, orig_out=65536, final_in=23, final_out=65536]
UpdateError largefile_4096_MB.dat librsync error 107 while in patch cycle
python: ERROR: (rs_job_iter) internal error: job made no progress
[orig_in=17533, orig_out=65536, final_in=17533, final_out=65536]
UpdateError largefile_5120_MB.dat librsync error 107 while in patch cycle

sure enough, the files mentioned in the output are no longer in sync:

$ md5sum tmp/*.dat tmp2/*.dat
cb79763db0567150a3962299d568ba81  tmp/largefile_1024_MB.dat
ea52ed724707b474208ff08baabd2020  tmp/largefile_2048_MB.dat
e2948b4fdbdd140392f166718e19f1fd  tmp/largefile_4096_MB.dat --> *
73aec8d4c7d0a57c3ecfe5e6be0755ae  tmp/largefile_5120_MB.dat --> *
b86db4919db290010b8610b3a234035a  tmp/largefile_512_MB.dat
cb79763db0567150a3962299d568ba81  tmp2/largefile_1024_MB.dat
ea52ed724707b474208ff08baabd2020  tmp2/largefile_2048_MB.dat
38b463c451c667c91430e82cfb611ed3  tmp2/largefile_4096_MB.dat -->*
4f864e56beac1ce384fe66608ee98e85  tmp2/largefile_5120_MB.dat --> *
b86db4919db290010b8610b3a234035a  tmp2/largefile_512_MB.dat

I have seen this behaviour on my servers on several occasion, and it
seems consistent. Also, this has nothing to do with ssh, because we are
making a local backup. However, the behaviour over ssh is exactly the
same. Now for good measure I used rdiff to see if this would generate a
similar error:

$ rdiff signature tmp2/largefile_4096_MB.dat signature.dat
$ rdiff delta signature.dat tmp/largefile_4096_MB.dat delta.dat
$ rdiff patch tmp2/largefile_4096_MB.dat delta.dat largefile_4096_MB.dat
rdiff: ERROR: (rs_job_iter) internal error: job made no progress
[orig_in=23, orig_out=16000, final_in=23, final_out=16000]
rdiff: ERROR: library internal error

It is notable that the error is given immediately after the 3rd command.
The output file is 0 bytes long. It may be worth to note that when I
tried this for files of the same size, but consisting of all zeros, no
problem occured. Perhaps the files are compressed during the rdiff
process and this has influence by making the datastream smaller?

Hopefully you can patch this. I dont have a clue where to start!

Sincerely
Gerard van Dijnsen








_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Reporting update bug for large files in rdiff-backup (>4Gb)

Vadim Kouzmine
Just in case, repeated it on Gentoo/rdiff-backup-1.0.1/python-2.4.2.

The same here:

python: ERROR: (rs_job_iter) internal error: job made no progress
[orig_in=13, orig_out=65536, final_in=13, final_out=65536]
UpdateError largefile_4096_MB.dat librsync error 107 while in patch
cycle
python: ERROR: (rs_job_iter) internal error: job made no progress
[orig_in=69754, orig_out=65536, final_in=69754, final_out=65536]
UpdateError largefile_5120_MB.dat librsync error 107 while in patch
cycle



On Tue, 2006-02-07 at 15:27 +0100, Gerard van Dijnsen wrote:

> Hi all,
>
> a while ago I reported seeing librsync error 107 when updating large
> files. The answer was to use the command rdiff to see if this was indeed
> rdiff-backup related. At the time I was mighty busy, but now I have
> taken the time to look into this more thorough and found out that it is
> indeed a problem in rdiff-backup. I would like to flag this as a bug.
> The following was carried out with rdiff-backup 1.0.3 with Debian. I
> know this is not the latest and greatest, but according to the
> changelist nothing even remotely resembling this problem has been fixed.
> Here goes:
>
> To start I created some files in a temporary directory:
>
> $ mkdir /bigdisk/tmp
> $ cd /bigdisk/tmp
> $ for i in 512 1024 2048 4096 5120; do echo $i; dd if=/dev/urandom
> of=largefile_${i}_MB.dat bs=1024K count=$i; done
>
> This creates files with random data sized 512Mb, 1, 2 , 4 and 5 Gb. I
> proceeded by making a backup of the temporary directory:
>
> $ cd ..
> $ rdiff-backup tmp/ tmp2
>
> This takes a while and succeeds. Everything is ok up to this point. The
> mirror is created, no errors reported, the files are identical in both
> directories. Now I changed the big files somewhat by adding some bytes:
>
> $ cd tmp
> $ for i in *.dat; do echo bladiebla >> $i; done
>
> Now restart the backup process:
>
> $ cd ..
> $ rdiff-backup tmp/ tmp2
>
> This again takes a while and then produces the following output:
> python: ERROR: (rs_job_iter) internal error: job made no progress
> [orig_in=23, orig_out=65536, final_in=23, final_out=65536]
> UpdateError largefile_4096_MB.dat librsync error 107 while in patch cycle
> python: ERROR: (rs_job_iter) internal error: job made no progress
> [orig_in=17533, orig_out=65536, final_in=17533, final_out=65536]
> UpdateError largefile_5120_MB.dat librsync error 107 while in patch cycle
>
> sure enough, the files mentioned in the output are no longer in sync:
>
> $ md5sum tmp/*.dat tmp2/*.dat
> cb79763db0567150a3962299d568ba81  tmp/largefile_1024_MB.dat
> ea52ed724707b474208ff08baabd2020  tmp/largefile_2048_MB.dat
> e2948b4fdbdd140392f166718e19f1fd  tmp/largefile_4096_MB.dat --> *
> 73aec8d4c7d0a57c3ecfe5e6be0755ae  tmp/largefile_5120_MB.dat --> *
> b86db4919db290010b8610b3a234035a  tmp/largefile_512_MB.dat
> cb79763db0567150a3962299d568ba81  tmp2/largefile_1024_MB.dat
> ea52ed724707b474208ff08baabd2020  tmp2/largefile_2048_MB.dat
> 38b463c451c667c91430e82cfb611ed3  tmp2/largefile_4096_MB.dat -->*
> 4f864e56beac1ce384fe66608ee98e85  tmp2/largefile_5120_MB.dat --> *
> b86db4919db290010b8610b3a234035a  tmp2/largefile_512_MB.dat
>
> I have seen this behaviour on my servers on several occasion, and it
> seems consistent. Also, this has nothing to do with ssh, because we are
> making a local backup. However, the behaviour over ssh is exactly the
> same. Now for good measure I used rdiff to see if this would generate a
> similar error:
>
> $ rdiff signature tmp2/largefile_4096_MB.dat signature.dat
> $ rdiff delta signature.dat tmp/largefile_4096_MB.dat delta.dat
> $ rdiff patch tmp2/largefile_4096_MB.dat delta.dat largefile_4096_MB.dat
> rdiff: ERROR: (rs_job_iter) internal error: job made no progress
> [orig_in=23, orig_out=16000, final_in=23, final_out=16000]
> rdiff: ERROR: library internal error
>
> It is notable that the error is given immediately after the 3rd command.
> The output file is 0 bytes long. It may be worth to note that when I
> tried this for files of the same size, but consisting of all zeros, no
> problem occured. Perhaps the files are compressed during the rdiff
> process and this has influence by making the datastream smaller?
>
> Hopefully you can patch this. I dont have a clue where to start!
>
> Sincerely
> Gerard van Dijnsen
>
>
>
>
>
>
>
>
> _______________________________________________
> rdiff-backup-users mailing list at [hidden email]
> http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
--
Vadim Kouzmine <[hidden email]>



_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Reporting update bug for large files in rdiff-backup (>4Gb)

Dave Kempe
In reply to this post by gerard van Dijnsen
Gerard van Dijnsen wrote:

> This again takes a while and then produces the following output:
> python: ERROR: (rs_job_iter) internal error: job made no progress
> [orig_in=23, orig_out=65536, final_in=23, final_out=65536]
> UpdateError largefile_4096_MB.dat librsync error 107 while in patch cycle
> python: ERROR: (rs_job_iter) internal error: job made no progress
> [orig_in=17533, orig_out=65536, final_in=17533, final_out=65536]
> UpdateError largefile_5120_MB.dat librsync error 107 while in patch cycle
>
>
these errors look fairly similar to me.

> I have seen this behaviour on my servers on several occasion, and it
> seems consistent. Also, this has nothing to do with ssh, because we
> are making a local backup. However, the behaviour over ssh is exactly
> the same. Now for good measure I used rdiff to see if this would
> generate a similar error:
>
> $ rdiff signature tmp2/largefile_4096_MB.dat signature.dat
> $ rdiff delta signature.dat tmp/largefile_4096_MB.dat delta.dat
> $ rdiff patch tmp2/largefile_4096_MB.dat delta.dat largefile_4096_MB.dat
> rdiff: ERROR: (rs_job_iter) internal error: job made no progress
> [orig_in=23, orig_out=16000, final_in=23, final_out=16000]
> rdiff: ERROR: library internal error

i thought if rdiff generated the same error, then its definitely a
librsync bug. rdiff-backup doesn't use rdiff for anything, despite the name.

Although chatting to Martin Pool (librsync author) a while back, he
reckons it needs a rewrite and it would take ages to do it properly.

I tried the same test on a CVS version of rdiff-backup from today, with
librsync 0.9.7 (ubuntu breezy)
I did not run out of temp space at all.
I believe its a librsync bug, and am pursuing the author to see where
its up to....

Same problem:
 rdiff-backup -v5 temp/ temp-backup
-----------------------------------------------------------------
Detected abilities for source (read only) file system:
  Access control lists                         On
  Extended attributes                          On
  Case sensitivity                             On
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
  Ownership changing                           On
  Hard linking                                 On
  fsync() directories                          On
  Directory inc permissions                    On
  High-bit permissions                         On
  Extended filenames                           On
  Access control lists                         On
  Extended attributes                          On
  Case sensitivity                             On
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Starting increment operation temp to temp-backup
Processing changed file .
Incrementing mirror file temp-backup
Processing changed file largefile_1024_MB.dat
Incrementing mirror file temp-backup/largefile_1024_MB.dat
Processing changed file largefile_2048_MB.dat
Incrementing mirror file temp-backup/largefile_2048_MB.dat
Processing changed file largefile_4096_MB.dat
python: ERROR: (rs_job_iter) internal error: job made no progress
[orig_in=13, orig_out=65536, final_in=13, final_out=65536]
Exception 'librsync error 107 while in patch cycle' raised of class
'rdiff_backup.librsync.librsyncError':
  File "/usr/lib/python2.4/site-packages/rdiff_backup/robust.py", line
32, in check_common_error
    try: return function(*args)
  File "/usr/lib/python2.4/site-packages/rdiff_backup/Rdiff.py", line
98, in patch_local
    if outrp: return outrp.write_from_fileobj(patchfile)
  File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line
1045, in write_from_fileobj
    copyfileobj(fp, outfp)
  File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line
58, in copyfileobj
    inbuf = inputfp.read(blocksize)
  File "/usr/lib/python2.4/site-packages/rdiff_backup/librsync.py", line
77, in read
    self._add_to_outbuf_once()
  File "/usr/lib/python2.4/site-packages/rdiff_backup/librsync.py", line
88, in _add_to_outbuf_once
    except _librsync.librsyncError, e: raise librsyncError(str(e))

UpdateError largefile_4096_MB.dat librsync error 107 while in patch cycle
Processing changed file largefile_5120_MB.dat
python: ERROR: (rs_job_iter) internal error: job made no progress
[orig_in=69754, orig_out=65536, final_in=69754, final_out=65536]
Exception 'librsync error 107 while in patch cycle' raised of class
'rdiff_backup.librsync.librsyncError':
  File "/usr/lib/python2.4/site-packages/rdiff_backup/robust.py", line
32, in check_common_error
    try: return function(*args)
  File "/usr/lib/python2.4/site-packages/rdiff_backup/Rdiff.py", line
98, in patch_local
    if outrp: return outrp.write_from_fileobj(patchfile)
  File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line
1045, in write_from_fileobj
    copyfileobj(fp, outfp)
  File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line
58, in copyfileobj
    inbuf = inputfp.read(blocksize)
  File "/usr/lib/python2.4/site-packages/rdiff_backup/librsync.py", line
77, in read
    self._add_to_outbuf_once()
  File "/usr/lib/python2.4/site-packages/rdiff_backup/librsync.py", line
88, in _add_to_outbuf_once
    except _librsync.librsyncError, e: raise librsyncError(str(e))

UpdateError largefile_5120_MB.dat librsync error 107 while in patch cycle
Processing changed file largefile_512_MB.dat
Incrementing mirror file temp-backup/largefile_512_MB.dat
                                                                             






_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Reporting update bug for large files in rdiff-backup (>4Gb)

gerard van Dijnsen
Hi,

Well the large file problem in librsync was solved by the maintainers
themselves as can be seen in
https://sourceforge.net/tracker/?func=detail&atid=479439&aid=1110812&group_id=56125.


I have tested it and it seems to work well for files of at least 5 Gb.
As far as I know, no new release of the library is produced, so you'll
have to patch manually if you have this problem, but at least it will
work. One interesting point may be that only the server side needs to be
patched as far as I can make out. I tested it and both backup and
restore actions work flawlessly with only the server side patched.

Thanks for your input on this!
Gerard

--
__________________________________________________________________
OpenBizz BV - Open Source Solutions

ir. Gerard van Dijnsen
Telefoon: +31648428223
e-mail: [hidden email]
Veldm. Montgomerylaan 537
5612 BM  EINDHOVEN
__________________________________________________________________
Open your mind, open your future! http://www.openbizz.nl



_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki