Problems with backup after using rdiff-backup-delete.py

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems with backup after using rdiff-backup-delete.py

David Bristow-2
Hello,
I had to delete all references to a folder in one of my backups, using
rdiff-backup-delete.py, which one of the devs basically wrote for us.
The script completely removes certain files from the backup going back
to the first time it was used.  This is due to the customer requiring
that all copies of certain files are removed.

We used rdiff-backup-delete.py to delete a folder, "ibest", last week.
The folder was removed, but every back up since then has been giving
errors (see below this message).

"ibest" was the name of the top-level folder whose contents, including
all sub-folders, totally about 250 files and about 2.4G storage, had
been deleted from the source. All metadata and references to these
folders should have been deleted by rdiff-bacup-delete.py.


We are running rdiff-backup-1.2.8-7.el6.x86_64 running on a RHEL box.

Thanks,
David

  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
324, in Main
    take_action(rps)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
337, in Backup
    backup_final_init(rpout)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
501, in backup_final_init
    checkdest_if_necessary(rpout)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
920, in checkdest_if_necessary
    dest_rp.conn.regress.Regress(dest_rp)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
line 71, in Regress
    for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
line 197, in iterate_meta_rfs
    for raw_rf, metadata_rorp in collated:
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/rorpiter.py",
line 100, in Collate2Iters
    try: relem2 = riter2.next()
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/eas_acls.py",
line 210, in join_ea_iter
    assert rorp, "Missing rorp for index %s" % (ea.index,)
Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
324, in Main
    take_action(rps)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
337, in Backup
    backup_final_init(rpout)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
501, in backup_final_init
    checkdest_if_necessary(rpout)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
920, in checkdest_if_necessary
    dest_rp.conn.regress.Regress(dest_rp)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
line 71, in Regress
    for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
line 197, in iterate_meta_rfs
    for raw_rf, metadata_rorp in collated:
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/rorpiter.py",
line 100, in Collate2Iters
    try: relem2 = riter2.next()
  File "/usr/lib64/python2.6/site-packages/rdiff_backup/eas_acls.py",
line 210, in join_ea_iter
    assert rorp, "Missing rorp for index %s" % (ea.index,)
AssertionError: Missing rorp for index ('ibest', 'original_data',
'gates_study', '0506', 'basicskillssesa56.dta')

Reply | Threaded
Open this post in threaded view
|

Re: Problems with backup after using rdiff-backup-delete.py

Joe Steele-2
On 11/4/2019 2:22 PM, David Bristow wrote:

> Hello,
> I had to delete all references to a folder in one of my backups, using
> rdiff-backup-delete.py, which one of the devs basically wrote for us.
> The script completely removes certain files from the backup going back
> to the first time it was used.  This is due to the customer requiring
> that all copies of certain files are removed.
>
> We used rdiff-backup-delete.py to delete a folder, "ibest", last week.
> The folder was removed, but every back up since then has been giving
> errors (see below this message).
>

Interesting.  I had not heard of rdiff-backup-delete.py before.  It
apparently was written by Sol1 a couple years ago and was added to the
github repo:

https://github.com/rdiff-backup/rdiff-backup/commit/858fb21806187aac09ba31286eb4d128f4613410

The utility would appear to be incomplete.  It makes no attempt to scrub
the extended_attributes.*.gz files and the access_control_lists.*.gz
files for references to files that are to be deleted.

Your error log implies that rdiff-backup was attempting to perform a
regression (regress.py), and in the process of doing that, it was
attempting to deal with extended attributes (eas_acls.py: join_ea_iter)
when suddenly it could not find a file it needed to process
(ibest/original_data/gates_study/0506/basicskillssesa56.dta).  I'm
guessing that that file name came from the extended_attributes file
which the utility didn't scrub.

Another area where the utility is deficient is its failure to look for
files that may have been stored under rdiff-backup-data/long_filename_data.

The utility needs some work.

--Joe


> "ibest" was the name of the top-level folder whose contents, including
> all sub-folders, totally about 250 files and about 2.4G storage, had
> been deleted from the source. All metadata and references to these
> folders should have been deleted by rdiff-bacup-delete.py.
>
>
> We are running rdiff-backup-1.2.8-7.el6.x86_64 running on a RHEL box.
>
> Thanks,
> David
>
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 304, in error_check_Main
>      try: Main(arglist)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 324, in Main
>      take_action(rps)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 280, in take_action
>      elif action == "backup": Backup(rps[0], rps[1])
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 337, in Backup
>      backup_final_init(rpout)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 501, in backup_final_init
>      checkdest_if_necessary(rpout)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 920, in checkdest_if_necessary
>      dest_rp.conn.regress.Regress(dest_rp)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
> line 71, in Regress
>      for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
> line 197, in iterate_meta_rfs
>      for raw_rf, metadata_rorp in collated:
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/rorpiter.py",
> line 100, in Collate2Iters
>      try: relem2 = riter2.next()
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/eas_acls.py",
> line 210, in join_ea_iter
>      assert rorp, "Missing rorp for index %s" % (ea.index,)
> Traceback (most recent call last):
>    File "/usr/bin/rdiff-backup", line 30, in <module>
>      rdiff_backup.Main.error_check_Main(sys.argv[1:])
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 304, in error_check_Main
>      try: Main(arglist)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 324, in Main
>      take_action(rps)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 280, in take_action
>      elif action == "backup": Backup(rps[0], rps[1])
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 337, in Backup
>      backup_final_init(rpout)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 501, in backup_final_init
>      checkdest_if_necessary(rpout)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 920, in checkdest_if_necessary
>      dest_rp.conn.regress.Regress(dest_rp)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
> line 71, in Regress
>      for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
> line 197, in iterate_meta_rfs
>      for raw_rf, metadata_rorp in collated:
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/rorpiter.py",
> line 100, in Collate2Iters
>      try: relem2 = riter2.next()
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/eas_acls.py",
> line 210, in join_ea_iter
>      assert rorp, "Missing rorp for index %s" % (ea.index,)
> AssertionError: Missing rorp for index ('ibest', 'original_data',
> 'gates_study', '0506', 'basicskillssesa56.dta')
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Problems with backup after using rdiff-backup-delete.py

ewl+rdiffbackup
Hi,

whatever Joe said (I didn't check in details but sounds reasonable), plus it must be clear that the scripts under misc are potentially dangerous, rather untested, and definitely not regularly tested, hence at this point not endorsed by the development team.

We apologize for the mess and will add a note to this regard before we'll look into improving the situation, as some of the utilities can possibly be of use.

In the meantime, @David Joe has given hints what needs to be removed to fix your repo (after having created a... backup of it), else let us know if you need more help.

KR, Eric

On November 4, 2019 9:35:41 PM UTC, Joe Steele <[hidden email]> wrote:

>On 11/4/2019 2:22 PM, David Bristow wrote:
>> Hello,
>> I had to delete all references to a folder in one of my backups,
>using
>> rdiff-backup-delete.py, which one of the devs basically wrote for us.
>> The script completely removes certain files from the backup going
>back
>> to the first time it was used.  This is due to the customer requiring
>> that all copies of certain files are removed.
>>
>> We used rdiff-backup-delete.py to delete a folder, "ibest", last
>week.
>> The folder was removed, but every back up since then has been giving
>> errors (see below this message).
>>
>
>Interesting.  I had not heard of rdiff-backup-delete.py before.  It
>apparently was written by Sol1 a couple years ago and was added to the
>github repo:
>
>https://github.com/rdiff-backup/rdiff-backup/commit/858fb21806187aac09ba31286eb4d128f4613410
>
>The utility would appear to be incomplete.  It makes no attempt to
>scrub
>the extended_attributes.*.gz files and the access_control_lists.*.gz
>files for references to files that are to be deleted.
>
>Your error log implies that rdiff-backup was attempting to perform a
>regression (regress.py), and in the process of doing that, it was
>attempting to deal with extended attributes (eas_acls.py: join_ea_iter)
>
>when suddenly it could not find a file it needed to process
>(ibest/original_data/gates_study/0506/basicskillssesa56.dta).  I'm
>guessing that that file name came from the extended_attributes file
>which the utility didn't scrub.
>
>Another area where the utility is deficient is its failure to look for
>files that may have been stored under
>rdiff-backup-data/long_filename_data.
>
>The utility needs some work.
>
>--Joe
>
>
>> "ibest" was the name of the top-level folder whose contents,
>including
>> all sub-folders, totally about 250 files and about 2.4G storage, had
>> been deleted from the source. All metadata and references to these
>> folders should have been deleted by rdiff-bacup-delete.py.
>>
>>
>> We are running rdiff-backup-1.2.8-7.el6.x86_64 running on a RHEL box.
>>
>> Thanks,
>> David
>>
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py",
>line
>> 304, in error_check_Main
>>      try: Main(arglist)
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py",
>line
>> 324, in Main
>>      take_action(rps)
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py",
>line
>> 280, in take_action
>>      elif action == "backup": Backup(rps[0], rps[1])
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py",
>line
>> 337, in Backup
>>      backup_final_init(rpout)
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py",
>line
>> 501, in backup_final_init
>>      checkdest_if_necessary(rpout)
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py",
>line
>> 920, in checkdest_if_necessary
>>      dest_rp.conn.regress.Regress(dest_rp)
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
>> line 71, in Regress
>>      for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index,
>rf)
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
>> line 197, in iterate_meta_rfs
>>      for raw_rf, metadata_rorp in collated:
>>    File
>"/usr/lib64/python2.6/site-packages/rdiff_backup/rorpiter.py",
>> line 100, in Collate2Iters
>>      try: relem2 = riter2.next()
>>    File
>"/usr/lib64/python2.6/site-packages/rdiff_backup/eas_acls.py",
>> line 210, in join_ea_iter
>>      assert rorp, "Missing rorp for index %s" % (ea.index,)
>> Traceback (most recent call last):
>>    File "/usr/bin/rdiff-backup", line 30, in <module>
>>      rdiff_backup.Main.error_check_Main(sys.argv[1:])
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py",
>line
>> 304, in error_check_Main
>>      try: Main(arglist)
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py",
>line
>> 324, in Main
>>      take_action(rps)
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py",
>line
>> 280, in take_action
>>      elif action == "backup": Backup(rps[0], rps[1])
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py",
>line
>> 337, in Backup
>>      backup_final_init(rpout)
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py",
>line
>> 501, in backup_final_init
>>      checkdest_if_necessary(rpout)
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py",
>line
>> 920, in checkdest_if_necessary
>>      dest_rp.conn.regress.Regress(dest_rp)
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
>> line 71, in Regress
>>      for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index,
>rf)
>>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
>> line 197, in iterate_meta_rfs
>>      for raw_rf, metadata_rorp in collated:
>>    File
>"/usr/lib64/python2.6/site-packages/rdiff_backup/rorpiter.py",
>> line 100, in Collate2Iters
>>      try: relem2 = riter2.next()
>>    File
>"/usr/lib64/python2.6/site-packages/rdiff_backup/eas_acls.py",
>> line 210, in join_ea_iter
>>      assert rorp, "Missing rorp for index %s" % (ea.index,)
>> AssertionError: Missing rorp for index ('ibest', 'original_data',
>> 'gates_study', '0506', 'basicskillssesa56.dta')
>>
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Problems with backup after using rdiff-backup-delete.py

Andrew Foster
Hi

EricZolf wrote:
> whatever Joe said (I didn't check in details but sounds reasonable),
> plus it must be clear that the scripts under misc are potentially
> dangerous, rather untested, and definitely not regularly tested, hence
> at this point not endorsed by the development team.

Yes, I can confirm that script was written by one of our staff as a
once-off in response to a single request and is not production ready. In
my opinion, moving forward, it should either not be in the repository or
at least have disclaimers all over it so when you run it you have to
specify a --i-really-acknowledge-this-script-might-eat-the-wrong-data
option.

- Andrew
(null)