How to cope with missing 'rdiff-backup-data' in destination directory

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

How to cope with missing 'rdiff-backup-data' in destination directory

R. Diez
Hi all:

I have a big set of backup files on a slow USB drive (my Linux laptop
only has USB 2.0 ports). In order to quickly update the backup, I have
been using rsync in "look only at modification time" mode.

Now I wish to switch to rdiff-backup. The destination directory already
contains an exact copy of the source directory, but is missing the
'rdiff-backup-data' directory, so I am getting error "Bad directory
/home/rdiez/blah/blah. It doesn't appear to be an rdiff-backup
destination dir".

I don't want to copy all files again, as it would take hours. Is there a
way to tell rdiff-backup to regenerate the 'rdiff-backup-data' directory?

By the way, this is the little wrapper script that I am using:

https://github.com/rdiez/Tools/blob/master/BackupFiles/update-backup-mirror-by-modification-time.sh

Thanks in advance,
   rdiez

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: How to cope with missing 'rdiff-backup-data' in destination directory

Dominic Raferd-3
The message I see in the same situation is:

Fatal Error: Destination directory
[dir]
exists, but does not look like a rdiff-backup directory.  Running
rdiff-backup like this could mess up what is currently in it.  If you
want to update or overwrite it, run rdiff-backup with the --force
option
.
Fatal Error: Lost connection to the remote system

If you do as it suggests and run rdiff-backup with --force option, it should complete successfully and convert the destination directory in an rdiff-backup repository.

Dominic


On 18 January 2016 at 08:15, R. Diez <[hidden email]> wrote:
Hi all:

I have a big set of backup files on a slow USB drive (my Linux laptop only has USB 2.0 ports). In order to quickly update the backup, I have been using rsync in "look only at modification time" mode.

Now I wish to switch to rdiff-backup. The destination directory already contains an exact copy of the source directory, but is missing the 'rdiff-backup-data' directory, so I am getting error "Bad directory /home/rdiez/blah/blah. It doesn't appear to be an rdiff-backup destination dir".

I don't want to copy all files again, as it would take hours. Is there a way to tell rdiff-backup to regenerate the 'rdiff-backup-data' directory?

By the way, this is the little wrapper script that I am using:

https://github.com/rdiez/Tools/blob/master/BackupFiles/update-backup-mirror-by-modification-time.sh

Thanks in advance,
  rdiez

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: How to cope with missing 'rdiff-backup-data' in destination directory

R. Diez


> The message I see in the same situation is:
> [...]

Ah, now I understand. My script was doing a first sweep with "--force", but also with "--remove-older-than", which of course will fail in this scenario. I wrote the thing some time ago. Thanks for your help.



> Fatal Error: Destination directory [dir]
> exists, but does not look like a rdiff-backup directory.  Running
> rdiff-backup like this could mess up what is currently in it.  If you
> want to update or overwrite it, run rdiff-backup with the --force
> option.
> Fatal Error: Lost connection to the remote system


> If you do as it suggests and run rdiff-backup with
> --force option, it should complete successfully and
> convert the destination directory in an rdiff-backup repository.

By the way, the wording in the rdiff-backup message above is probably well meaning, but it does inflict some fear, especially if you have a huge backup on a slow USB drive. I would suggest mentioning in the documentation that --force is fine in this scenario. It is not untypical to migrate from other backup systems to rdiff-backup, so mentioning that this regenerates 'rdiff-backup-data' would help.


Regards,
  rdiez

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: How to cope with missing 'rdiff-backup-data' in destination directory

R. Diez
Hallo Dominic:

Some time ago I posted this question:

https://lists.nongnu.org/archive/html/rdiff-backup-users/2016-01/msg00006.html

And your answer was:

 > If you do as it suggests and run rdiff-backup with --force option, it
 > should complete successfully and convert the destination directory in
 > an rdiff-backup repository.

Like I said, I found what the problem in my script was:

> Ah, now I understand. My script was doing a first sweep with "--force",
 > but also with "--remove-older-than", which of course will fail in
 > this scenario. I wrote the thing some time ago. Thanks for your help.

However, I have just realised that "--force" is not a "complete"
solution, because it does not actually regenerate the metadata inside
the 'rdiff-backup-data'. I noticed because my script does a verification
step at the end, which then issues the following warnings:

rdiff-backup --verify --no-acls "dest"
Warning: Cannot find SHA1 digest for file file1.txt,
perhaps because this feature was added in v1.1.1
Warning: Cannot find SHA1 digest for file file2.txt,
perhaps because this feature was added in v1.1.1
Every file verified successfully.

As mentioned, I am taking over a previous rsync backup with a huge
number of files, and there are too many warnings like that. Copying all
files again from an external, very slow USB drive would take many hours.

Is there a way to regenerate the SHA1 digests for all files in the
destination directory?

Regards,
   rdiez


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: How to cope with missing 'rdiff-backup-data' in destination directory

Dominic Raferd-3
Hello rdiez,

I have done a simple test of this scenario but when I run rdiff-backup --verify I don't see any warning messages. (I believe the SHA1 digests are held in the rdiff-backup-data/mirror_metadata.* files.)

See some postings here and bug report (with proposed fix) here: similar messages were generated by files with more than one hardlink.  (Note to Sol1: even if this is not the cause of OP's problem here, it seems like a good bug to fix...)

IMO it would be worth the initial hours to do a 'proper' initial rdiff-backup run and start with a clean repository that verifies without any warnings.

Dominic

On 18 February 2016 at 09:05, R. Diez <[hidden email]> wrote:
Hallo Dominic:

Some time ago I posted this question:

https://lists.nongnu.org/archive/html/rdiff-backup-users/2016-01/msg00006.html

And your answer was:

> If you do as it suggests and run rdiff-backup with --force option, it > should complete successfully and convert the destination directory in > an rdiff-backup repository.

Like I said, I found what the problem in my script was:

Ah, now I understand. My script was doing a first sweep with "--force",
> but also with "--remove-older-than", which of course will fail in
> this scenario. I wrote the thing some time ago. Thanks for your help.

However, I have just realised that "--force" is not a "complete" solution, because it does not actually regenerate the metadata inside the 'rdiff-backup-data'. I noticed because my script does a verification step at the end, which then issues the following warnings:

rdiff-backup --verify --no-acls "dest"
Warning: Cannot find SHA1 digest for file file1.txt,
perhaps because this feature was added in v1.1.1
Warning: Cannot find SHA1 digest for file file2.txt,
perhaps because this feature was added in v1.1.1
Every file verified successfully.

As mentioned, I am taking over a previous rsync backup with a huge number of files, and there are too many warnings like that. Copying all files again from an external, very slow USB drive would take many hours.

Is there a way to regenerate the SHA1 digests for all files in the destination directory?

Regards,
  rdiez



_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki