Files Missing In Backup Resulting in Update Error

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Files Missing In Backup Resulting in Update Error

M. Verkerk
Dear all,

Thanks for this script! Really fills the gap between rsync and more advanced backup tools! I ran into some trouble I hope to find some help here!

I’m running rdiff-backup with the following cronjob script:

—Cut--
#!/bin/bash

if [ -e /tmp/backup-XXX.lock ]
then
        echo "Previous backup still in progress! Bailing out!";
        exit 0;
fi

echo "Setting lock file!";
touch /tmp/backup-XXX.lock

echo "Backing up etc!";
nice -n -3 rdiff-backup --print-statistics /etc YYY.domain.nl::/data/backup/XXX/rdiff-disk-backup/etc;
echo "Done!";

echo "Backing up home!";
nice -n -3 rdiff-backup --print-statistics /home YYY.domain.nl::/data/backup/XXX/rdiff-disk-backup/home;
echo "Done!”;

echo "Removing lock file!";
rm /tmp/backup-XXX.lock;

echo "Done!";
-Cut-

During the last backup I got several UpdateErrors with seemingly random backup files (just including three):

—Cut-
UpdateError 529841/top100000.txt Updated mirror temp file /data/backup/XXX/rdiff-disk-backup/home/529841/rdiff-backup.tmp.265 does not match source
UpdateError marijn/src/perl-5.16.1/x2p/s2p Updated mirror temp file /data/backup/XXX/rdiff-disk-backup/home/marijn/src/perl-5.16.1/x2p/rdiff-backup.tmp.4011 does not match source
UpdateError 529841/FVC/Meta_all/ecrhs_ALL1.txt Updated mirror temp file /data/backup/XXX/rdiff-disk-backup/home/529841/FVC/Meta_all/rdiff-backup.tmp.231 does not match source
-Pase-

The files mentioned here ARE existing on the backup source drive with no peculiar ownership or permissions. Somehow they are NOT on the backup folder - they should have been, some of the files are there and not changed since we start using rdiff-backup. I think the error message is indirectly related to the problem. Could this be a network failure - as you can see in the script we are using ssh?

More important: what can I do to trigger the rdiff-backup to reconsider en copy these omitted files?

Thanks very much in advanced!

Marijn





_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Files Missing In Backup Resulting in Update Error

Dominic Raferd-3
Marijn, a similar situation was discussed here quite recently, see http://lists.gnu.org/archive/html/rdiff-backup-users/2014-03/msg00012.html. The cause in this case, and probably in yours, is that the source data is changing while the backup is proceeding. I note that in your case some of the source files that are missing in backup have not changed, but maybe their metadata changed? What filesystem are they on?

If you really need to backup the missing files, using a static copy of the source (e.g. a snapshot) should solve the problem. If the missing files are unimportant, Chris Wilson suggested a way to stop these messages appearing.

Dominic

On 06/05/2014 15:01, M. Verkerk wrote:
Dear all,

Thanks for this script! Really fills the gap between rsync and more advanced backup tools! I ran into some trouble I hope to find some help here!

I’m running rdiff-backup with the following cronjob script:

—Cut--
#!/bin/bash

if [ -e /tmp/backup-XXX.lock ]
then
        echo "Previous backup still in progress! Bailing out!";
        exit 0;
fi

echo "Setting lock file!";
touch /tmp/backup-XXX.lock

echo "Backing up etc!";
nice -n -3 rdiff-backup --print-statistics /etc YYY.domain.nl::/data/backup/XXX/rdiff-disk-backup/etc;
echo "Done!";

echo "Backing up home!";
nice -n -3 rdiff-backup --print-statistics /home YYY.domain.nl::/data/backup/XXX/rdiff-disk-backup/home;
echo "Done!”;

echo "Removing lock file!";
rm /tmp/backup-XXX.lock;

echo "Done!";
-Cut-

During the last backup I got several UpdateErrors with seemingly random backup files (just including three):

—Cut-
UpdateError 529841/top100000.txt Updated mirror temp file /data/backup/XXX/rdiff-disk-backup/home/529841/rdiff-backup.tmp.265 does not match source
UpdateError marijn/src/perl-5.16.1/x2p/s2p Updated mirror temp file /data/backup/XXX/rdiff-disk-backup/home/marijn/src/perl-5.16.1/x2p/rdiff-backup.tmp.4011 does not match source
UpdateError 529841/FVC/Meta_all/ecrhs_ALL1.txt Updated mirror temp file /data/backup/XXX/rdiff-disk-backup/home/529841/FVC/Meta_all/rdiff-backup.tmp.231 does not match source
-Pase-

The files mentioned here ARE existing on the backup source drive with no peculiar ownership or permissions. Somehow they are NOT on the backup folder - they should have been, some of the files are there and not changed since we start using rdiff-backup. I think the error message is indirectly related to the problem. Could this be a network failure - as you can see in the script we are using ssh?

More important: what can I do to trigger the rdiff-backup to reconsider en copy these omitted files?

Thanks very much in advanced!

Marijn






_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


--
TimeDicer: Free File Recovery from Whenever

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Files Missing In Backup Resulting in Update Error

M. Verkerk
Dear Dominic,

Thanks for your response!

Some of these files are from inactive users and I’m quite sure their files weren’t changed during the rdiff backup!

Did you read the email from Mike Fleetwood? These are similar error messages, allegedley caused by retrieving wrong hid and gid on NFS file systems. I’m using a remote filesystem as well, SSHFS, over here!

I have applied this patch and running a backup at this very moment. Unfortunately it’s taking quite long because I had to interrupt a previous backup and rdiff is busy with ‘regressing’.

Regards,
Marijn

On 14 May 2014, at 14:07, Dominic Raferd <[hidden email]> wrote:

Marijn, a similar situation was discussed here quite recently, see http://lists.gnu.org/archive/html/rdiff-backup-users/2014-03/msg00012.html. The cause in this case, and probably in yours, is that the source data is changing while the backup is proceeding. I note that in your case some of the source files that are missing in backup have not changed, but maybe their metadata changed? What filesystem are they on?

If you really need to backup the missing files, using a static copy of the source (e.g. a snapshot) should solve the problem. If the missing files are unimportant, Chris Wilson suggested a way to stop these messages appearing.

Dominic


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Files Missing In Backup Resulting in Update Error

Dominic Raferd-3
Hello Marijn

Yes from Mike's email it does seem there is a problem with rdiff-backup if the destination filesystem is accessed via NFS.

I hadn't realised you use sshfs, the script you gave appeared to use rdiff-backup's internal connection method. Yet I use sshfs to mount a remote source directory (for a source machine which cannot run rdiff-backup) and I have never had any problems with it, my remote filesystem is ext4 (previously reiserfs), the destination filesystem is ext4. I have always used rdiff-backup switches --no-eas and --no-acls which may or may not be significant, and I backup from an LVM snapshot of the source.

What are the underlying filesystems for the sources (/etc and /home) and for the destination (YYY.domain.nl::/data/backup/XXX/rdiff-disk-backup)?

Dominic
--
TimeDicer: Free File Recovery from Whenever

On 14/05/2014 13:43, M. Verkerk wrote:
Dear Dominic,

Thanks for your response!

Some of these files are from inactive users and I’m quite sure their files weren’t changed during the rdiff backup!

Did you read the email from Mike Fleetwood? These are similar error messages, allegedley caused by retrieving wrong hid and gid on NFS file systems. I’m using a remote filesystem as well, SSHFS, over here!

I have applied this patch and running a backup at this very moment. Unfortunately it’s taking quite long because I had to interrupt a previous backup and rdiff is busy with ‘regressing’.

Regards,
Marijn

On 14 May 2014, at 14:07, Dominic Raferd <[hidden email]> wrote:

Marijn, a similar situation was discussed here quite recently, see http://lists.gnu.org/archive/html/rdiff-backup-users/2014-03/msg00012.html. The cause in this case, and probably in yours, is that the source data is changing while the backup is proceeding. I note that in your case some of the source files that are missing in backup have not changed, but maybe their metadata changed? What filesystem are they on?

If you really need to backup the missing files, using a static copy of the source (e.g. a snapshot) should solve the problem. If the missing files are unimportant, Chris Wilson suggested a way to stop these messages appearing.

Dominic



_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Files Missing In Backup Resulting in Update Error

M. Verkerk
Dominic,

I’m mistaking, you’re right.. A previous setup was using sshfs and it’s actually the internal system which is being used now.

The destination folder has a gfs2 file system.

The source folders have ext4.

attributes / access is not really essential, but preferred! 

Thanks,
Marijn


On 15 May 2014, at 10:45, Dominic Raferd <[hidden email]> wrote:

Hello Marijn

Yes from Mike's email it does seem there is a problem with rdiff-backup if the destination filesystem is accessed via NFS.

I hadn't realised you use sshfs, the script you gave appeared to use rdiff-backup's internal connection method. Yet I use sshfs to mount a remote source directory (for a source machine which cannot run rdiff-backup) and I have never had any problems with it, my remote filesystem is ext4 (previously reiserfs), the destination filesystem is ext4. I have always used rdiff-backup switches --no-eas and --no-acls which may or may not be significant, and I backup from an LVM snapshot of the source.

What are the underlying filesystems for the sources (/etc and /home) and for the destination (YYY.domain.nl::/data/backup/XXX/rdiff-disk-backup)?

Dominic
--
TimeDicer: Free File Recovery from Whenever

On 14/05/2014 13:43, M. Verkerk wrote:
Dear Dominic,

Thanks for your response!

Some of these files are from inactive users and I’m quite sure their files weren’t changed during the rdiff backup!

Did you read the email from Mike Fleetwood? These are similar error messages, allegedley caused by retrieving wrong hid and gid on NFS file systems. I’m using a remote filesystem as well, SSHFS, over here!

I have applied this patch and running a backup at this very moment. Unfortunately it’s taking quite long because I had to interrupt a previous backup and rdiff is busy with ‘regressing’.

Regards,
Marijn

On 14 May 2014, at 14:07, Dominic Raferd <[hidden email]> wrote:

Marijn, a similar situation was discussed here quite recently, see http://lists.gnu.org/archive/html/rdiff-backup-users/2014-03/msg00012.html. The cause in this case, and probably in yours, is that the source data is changing while the backup is proceeding. I note that in your case some of the source files that are missing in backup have not changed, but maybe their metadata changed? What filesystem are they on?

If you really need to backup the missing files, using a static copy of the source (e.g. a snapshot) should solve the problem. If the missing files are unimportant, Chris Wilson suggested a way to stop these messages appearing.

Dominic


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Files Missing In Backup Resulting in Update Error

Dominic Raferd-3
Marijn,

My guess is that it is something to do with gfs2, there doesn't seem to be much experience of this with rdiff-backup.

Please let us know if Mike's patch works for you, as this would suggest it might have wider applicability.

If not, is it possible for you to run rdiff-backup the other way around i.e. run rdiff-backup on the destination (with local gfs2) pulling from remote ext4 source? And/or try it with --no-eas and/or -no-acls.

Dominic

On 15/05/2014 10:06, M. Verkerk wrote:
Dominic,

I’m mistaking, you’re right.. A previous setup was using sshfs and it’s actually the internal system which is being used now.

The destination folder has a gfs2 file system.

The source folders have ext4.

attributes / access is not really essential, but preferred! 

Thanks,
Marijn


On 15 May 2014, at 10:45, Dominic Raferd <[hidden email]> wrote:

Hello Marijn

Yes from Mike's email it does seem there is a problem with rdiff-backup if the destination filesystem is accessed via NFS.

I hadn't realised you use sshfs, the script you gave appeared to use rdiff-backup's internal connection method. Yet I use sshfs to mount a remote source directory (for a source machine which cannot run rdiff-backup) and I have never had any problems with it, my remote filesystem is ext4 (previously reiserfs), the destination filesystem is ext4. I have always used rdiff-backup switches --no-eas and --no-acls which may or may not be significant, and I backup from an LVM snapshot of the source.

What are the underlying filesystems for the sources (/etc and /home) and for the destination (YYY.domain.nl::/data/backup/XXX/rdiff-disk-backup)?

Dominic
--
TimeDicer: Free File Recovery from Whenever

On 14/05/2014 13:43, M. Verkerk wrote:
Dear Dominic,

Thanks for your response!

Some of these files are from inactive users and I’m quite sure their files weren’t changed during the rdiff backup!

Did you read the email from Mike Fleetwood? These are similar error messages, allegedley caused by retrieving wrong hid and gid on NFS file systems. I’m using a remote filesystem as well, SSHFS, over here!

I have applied this patch and running a backup at this very moment. Unfortunately it’s taking quite long because I had to interrupt a previous backup and rdiff is busy with ‘regressing’.

Regards,
Marijn

On 14 May 2014, at 14:07, Dominic Raferd <[hidden email]> wrote:

Marijn, a similar situation was discussed here quite recently, see http://lists.gnu.org/archive/html/rdiff-backup-users/2014-03/msg00012.html. The cause in this case, and probably in yours, is that the source data is changing while the backup is proceeding. I note that in your case some of the source files that are missing in backup have not changed, but maybe their metadata changed? What filesystem are they on?

If you really need to backup the missing files, using a static copy of the source (e.g. a snapshot) should solve the problem. If the missing files are unimportant, Chris Wilson suggested a way to stop these messages appearing.

Dominic


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


--
TimeDicer: Free File Recovery from Whenever

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Files Missing In Backup Resulting in Update Error

Mike Fleetwood
Hi Marijn,

You may find the attached patch useful.  I used it to debug my issue
and determine why rdiff-backup was reporting "Update temp file ...
does not match" errors backing up to my NFS share.  For me it gave
results like this:

Sun May  4 14:41:54 2014  Processing changed file bin/red
Sun May  4 14:41:54 2014  Hard linking
/mnt/autofs/backup/bin/rdiff-backup.tmp.1 to /mnt/autofs/backup/bin/ed
Sun May  4 14:41:54 2014  Loose compare ('bin', 'red') to ('bin',
'rdiff-backup.tmp.1'): user group mismatch:
user_group.map_rpath(first)=(0, 0) != second.getuidgid()=(99, 99)
Sun May  4 14:41:54 2014  UpdateError bin/red Updated mirror temp file
/mnt/autofs/backup/bin/rdiff-backup.tmp.1 does not match source
Sun May  4 14:41:54 2014  Deleting /mnt/autofs/backup/bin/rdiff-backup.tmp.1

Use rdiff-backup -v 7 or greater.  The extra line is the "Loose
compare ..." one.

Thanks,
Mike

On 15 May 2014 10:27, Dominic Raferd <[hidden email]> wrote:

> Marijn,
>
> My guess is that it is something to do with gfs2, there doesn't seem to be
> much experience of this with rdiff-backup.
>
> Please let us know if Mike's patch works for you, as this would suggest it
> might have wider applicability.
>
> If not, is it possible for you to run rdiff-backup the other way around i.e.
> run rdiff-backup on the destination (with local gfs2) pulling from remote
> ext4 source? And/or try it with --no-eas and/or -no-acls.
>
> Dominic
>
>
> On 15/05/2014 10:06, M. Verkerk wrote:
>
> Dominic,
>
> I’m mistaking, you’re right.. A previous setup was using sshfs and it’s
> actually the internal system which is being used now.
>
> The destination folder has a gfs2 file system.
>
> The source folders have ext4.
>
> attributes / access is not really essential, but preferred!
>
> Thanks,
> Marijn
>
>
> On 15 May 2014, at 10:45, Dominic Raferd <[hidden email]> wrote:
>
> Hello Marijn
>
> Yes from Mike's email it does seem there is a problem with rdiff-backup if
> the destination filesystem is accessed via NFS.
>
> I hadn't realised you use sshfs, the script you gave appeared to use
> rdiff-backup's internal connection method. Yet I use sshfs to mount a remote
> source directory (for a source machine which cannot run rdiff-backup) and I
> have never had any problems with it, my remote filesystem is ext4
> (previously reiserfs), the destination filesystem is ext4. I have always
> used rdiff-backup switches --no-eas and --no-acls which may or may not be
> significant, and I backup from an LVM snapshot of the source.
>
> What are the underlying filesystems for the sources (/etc and /home) and for
> the destination (YYY.domain.nl::/data/backup/XXX/rdiff-disk-backup)?
>
> Dominic
> --
> TimeDicer: Free File Recovery from Whenever
>
> On 14/05/2014 13:43, M. Verkerk wrote:
>
> Dear Dominic,
>
> Thanks for your response!
>
> Some of these files are from inactive users and I’m quite sure their files
> weren’t changed during the rdiff backup!
>
> Did you read the email from Mike Fleetwood? These are similar error
> messages, allegedley caused by retrieving wrong hid and gid on NFS file
> systems. I’m using a remote filesystem as well, SSHFS, over here!
>
> I have applied this patch and running a backup at this very moment.
> Unfortunately it’s taking quite long because I had to interrupt a previous
> backup and rdiff is busy with ‘regressing’.
>
> Regards,
> Marijn
>
> On 14 May 2014, at 14:07, Dominic Raferd <[hidden email]> wrote:
>
> Marijn, a similar situation was discussed here quite recently, see
> http://lists.gnu.org/archive/html/rdiff-backup-users/2014-03/msg00012.html.
> The cause in this case, and probably in yours, is that the source data is
> changing while the backup is proceeding. I note that in your case some of
> the source files that are missing in backup have not changed, but maybe
> their metadata changed? What filesystem are they on?
>
> If you really need to backup the missing files, using a static copy of the
> source (e.g. a snapshot) should solve the problem. If the missing files are
> unimportant, Chris Wilson suggested a way to stop these messages appearing.
>
> Dominic
>
>
> _______________________________________________
> rdiff-backup-users mailing list at [hidden email]
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL:
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>
>
>
> --
> TimeDicer: Free File Recovery from Whenever
>
> _______________________________________________
> rdiff-backup-users mailing list at [hidden email]
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL:
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

rdiff-backup-debug-log-causes-of-update-error.patch (2K) Download Attachment