Why do I see a huge variation in the time rdiff-backup takes?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Why do I see a huge variation in the time rdiff-backup takes?

Segundo Bob
I run rdiff-backup on my desktop computer to backup data from my
internal hard drive to a removable internal hard drive.  I use a script
to do the backup every Sunday.  Unpredictably, infrequently rdiff-backup
runs much, much longer than usual.  Why?

I'm asking now because yesterday rdiff-backup ran for 3 hours and 6
minutes when I expected it to run between 5 minutes and 20 minutes.

Is there a problem switching from 32-bit to 64-bit?

Thanks for any help you can give.

Here are the backup statistics and my notes:

        2015-03-15 - Created a new archive file.
                Xubuntu32 12.04
                rdiff-backup 1.2.8
                Removable drive connected to a
                motherboard SATA port
                19 minutes 17.45 seconds
        2015-03-22 - Xubuntu32 12.04
                rdiff-backup 1.2.8
                Removable drive connected to a
                motherboard SATA port
                3 minutes 43.82 seconds
        2015-03-29 - Xubuntu64 14.04
                rdiff-backup 1.2.8
                Removable drive connected to
                a PCIe SATA card
                3 hours 5 minutes 36.81 seconds
        2015-03-30 - Xubuntu64 14.04
                rdiff-backup 1.2.8
                Removable drive connected to
                a PCIe SATA card
                4 minutes 42.97 seconds


session_statistics.2015-03-15T17:44:28-07:00.data

StartTime 1426466668.00 (Sun Mar 15 17:44:28 2015)
EndTime 1426467825.45 (Sun Mar 15 18:03:45 2015)
ElapsedTime 1157.45 (19 minutes 17.45 seconds)
SourceFiles 190944
SourceFileSize 37069994870 (34.5 GB)
MirrorFiles 1
MirrorFileSize 0 (0 bytes)
NewFiles 190943
NewFileSize 37069994870 (34.5 GB)
DeletedFiles 0
DeletedFileSize 0 (0 bytes)
ChangedFiles 1
ChangedSourceSize 0 (0 bytes)
ChangedMirrorSize 0 (0 bytes)
IncrementFiles 0
IncrementFileSize 0 (0 bytes)
TotalDestinationSizeChange 37069994870 (34.5 GB)
Errors 1


session_statistics.2015-03-22T11:18:18-07:00.data

StartTime 1427048298.00 (Sun Mar 22 11:18:18 2015)
EndTime 1427048521.82 (Sun Mar 22 11:22:01 2015)
ElapsedTime 223.82 (3 minutes 43.82 seconds)
SourceFiles 191334
SourceFileSize 37080046335 (34.5 GB)
MirrorFiles 190944
MirrorFileSize 37069994870 (34.5 GB)
NewFiles 411
NewFileSize 9350513 (8.92 MB)
DeletedFiles 21
DeletedFileSize 8768177 (8.36 MB)
ChangedFiles 465
ChangedSourceSize 408118380 (389 MB)
ChangedMirrorSize 398649251 (380 MB)
IncrementFiles 899
IncrementFileSize 33979051 (32.4 MB)
TotalDestinationSizeChange 44030516 (42.0 MB)
Errors 1

session_statistics.2015-03-29T14:01:04-07:00.data

StartTime 1427662864.00 (Sun Mar 29 14:01:04 2015)
EndTime 1427674000.81 (Sun Mar 29 17:06:40 2015)
ElapsedTime 11136.81 (3 hours 5 minutes 36.81 seconds)
SourceFiles 190331
SourceFileSize 37115088389 (34.6 GB)
MirrorFiles 191334
MirrorFileSize 37080046335 (34.5 GB)
NewFiles 1010
NewFileSize 68810627 (65.6 MB)
DeletedFiles 2013
DeletedFileSize 26435301 (25.2 MB)
ChangedFiles 187778
ChangedSourceSize 37035986594 (34.5 GB)
ChangedMirrorSize 37043319866 (34.5 GB)
IncrementFiles 190801
IncrementFileSize 42530381 (40.6 MB)
TotalDestinationSizeChange 77572435 (74.0 MB)
Errors 0

session_statistics.2015-03-30T15:32:21-07:00.data

StartTime 1427754741.00 (Mon Mar 30 15:32:21 2015)
EndTime 1427755023.97 (Mon Mar 30 15:37:03 2015)
ElapsedTime 282.97 (4 minutes 42.97 seconds)
SourceFiles 190412
SourceFileSize 37121646218 (34.6 GB)
MirrorFiles 190331
MirrorFileSize 37115088389 (34.6 GB)
NewFiles 89
NewFileSize 4118761 (3.93 MB)
DeletedFiles 8
DeletedFileSize 410210 (401 KB)
ChangedFiles 2982
ChangedSourceSize 578210673 (551 MB)
ChangedMirrorSize 575361395 (549 MB)
IncrementFiles 3079
IncrementFileSize 4893512 (4.67 MB)
TotalDestinationSizeChange 11451341 (10.9 MB)
Errors 0

--
Bob Hossley
[hidden email]

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Why do I see a huge variation in the time rdiff-backup takes?

Adrian Klaver-2
On 03/30/2015 04:41 PM, Segundo Bob wrote:

> I run rdiff-backup on my desktop computer to backup data from my
> internal hard drive to a removable internal hard drive.  I use a script
> to do the backup every Sunday.  Unpredictably, infrequently rdiff-backup
> runs much, much longer than usual.  Why?
>
> I'm asking now because yesterday rdiff-backup ran for 3 hours and 6
> minutes when I expected it to run between 5 minutes and 20 minutes.
>
> Is there a problem switching from 32-bit to 64-bit?
>
> Thanks for any help you can give.
>
> Here are the backup statistics and my notes:
>
> 2015-03-15 - Created a new archive file.
> Xubuntu32 12.04
> rdiff-backup 1.2.8
> Removable drive connected to a
> motherboard SATA port
> 19 minutes 17.45 seconds
> 2015-03-22 - Xubuntu32 12.04
> rdiff-backup 1.2.8
> Removable drive connected to a
> motherboard SATA port
> 3 minutes 43.82 seconds

So is the same machine above and below?

> 2015-03-29 - Xubuntu64 14.04
> rdiff-backup 1.2.8
> Removable drive connected to
> a PCIe SATA card
> 3 hours 5 minutes 36.81 seconds
> 2015-03-30 - Xubuntu64 14.04
> rdiff-backup 1.2.8
> Removable drive connected to
> a PCIe SATA card
> 4 minutes 42.97 seconds
>
>
> session_statistics.2015-03-15T17:44:28-07:00.data
>
> StartTime 1426466668.00 (Sun Mar 15 17:44:28 2015)
> EndTime 1426467825.45 (Sun Mar 15 18:03:45 2015)
> ElapsedTime 1157.45 (19 minutes 17.45 seconds)
> SourceFiles 190944
> SourceFileSize 37069994870 (34.5 GB)
> MirrorFiles 1
> MirrorFileSize 0 (0 bytes)
> NewFiles 190943
> NewFileSize 37069994870 (34.5 GB)
> DeletedFiles 0
> DeletedFileSize 0 (0 bytes)
> ChangedFiles 1
> ChangedSourceSize 0 (0 bytes)
> ChangedMirrorSize 0 (0 bytes)
> IncrementFiles 0
> IncrementFileSize 0 (0 bytes)
> TotalDestinationSizeChange 37069994870 (34.5 GB)
> Errors 1
>
>
> session_statistics.2015-03-22T11:18:18-07:00.data
>
> StartTime 1427048298.00 (Sun Mar 22 11:18:18 2015)
> EndTime 1427048521.82 (Sun Mar 22 11:22:01 2015)
> ElapsedTime 223.82 (3 minutes 43.82 seconds)
> SourceFiles 191334
> SourceFileSize 37080046335 (34.5 GB)
> MirrorFiles 190944
> MirrorFileSize 37069994870 (34.5 GB)
> NewFiles 411
> NewFileSize 9350513 (8.92 MB)
> DeletedFiles 21
> DeletedFileSize 8768177 (8.36 MB)
> ChangedFiles 465
> ChangedSourceSize 408118380 (389 MB)
> ChangedMirrorSize 398649251 (380 MB)
> IncrementFiles 899
> IncrementFileSize 33979051 (32.4 MB)
> TotalDestinationSizeChange 44030516 (42.0 MB)
> Errors 1
>
> session_statistics.2015-03-29T14:01:04-07:00.data
>
> StartTime 1427662864.00 (Sun Mar 29 14:01:04 2015)
> EndTime 1427674000.81 (Sun Mar 29 17:06:40 2015)
> ElapsedTime 11136.81 (3 hours 5 minutes 36.81 seconds)
> SourceFiles 190331
> SourceFileSize 37115088389 (34.6 GB)
> MirrorFiles 191334
> MirrorFileSize 37080046335 (34.5 GB)
> NewFiles 1010
> NewFileSize 68810627 (65.6 MB)
> DeletedFiles 2013
> DeletedFileSize 26435301 (25.2 MB)
> ChangedFiles 187778

The above would seem to be the cause. What changed the files?

> ChangedSourceSize 37035986594 (34.5 GB)
> ChangedMirrorSize 37043319866 (34.5 GB)
> IncrementFiles 190801
> IncrementFileSize 42530381 (40.6 MB)
> TotalDestinationSizeChange 77572435 (74.0 MB)
> Errors 0
>
> session_statistics.2015-03-30T15:32:21-07:00.data
>
> StartTime 1427754741.00 (Mon Mar 30 15:32:21 2015)
> EndTime 1427755023.97 (Mon Mar 30 15:37:03 2015)
> ElapsedTime 282.97 (4 minutes 42.97 seconds)
> SourceFiles 190412
> SourceFileSize 37121646218 (34.6 GB)
> MirrorFiles 190331
> MirrorFileSize 37115088389 (34.6 GB)
> NewFiles 89
> NewFileSize 4118761 (3.93 MB)
> DeletedFiles 8
> DeletedFileSize 410210 (401 KB)
> ChangedFiles 2982
> ChangedSourceSize 578210673 (551 MB)
> ChangedMirrorSize 575361395 (549 MB)
> IncrementFiles 3079
> IncrementFileSize 4893512 (4.67 MB)
> TotalDestinationSizeChange 11451341 (10.9 MB)
> Errors 0
>


--
Adrian Klaver
[hidden email]

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Why do I see a huge variation in the time rdiff-backup takes?

Robert Nichols-2
In reply to this post by Segundo Bob
On 03/30/2015 06:41 PM, Segundo Bob wrote:
> I run rdiff-backup on my desktop computer to backup data from my
> internal hard drive to a removable internal hard drive.  I use a script
> to do the backup every Sunday.  Unpredictably, infrequently rdiff-backup
> runs much, much longer than usual.  Why?
>
> I'm asking now because yesterday rdiff-backup ran for 3 hours and 6
> minutes when I expected it to run between 5 minutes and 20 minutes.
>
> Is there a problem switching from 32-bit to 64-bit?

Any change in the recorded metadata for a file will cause rdiff-backup
to do a time-consuming diff of the old and new files. For files with
just a single link, that data includes ModTime, Uid, Uname, Gid, Gname,
and Permissions. For files with more than one hard link, add the device
and inode numbers and the number of hard links to that. Systems that
discover their devices in an inconsistent order cause device numbers
to change. If you have more than one OS installed on that laptop, you
should always use the same one to do the backups.

--
Bob Nichols     "NOSPAM" is really part of my email address.
                 Do NOT delete it.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Why do I see a huge variation in the time rdiff-backup takes?

Segundo Bob
In reply to this post by Adrian Klaver-2
On 03/30/2015 04:52 PM, Adrian Klaver wrote:
> So is the same machine above and below?

Yes.  All the rdiff-backup runs listed were done on my desktop computer.

On 03/30/2015 04:52 PM, Adrian Klaver wrote:
>> ChangedFiles 187778
>
> The above would seem to be the cause. What changed the files?

Thank you.  I should have compared the statistics more carefully.  The
forest of insignificant (in this case) data hid the significant data
from me.

In all the rdiff-backup runs, the ElapsedTime is roughly 0.5 seconds
times the number of ChangedFiles.  Hence, the ElapsedTime data are fully
explained.

What changed 98.7% of my files?  Nothing changed them.  They were not
changed.  My explanation continues in my reply to Robert Nichols.

Thank you very much for your help.

--
Bob Hossley
[hidden email]

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Why do I see a huge variation in the time rdiff-backup takes?

Segundo Bob
In reply to this post by Robert Nichols-2
On 03/30/2015 07:39 PM, Robert Nichols wrote:
> Any change in the recorded metadata for a file will cause rdiff-backup
> to do a time-consuming diff of the old and new files. For files with
> just a single link, that data includes ModTime, Uid, Uname, Gid, Gname,
> and Permissions.

Thank you.  I think this explains why rdiff-backup mistakenly thought
that 98.7% of my files had been changed.

Between 2015-03-22 Sun and 2015-03-29 Sun, I switched from Xubuntu32
12.04 to Xubuntu64 14.04.  Below I list the file uid, uname, gid, and
gname for about 98.7% of the files backed up in each rdiff-backup run:

Xubuntu32 12.04
    uid 1000
    uname bob05
    gid 1000
    gname bob05

Xubuntu64 14:04
    uid 1000
    uname bob06
    gid 1000
    gname bob06

That is, the uname and gname changed for 98.7% of the files.  This
caused rdiff-backup to needlessly compare 98.7% of the files and to find
very few of them changed.

Perhaps uname and gname should not be included in the metadata?  They
are not part of the data stored in the file or the file's directory entry.

In the future, I will start a new archive file whenever I change from
one system-boot to another, if my uname or gname differs between the two
systems.

Thank you very much for your help.

--
Bob Hossley
[hidden email]

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Why do I see a huge variation in the time rdiff-backup takes?

Adrian Klaver-2
On 03/31/2015 03:37 PM, Segundo Bob wrote:

> On 03/30/2015 07:39 PM, Robert Nichols wrote:
>> Any change in the recorded metadata for a file will cause rdiff-backup
>> to do a time-consuming diff of the old and new files. For files with
>> just a single link, that data includes ModTime, Uid, Uname, Gid, Gname,
>> and Permissions.
>
> Thank you.  I think this explains why rdiff-backup mistakenly thought
> that 98.7% of my files had been changed.
>
> Between 2015-03-22 Sun and 2015-03-29 Sun, I switched from Xubuntu32
> 12.04 to Xubuntu64 14.04.  Below I list the file uid, uname, gid, and
> gname for about 98.7% of the files backed up in each rdiff-backup run:
>
> Xubuntu32 12.04
>      uid 1000
>      uname bob05
>      gid 1000
>      gname bob05
>
> Xubuntu64 14:04
>      uid 1000
>      uname bob06
>      gid 1000
>      gname bob06
>
> That is, the uname and gname changed for 98.7% of the files.  This
> caused rdiff-backup to needlessly compare 98.7% of the files and to find
> very few of them changed.
>
> Perhaps uname and gname should not be included in the metadata?  They
> are not part of the data stored in the file or the file's directory entry.

http://www.nongnu.org/rdiff-backup/rdiff-backup.1.html

--preserve-numerical-ids
               If  set,  rdiff-backup will preserve uids/gids instead of
trying
               to preserve unames and gnames.  See the USERS AND GROUPS
section
               for more information.

I would also read the USERS AND GROUPS section for the caveats and other
ways to deal with this.

>
> In the future, I will start a new archive file whenever I change from
> one system-boot to another, if my uname or gname differs between the two
> systems.
>
> Thank you very much for your help.
>


--
Adrian Klaver
[hidden email]

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Why do I see a huge variation in the time rdiff-backup takes?

Segundo Bob
Adrian Klaver,

Thank you very much for your help---which I wouldn't need so much of if
I only read the FINE documentation more closely.

Now I see that there are very good reasons why rdiff-backup does what it
does in the default case and that the options allow one to deal with
problems like mine.

You are an impressively quick responder.
--
Bob Hossley
[hidden email]

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Why do I see a huge variation in the time rdiff-backup takes?

Robert Nichols-2
In reply to this post by Adrian Klaver-2
On 03/31/2015 05:42 PM, Adrian Klaver wrote:

> http://www.nongnu.org/rdiff-backup/rdiff-backup.1.html
>
> --preserve-numerical-ids
>                If  set,  rdiff-backup will preserve uids/gids instead of
> trying
>                to preserve unames and gnames.  See the USERS AND GROUPS
> section
>                for more information.
>
> I would also read the USERS AND GROUPS section for the caveats and other
> ways to deal with this.

Better test that first. I believe that the original Uname is still
recorded in the mirror_metadata files, and any difference would
be treated as a change. Mapping of UIDs vs. Unames would affect
the actual ownership of the files in the mirror and increments
directories and the ownership of restored files.

--
Bob Nichols     "NOSPAM" is really part of my email address.
                 Do NOT delete it.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki