rdiff-backup result is missing text files but no errors, sourcedir to big >2TB

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

rdiff-backup result is missing text files but no errors, sourcedir to big >2TB

Jelle de Jong
Hello everybody,

I am trying to run an rdiff-backup and it keeps missing some documents
compared with the source, we are using a simple test.txt file to check
if the backups are still working and I got one big source backup of
+2.4TB. When running rdiff-backup there are no errors. So I can not
figure out what is going wrong.

I split the backup to make the source smaller and then it does make
backups of the test.txt file.

How can rdiff-backup handle bigger 3TB backups? The command I am running?

/usr/bin/python /usr/bin/rdiff-backup --tempdir
/srv/storage/backups/tempdir --exclude-device-files --exclude-sockets
--exclude-fifos /mnt/sr8-sdb2/
/srv/storage/backups/libvirt-filesystems/sr8-sdb2/

Kind regards,

Jelle de Jong
The Netherlands


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup result is missing text files but no errors, sourcedir to big >2TB

Lew Wolfgang
On 07/17/2019 04:48 AM, Jelle de Jong wrote:

> Hello everybody,
>
> I am trying to run an rdiff-backup and it keeps missing some documents
> compared with the source, we are using a simple test.txt file to check
> if the backups are still working and I got one big source backup of
> +2.4TB. When running rdiff-backup there are no errors. So I can not
> figure out what is going wrong.
>
> I split the backup to make the source smaller and then it does make
> backups of the test.txt file.
>
> How can rdiff-backup handle bigger 3TB backups? The command I am running?
>
> /usr/bin/python /usr/bin/rdiff-backup --tempdir
> /srv/storage/backups/tempdir --exclude-device-files --exclude-sockets
> --exclude-fifos /mnt/sr8-sdb2/
> /srv/storage/backups/libvirt-filesystems/sr8-sdb2/

Just a wild guess:  a size limitation in your disk partition, filesystem, or
operating system?

What operating system:
32 or 64-bits:
Partition table:  (MBR or GPT)
Filesystem type:  (NTFS, EXT4, ZFS, XFS, etc)
Size of destination partition:
BIOS type:  (Legacy or UEFI)

Regards,
Lew


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup result is missing text files but no errors, sourcedir to big >2TB

Yves Bellefeuille
"Lew Wolfgang" <[hidden email]> wrote:

>  Just a wild guess: a size limitation in your disk partition,
>  filesystem, or operating system?

I'd suspect the same thing. Can you clarify whether 2 TB is the size
of a file, of the entire backup, of the partition, or what?

ext2 and ext3 have a maximum file size of 2 TB (actually 2 TiB).

--
Yves Bellefeuille
<[hidden email]>



_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup result is missing text files but no errors, sourcedir to big >2TB

Jelle de Jong
In reply to this post by Lew Wolfgang
On 7/18/19 4:45 PM, Lew Wolfgang wrote:

> On 07/17/2019 04:48 AM, Jelle de Jong wrote:
>> Hello everybody,
>>
>> I am trying to run an rdiff-backup and it keeps missing some documents
>> compared with the source, we are using a simple test.txt file to check
>> if the backups are still working and I got one big source backup of
>> +2.4TB. When running rdiff-backup there are no errors. So I can not
>> figure out what is going wrong.
>>
>> I split the backup to make the source smaller and then it does make
>> backups of the test.txt file.
>>
>> How can rdiff-backup handle bigger 3TB backups? The command I am running?
>>
>> /usr/bin/python /usr/bin/rdiff-backup --tempdir
>> /srv/storage/backups/tempdir --exclude-device-files --exclude-sockets
>> --exclude-fifos /mnt/sr8-sdb2/
>> /srv/storage/backups/libvirt-filesystems/sr8-sdb2/
>
> Just a wild guess:  a size limitation in your disk partition,
> filesystem, or
> operating system?
>
> What operating system:
> 32 or 64-bits:
> Partition table:  (MBR or GPT)
> Filesystem type:  (NTFS, EXT4, ZFS, XFS, etc)
> Size of destination partition:
> BIOS type:  (Legacy or UEFI)

Thank you all for responding, here is the information.

# rdiff-backup --version
rdiff-backup 1.2.8

root@backup:~# uname -a
Linux backup 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5 (2019-06-19) x86_64
GNU/Linux

root@backup:~# df -hal /srv/storage/backups/libvirt-filesystems/sr8-sdb2
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/lvm0--vol-storage   22T   15T  7.6T  66% /srv/storage

root@backup:~# mount | grep  /dev/mapper/lvm0--vol-storage
/dev/mapper/lvm0--vol-storage on /srv/storage type xfs
(rw,relatime,attr2,inode64,logbsize=64k,sunit=128,swidth=1024,noquota)

root@backup:~# mount | grep /boot
/dev/sdr1 on /boot type ext3 (rw,relatime)

root@backup:~# parted /dev/sdr1 print
Model: Unknown (unknown)
Disk /dev/sdr1: 30.7GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
  1      0.00B  30.7GB  30.7GB  ext3

# Legacy bios
root@backup:~# dmidecode -t bios
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 2.5 present.

Handle 0x0005, DMI type 0, 24 bytes
BIOS Information
        Vendor: Intel Corp.
        Version: S5500.86B.01.00.0061.030920121535
        Release Date: 03/09/2012
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 8192 kB
        Characteristics:
                PCI is supported
                PNP is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                EDD is supported
                3.5"/2.88 MB floppy services are supported (int 13h)
                Print screen service is supported (int 5h)
                8042 keyboard services are supported (int 9h)
                Serial services are supported (int 14h)
                CGA/mono video services are supported (int 10h)
                ACPI is supported
                USB legacy is supported
                LS-120 boot is supported
                ATAPI Zip drive boot is supported
                Function key-initiated network boot is supported
                Targeted content distribution is supported
        BIOS Revision: 17.18
        Firmware Revision: 0.0

Handle 0x002D, DMI type 13, 22 bytes
BIOS Language Information
        Language Description Format: Long
        Installable Languages: 1
                en|US|iso8859-1
        Currently Installed Language: en|US|iso8859-1

/usr/bin/python /usr/bin/rdiff-backup --tempdir
/srv/storage/backups/tempdir --exclude-device-files --exclude-sockets
--exclude-fifos /mnt/sr8-sda2
/srv/storage/backups/libvirt-filesystems/sr8-sda2

root@backup:~# dd if=/mnt/sr8-sda2/pagefile.sys
of=/srv/storage/backups/tempdir/pagefile.sys
2621440+0 records in
2621440+0 records out
1342177280 bytes (1.3 GB, 1.2 GiB) copied, 11.8273 s, 113 MB/s

Is there an option to speed up /usr/bin/rdiff-backup it runs with about
01-10M/s compared to the possible 113M/s on the underlying systems.

Kind regards,

Jelle de Jong


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup result is missing text files but no errors, sourcedir to big >2TB

Dominic Raferd-3
On Fri, 19 Jul 2019 at 20:43, Jelle de Jong <[hidden email]>
wrote:

> On 7/18/19 4:45 PM, Lew Wolfgang wrote:
> > On 07/17/2019 04:48 AM, Jelle de Jong wrote:
> >> Hello everybody,
> >>
> >> I am trying to run an rdiff-backup and it keeps missing some documents
> >> compared with the source, we are using a simple test.txt file to check
> >> if the backups are still working and I got one big source backup of
> >> +2.4TB. When running rdiff-backup there are no errors. So I can not
> >> figure out what is going wrong.
> >>
> >> I split the backup to make the source smaller and then it does make
> >> backups of the test.txt file.
> >>
> >> How can rdiff-backup handle bigger 3TB backups? The command I am
> running?
> >>
> >> /usr/bin/python /usr/bin/rdiff-backup --tempdir
> >> /srv/storage/backups/tempdir --exclude-device-files --exclude-sockets
> >> --exclude-fifos /mnt/sr8-sdb2/
> >> /srv/storage/backups/libvirt-filesystems/sr8-sdb2/
> >
> > Just a wild guess:  a size limitation in your disk partition,
> > filesystem, or
> > operating system?
> >
> > What operating system:
> > 32 or 64-bits:
> > Partition table:  (MBR or GPT)
> > Filesystem type:  (NTFS, EXT4, ZFS, XFS, etc)
> > Size of destination partition:
> > BIOS type:  (Legacy or UEFI)
>
> Thank you all for responding, here is the information.
>
> # rdiff-backup --version
> rdiff-backup 1.2.8
>
> root@backup:~# uname -a
> Linux backup 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5 (2019-06-19) x86_64
> GNU/Linux
>
> root@backup:~# df -hal /srv/storage/backups/libvirt-filesystems/sr8-sdb2
> Filesystem                     Size  Used Avail Use% Mounted on
> /dev/mapper/lvm0--vol-storage   22T   15T  7.6T  66% /srv/storage
>
> root@backup:~# mount | grep  /dev/mapper/lvm0--vol-storage
> /dev/mapper/lvm0--vol-storage on /srv/storage type xfs
> (rw,relatime,attr2,inode64,logbsize=64k,sunit=128,swidth=1024,noquota)
>
> root@backup:~# mount | grep /boot
> /dev/sdr1 on /boot type ext3 (rw,relatime)
>
> root@backup:~# parted /dev/sdr1 print
> Model: Unknown (unknown)
> Disk /dev/sdr1: 30.7GB
> Sector size (logical/physical): 512B/512B
> Partition Table: loop
> Disk Flags:
>
> Number  Start  End     Size    File system  Flags
>   1      0.00B  30.7GB  30.7GB  ext3
>
> # Legacy bios
> root@backup:~# dmidecode -t bios
> # dmidecode 3.2
> Getting SMBIOS data from sysfs.
> SMBIOS 2.5 present.
>
> Handle 0x0005, DMI type 0, 24 bytes
> BIOS Information
>         Vendor: Intel Corp.
>         Version: S5500.86B.01.00.0061.030920121535
>         Release Date: 03/09/2012
>         Address: 0xF0000
>         Runtime Size: 64 kB
>         ROM Size: 8192 kB
>         Characteristics:
>                 PCI is supported
>                 PNP is supported
>                 BIOS is upgradeable
>                 BIOS shadowing is allowed
>                 Boot from CD is supported
>                 Selectable boot is supported
>                 EDD is supported
>                 3.5"/2.88 MB floppy services are supported (int 13h)
>                 Print screen service is supported (int 5h)
>                 8042 keyboard services are supported (int 9h)
>                 Serial services are supported (int 14h)
>                 CGA/mono video services are supported (int 10h)
>                 ACPI is supported
>                 USB legacy is supported
>                 LS-120 boot is supported
>                 ATAPI Zip drive boot is supported
>                 Function key-initiated network boot is supported
>                 Targeted content distribution is supported
>         BIOS Revision: 17.18
>         Firmware Revision: 0.0
>
> Handle 0x002D, DMI type 13, 22 bytes
> BIOS Language Information
>         Language Description Format: Long
>         Installable Languages: 1
>                 en|US|iso8859-1
>         Currently Installed Language: en|US|iso8859-1
>
> /usr/bin/python /usr/bin/rdiff-backup --tempdir
> /srv/storage/backups/tempdir --exclude-device-files --exclude-sockets
> --exclude-fifos /mnt/sr8-sda2
> /srv/storage/backups/libvirt-filesystems/sr8-sda2
>
> root@backup:~# dd if=/mnt/sr8-sda2/pagefile.sys
> of=/srv/storage/backups/tempdir/pagefile.sys
> 2621440+0 records in
> 2621440+0 records out
> 1342177280 bytes (1.3 GB, 1.2 GiB) copied, 11.8273 s, 113 MB/s
>
> Is there an option to speed up /usr/bin/rdiff-backup it runs with about
> 01-10M/s compared to the possible 113M/s on the underlying systems.


Could there be some permission / file lock issues on the source? Have you
tried creating an LVM snapshot of the source and backing up from that
instead?

Regarding speed, you can try turning off compression in rdiff-backup with
the --no-compression switch.
_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki