Backup freezing destination computer

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Backup freezing destination computer

Robert Farrington
I am using rdiff-backup 1.2.8 on Centos 7, backing up a remote Centos 7 computer.  It occasionally freezes up the destination computer, and I can't log in or ssh in to see what's wrong.  If I leave it logged in and it freezes, I still can't do anything to find out what's wrong.  I have to power cycle to get it back.  Any suggestions on how to fix this?Bob
_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Backup freezing destination computer

Patrik Dufresne-2
It's probably not directly related to rdiff-backup but the high volume of
IO on the server. If you have a monitoring system, could you take a look at
loadaverage ? It's probably going up when the backup is running. It's
telling you the IO of the server is probably not capable enough. It might
be related to various factor. Could you provide more technical details
regarding your setup ? Hardware, RAID config, SATA or SAS, FileSystem,
ext4, LVM ?

As an example. On one of our server, an LXC on Proxmox, running on ZFS, I
remeber I needed to disable a kernel feature related to huge memory page.
Otherwise, the container would died without apparent reason.

--
Patrik Dufresne Service Logiciel inc.
http://www.patrikdufresne.com <http://patrikdufresne.com/>/
514-971-6442
130 rue Doris
St-Colomban, QC J5K 1T9


On Wed, Jun 12, 2019 at 8:49 PM Robert Farrington <[hidden email]>
wrote:

> I am using rdiff-backup 1.2.8 on Centos 7, backing up a remote Centos 7
> computer.  It occasionally freezes up the destination computer, and I can't
> log in or ssh in to see what's wrong.  If I leave it logged in and it
> freezes, I still can't do anything to find out what's wrong.  I have to
> power cycle to get it back.  Any suggestions on how to fix this?Bob
> _______________________________________________
> rdiff-backup-users mailing list at [hidden email]
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL:
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Backup freezing destination computer

Robert Farrington
 Thanks for your response.  I don't think it's a hardware problem, but here's our hardware setup:
Pentium 4 3GHz2 1TB Western Digital hard drives in Raid-1 config, SATA-2 interfaceIt looks like we are using XFS and LVM.
 I know it's an old cpu, but the disks have been replaced recently.
I can't usually look at the load average when the system freezes because I can't do anything. But I have seen the CPU used 100% during a backup.

Bob
    On Wednesday, June 12, 2019, 8:00:03 PM PDT, Patrik Dufresne <[hidden email]> wrote:  
 
 It's probably not directly related to rdiff-backup but the high volume of IO on the server. If you have a monitoring system, could you take a look at loadaverage ? It's probably going up when the backup is running. It's telling you the IO of the server is probably not capable enough. It might be related to various factor. Could you provide more technical details regarding your setup ? Hardware, RAID config, SATA or SAS, FileSystem, ext4, LVM ?
As an example. On one of our server, an LXC on Proxmox, running on ZFS, I remeber I needed to disable a kernel feature related to huge memory page. Otherwise, the container would died without apparent reason.
--Patrik Dufresne Service Logiciel inc.http://www.patrikdufresne.com/
514-971-6442130 rue DorisSt-Colomban, QC J5K 1T9

On Wed, Jun 12, 2019 at 8:49 PM Robert Farrington <[hidden email]> wrote:

I am using rdiff-backup 1.2.8 on Centos 7, backing up a remote Centos 7 computer.  It occasionally freezes up the destination computer, and I can't log in or ssh in to see what's wrong.  If I leave it logged in and it freezes, I still can't do anything to find out what's wrong.  I have to power cycle to get it back.  Any suggestions on how to fix this?Bob
_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
 
_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Backup freezing destination computer

Patrik Dufresne-2
I would really continue digging into the IO performance. The CPU should not
reach 100% during a backup, it's supposed to mostly wait for the disk IO.
Most likely, your CPU is doing some extra work that should not take place.

You may also take a look at `iostat` to see if the disk are busy. You may
also run a benchmark on this system. In the past, I used "phoronix test
suite" which provide multiple test. In my case, one of the test never
complete and server died.

--
Patrik Dufresne Service Logiciel inc.
http://www.patrikdufresne.com <http://patrikdufresne.com/>/
514-971-6442
130 rue Doris
St-Colomban, QC J5K 1T9


On Fri, Jun 14, 2019 at 2:32 PM Robert Farrington <[hidden email]>
wrote:

> Thanks for your response.  I don't think it's a hardware problem, but
> here's our hardware setup:
>
> Pentium 4 3GHz
> 2 1TB Western Digital hard drives in Raid-1 config, SATA-2 interface
> It looks like we are using XFS and LVM.
>
> I know it's an old cpu, but the disks have been replaced recently.
>
> I can't usually look at the load average when the system freezes because I
> can't do anything. But I have seen the CPU used 100% during a backup.
>
> Bob
> On Wednesday, June 12, 2019, 8:00:03 PM PDT, Patrik Dufresne <
> [hidden email]> wrote:
>
>
> It's probably not directly related to rdiff-backup but the high volume of
> IO on the server. If you have a monitoring system, could you take a look at
> loadaverage ? It's probably going up when the backup is running. It's
> telling you the IO of the server is probably not capable enough. It might
> be related to various factor. Could you provide more technical details
> regarding your setup ? Hardware, RAID config, SATA or SAS, FileSystem,
> ext4, LVM ?
>
> As an example. On one of our server, an LXC on Proxmox, running on ZFS, I
> remeber I needed to disable a kernel feature related to huge memory page.
> Otherwise, the container would died without apparent reason.
>
> --
> Patrik Dufresne Service Logiciel inc.
> http://www.patrikdufresne.com <http://patrikdufresne.com/>/
> 514-971-6442
> 130 rue Doris
> St-Colomban, QC J5K 1T9
>
>
> On Wed, Jun 12, 2019 at 8:49 PM Robert Farrington <[hidden email]>
> wrote:
>
> I am using rdiff-backup 1.2.8 on Centos 7, backing up a remote Centos 7
> computer.  It occasionally freezes up the destination computer, and I can't
> log in or ssh in to see what's wrong.  If I leave it logged in and it
> freezes, I still can't do anything to find out what's wrong.  I have to
> power cycle to get it back.  Any suggestions on how to fix this?Bob
>
> _______________________________________________
> rdiff-backup-users mailing list at [hidden email]
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL:
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>
>
_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Backup freezing destination computer

Chris Wilson-2
It's also possible that rdiff-backup on the server is using too much memory
(or space on /tmp, if that's a ramdisk), causing the system to swap to
death. You could run vmstat on the server during the backup, and when it
freezes, look at the recent statistics (the last few lines before it froze).

Thanks, Chris.

On Fri, 14 Jun 2019 at 20:18, Patrik Dufresne <[hidden email]>
wrote:

> I would really continue digging into the IO performance. The CPU should not
> reach 100% during a backup, it's supposed to mostly wait for the disk IO.
> Most likely, your CPU is doing some extra work that should not take place.
>
> You may also take a look at `iostat` to see if the disk are busy. You may
> also run a benchmark on this system. In the past, I used "phoronix test
> suite" which provide multiple test. In my case, one of the test never
> complete and server died.
>
> --
> Patrik Dufresne Service Logiciel inc.
> http://www.patrikdufresne.com <http://patrikdufresne.com/>/
> 514-971-6442
> 130 rue Doris
> St-Colomban, QC J5K 1T9
>
>
> On Fri, Jun 14, 2019 at 2:32 PM Robert Farrington <[hidden email]>
> wrote:
>
> > Thanks for your response.  I don't think it's a hardware problem, but
> > here's our hardware setup:
> >
> > Pentium 4 3GHz
> > 2 1TB Western Digital hard drives in Raid-1 config, SATA-2 interface
> > It looks like we are using XFS and LVM.
> >
> > I know it's an old cpu, but the disks have been replaced recently.
> >
> > I can't usually look at the load average when the system freezes because
> I
> > can't do anything. But I have seen the CPU used 100% during a backup.
> >
> > Bob
> > On Wednesday, June 12, 2019, 8:00:03 PM PDT, Patrik Dufresne <
> > [hidden email]> wrote:
> >
> >
> > It's probably not directly related to rdiff-backup but the high volume of
> > IO on the server. If you have a monitoring system, could you take a look
> at
> > loadaverage ? It's probably going up when the backup is running. It's
> > telling you the IO of the server is probably not capable enough. It might
> > be related to various factor. Could you provide more technical details
> > regarding your setup ? Hardware, RAID config, SATA or SAS, FileSystem,
> > ext4, LVM ?
> >
> > As an example. On one of our server, an LXC on Proxmox, running on ZFS, I
> > remeber I needed to disable a kernel feature related to huge memory page.
> > Otherwise, the container would died without apparent reason.
> >
> > --
> > Patrik Dufresne Service Logiciel inc.
> > http://www.patrikdufresne.com <http://patrikdufresne.com/>/
> > 514-971-6442
> > 130 rue Doris
> > St-Colomban, QC J5K 1T9
> >
> >
> > On Wed, Jun 12, 2019 at 8:49 PM Robert Farrington <[hidden email]
> >
> > wrote:
> >
> > I am using rdiff-backup 1.2.8 on Centos 7, backing up a remote Centos 7
> > computer.  It occasionally freezes up the destination computer, and I
> can't
> > log in or ssh in to see what's wrong.  If I leave it logged in and it
> > freezes, I still can't do anything to find out what's wrong.  I have to
> > power cycle to get it back.  Any suggestions on how to fix this?Bob
> >
> > _______________________________________________
> > rdiff-backup-users mailing list at [hidden email]
> > https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> > Wiki URL:
> > http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
> >
> >
> _______________________________________________
> rdiff-backup-users mailing list at [hidden email]
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL:
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki