rdiff-backup performance -- slow operation on initial backup?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

rdiff-backup performance -- slow operation on initial backup?

Derek Atkins-2
Hi,

I'm trying to use rdiff-backup to backup a bunch of servers.  One
particular server contains about 160GB of data, but when I try to perform
the rdiff-backup it's saving the data at a measly 1MB/s.

Here's my configuration:

  [server] <--ssh--> [backup-server]{encfs} <--nfs--> [freenas]

I ran a bunch of tests to try to figure out my bottlenecks.

I ran a bunch of dd tests (using dd if=/dev/zero bs=1M count=1000) on the
backup server.  Going directly to FreeNAS via NFS (bybassing encfs) I get
50.2MB/s.  If I run dd directly on the backup server (through encfs) I get
20.1MB/s.  If I go over SSH from the backup server to the target server
and run the dd on the target server, then write to FreeNas through encfs
declines to 7.6MB/s.

Note that in those SSH tests, however, I forgot to turn off compression.
When I do that, the throughput for the dd test reduced to 6.6BM/s.  (Note
that this is running simultaneously with a running rdiff-backup, so it's
possible that they are reducing performance).

Then I ran an scp test to the same target server; copying about 1.4GB of
photos.  Files ranged in size from 10KB to 5MB.  When run in standard mode
(displaying each file status) I got 4.4MB/s.  Running in quiet mode I get
5.1MB/s.

So clearly the bottleneck is in rdiff-backup -- performance (IMHO) should
not be significantly slower than the last dd-over-ssh test.  It appears
rdiff-backup is slowing me down by a factor of 5x throughput versus scp.

I found a message from Ben from 2005 where he suggests increasing the
blocksize and conn_bufsiz settings in Globals.py:
https://lists.gnu.org/archive/html/rdiff-backup-users/2005-10/msg00062.html

What he didn't say was whether this needed to be changed on the target
server, the backup server, or both.  Nor do I know if that would actually
help this situation.

Do you have any ideas?

Thanks,

-derek

PS: According to rpm, both systems are running version 1.2.8.

--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup performance -- slow operation on initial backup?

Derek Atkins-2
Just a quick update:  I tried making these changes on both sides and it
really didn't make a difference.  Full backup of 202852072 Kbytes
required 2267m25.913s (previously it took 2346m57.800s, so it only sped
up a factor of 3%.

Only thing I have not yet tried is running a raw rsync to see how fast
that runs.  I'll do that next.

So, back to my orignal question: anyone have any idea how to get initial
transfers to run faster (or indeed any significant data transfers)?

Thanks,

-derek

"Derek Atkins" <[hidden email]> writes:

> Hi,
>
> I'm trying to use rdiff-backup to backup a bunch of servers.  One
> particular server contains about 160GB of data, but when I try to perform
> the rdiff-backup it's saving the data at a measly 1MB/s.
>
> Here's my configuration:
>
>   [server] <--ssh--> [backup-server]{encfs} <--nfs--> [freenas]
>
> I ran a bunch of tests to try to figure out my bottlenecks.
>
> I ran a bunch of dd tests (using dd if=/dev/zero bs=1M count=1000) on the
> backup server.  Going directly to FreeNAS via NFS (bybassing encfs) I get
> 50.2MB/s.  If I run dd directly on the backup server (through encfs) I get
> 20.1MB/s.  If I go over SSH from the backup server to the target server
> and run the dd on the target server, then write to FreeNas through encfs
> declines to 7.6MB/s.
>
> Note that in those SSH tests, however, I forgot to turn off compression.
> When I do that, the throughput for the dd test reduced to 6.6BM/s.  (Note
> that this is running simultaneously with a running rdiff-backup, so it's
> possible that they are reducing performance).
>
> Then I ran an scp test to the same target server; copying about 1.4GB of
> photos.  Files ranged in size from 10KB to 5MB.  When run in standard mode
> (displaying each file status) I got 4.4MB/s.  Running in quiet mode I get
> 5.1MB/s.
>
> So clearly the bottleneck is in rdiff-backup -- performance (IMHO) should
> not be significantly slower than the last dd-over-ssh test.  It appears
> rdiff-backup is slowing me down by a factor of 5x throughput versus scp.
>
> I found a message from Ben from 2005 where he suggests increasing the
> blocksize and conn_bufsiz settings in Globals.py:
> https://lists.gnu.org/archive/html/rdiff-backup-users/2005-10/msg00062.html
>
> What he didn't say was whether this needed to be changed on the target
> server, the backup server, or both.  Nor do I know if that would actually
> help this situation.
>
> Do you have any ideas?
>
> Thanks,
>
> -derek
>
> PS: According to rpm, both systems are running version 1.2.8.

--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup performance -- slow operation on initial backup?

Dominic Raferd-3
Is this really your first rdiff-backup to this location? If you have any previous rdiff-backup runs to this repository then the situation is complicated by rdiff-backup's need to create a new set of reverse diff files to be able to regress to previous file contents.

What is your /tmp location? rdiff-backup uses this location for some operations though not AFAIK for standard backup runs. Still, if /tmp is on encfs maybe it could be a culprit; you can override rdiff-backup's temporary file location with --tempdir and --remote-tempdir.

Might also be worth trying --ssh-no-compression.

Dominic


On 28 March 2016 at 14:37, Derek Atkins <[hidden email]> wrote:
Just a quick update:  I tried making these changes on both sides and it
really didn't make a difference.  Full backup of 202852072 Kbytes
required 2267m25.913s (previously it took 2346m57.800s, so it only sped
up a factor of 3%.

Only thing I have not yet tried is running a raw rsync to see how fast
that runs.  I'll do that next.

So, back to my orignal question: anyone have any idea how to get initial
transfers to run faster (or indeed any significant data transfers)?

Thanks,

-derek

"Derek Atkins" <[hidden email]> writes:

> Hi,
>
> I'm trying to use rdiff-backup to backup a bunch of servers.  One
> particular server contains about 160GB of data, but when I try to perform
> the rdiff-backup it's saving the data at a measly 1MB/s.
>
> Here's my configuration:
>
>   [server] <--ssh--> [backup-server]{encfs} <--nfs--> [freenas]
>
> I ran a bunch of tests to try to figure out my bottlenecks.
>
> I ran a bunch of dd tests (using dd if=/dev/zero bs=1M count=1000) on the
> backup server.  Going directly to FreeNAS via NFS (bybassing encfs) I get
> 50.2MB/s.  If I run dd directly on the backup server (through encfs) I get
> 20.1MB/s.  If I go over SSH from the backup server to the target server
> and run the dd on the target server, then write to FreeNas through encfs
> declines to 7.6MB/s.
>
> Note that in those SSH tests, however, I forgot to turn off compression.
> When I do that, the throughput for the dd test reduced to 6.6BM/s.  (Note
> that this is running simultaneously with a running rdiff-backup, so it's
> possible that they are reducing performance).
>
> Then I ran an scp test to the same target server; copying about 1.4GB of
> photos.  Files ranged in size from 10KB to 5MB.  When run in standard mode
> (displaying each file status) I got 4.4MB/s.  Running in quiet mode I get
> 5.1MB/s.
>
> So clearly the bottleneck is in rdiff-backup -- performance (IMHO) should
> not be significantly slower than the last dd-over-ssh test.  It appears
> rdiff-backup is slowing me down by a factor of 5x throughput versus scp.
>
> I found a message from Ben from 2005 where he suggests increasing the
> blocksize and conn_bufsiz settings in Globals.py:
> https://lists.gnu.org/archive/html/rdiff-backup-users/2005-10/msg00062.html
>
> What he didn't say was whether this needed to be changed on the target
> server, the backup server, or both.  Nor do I know if that would actually
> help this situation.
>
> Do you have any ideas?
>
> Thanks,
>
> -derek
>
> PS: According to rpm, both systems are running version 1.2.8.

--
       Derek Atkins                 <a href="tel:617-623-3745" value="+16176233745">617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup performance -- slow operation on initial backup?

Derek Atkins-2
Hi,

Thank you for taking the time to look at this..

On Mon, March 28, 2016 10:41 am, Dominic Raferd wrote:
> Is this really your first rdiff-backup to this location? If you have any
> previous rdiff-backup runs to this repository then the situation is
> complicated by rdiff-backup's need to create a new set of reverse diff
> files to be able to regress to previous file contents.

Yes, this s really the first rdiff-backup to this location.

A second backup run shortly after the first one completed finished in 55
minutes.

> What is your /tmp location? rdiff-backup uses this location for some
> operations though not AFAIK for standard backup runs. Still, if /tmp is on
> encfs maybe it could be a culprit; you can override rdiff-backup's
> temporary file location with --tempdir and --remote-tempdir.

If it is truly /tmp then no; /tmp is a ramdisk on the backup server and is
on the root disk on the target server.  Neither are being run through
encfs.

If, however, you mean the rdiff-backup-data.tmp files, those ARE being run
through encfs.

> Might also be worth trying --ssh-no-compression.

I already have "Compression no" set in ~/.ssh/config so I'm not sure what
this would add?

> Dominic
> http://www.timedicer.co.uk

-derek

PS: I'm running a raw rsync command now just to see how it behaves -- so
far I'm only seeing about 2MB/s, but it's only been running for 10 minutes
or so.

>
> On 28 March 2016 at 14:37, Derek Atkins <[hidden email]> wrote:
>
>> Just a quick update:  I tried making these changes on both sides and it
>> really didn't make a difference.  Full backup of 202852072 Kbytes
>> required 2267m25.913s (previously it took 2346m57.800s, so it only sped
>> up a factor of 3%.
>>
>> Only thing I have not yet tried is running a raw rsync to see how fast
>> that runs.  I'll do that next.
>>
>> So, back to my orignal question: anyone have any idea how to get initial
>> transfers to run faster (or indeed any significant data transfers)?
>>
>> Thanks,
>>
>> -derek
>>
>> "Derek Atkins" <[hidden email]> writes:
>>
>> > Hi,
>> >
>> > I'm trying to use rdiff-backup to backup a bunch of servers.  One
>> > particular server contains about 160GB of data, but when I try to
>> perform
>> > the rdiff-backup it's saving the data at a measly 1MB/s.
>> >
>> > Here's my configuration:
>> >
>> >   [server] <--ssh--> [backup-server]{encfs} <--nfs--> [freenas]
>> >
>> > I ran a bunch of tests to try to figure out my bottlenecks.
>> >
>> > I ran a bunch of dd tests (using dd if=/dev/zero bs=1M count=1000) on
>> the
>> > backup server.  Going directly to FreeNAS via NFS (bybassing encfs) I
>> get
>> > 50.2MB/s.  If I run dd directly on the backup server (through encfs) I
>> get
>> > 20.1MB/s.  If I go over SSH from the backup server to the target
>> server
>> > and run the dd on the target server, then write to FreeNas through
>> encfs
>> > declines to 7.6MB/s.
>> >
>> > Note that in those SSH tests, however, I forgot to turn off
>> compression.
>> > When I do that, the throughput for the dd test reduced to 6.6BM/s.
>> (Note
>> > that this is running simultaneously with a running rdiff-backup, so
>> it's
>> > possible that they are reducing performance).
>> >
>> > Then I ran an scp test to the same target server; copying about 1.4GB
>> of
>> > photos.  Files ranged in size from 10KB to 5MB.  When run in standard
>> mode
>> > (displaying each file status) I got 4.4MB/s.  Running in quiet mode I
>> get
>> > 5.1MB/s.
>> >
>> > So clearly the bottleneck is in rdiff-backup -- performance (IMHO)
>> should
>> > not be significantly slower than the last dd-over-ssh test.  It
>> appears
>> > rdiff-backup is slowing me down by a factor of 5x throughput versus
>> scp.
>> >
>> > I found a message from Ben from 2005 where he suggests increasing the
>> > blocksize and conn_bufsiz settings in Globals.py:
>> >
>> https://lists.gnu.org/archive/html/rdiff-backup-users/2005-10/msg00062.html
>> >
>> > What he didn't say was whether this needed to be changed on the target
>> > server, the backup server, or both.  Nor do I know if that would
>> actually
>> > help this situation.
>> >
>> > Do you have any ideas?
>> >
>> > Thanks,
>> >
>> > -derek
>> >
>> > PS: According to rpm, both systems are running version 1.2.8.
>>
>> --
>>        Derek Atkins                 617-623-3745
>>        [hidden email]             www.ihtfp.com
>>        Computer and Internet Security Consultant
>>
>> _______________________________________________
>> rdiff-backup-users mailing list at [hidden email]
>> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>> Wiki URL:
>> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>>
>


--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup performance -- slow operation on initial backup?

Derek Atkins-2
Hi,

On Mon, March 28, 2016 10:56 am, Derek Atkins wrote:

> Hi,
>
> Thank you for taking the time to look at this..
>
> On Mon, March 28, 2016 10:41 am, Dominic Raferd wrote:
>> Is this really your first rdiff-backup to this location? If you have any
>> previous rdiff-backup runs to this repository then the situation is
>> complicated by rdiff-backup's need to create a new set of reverse diff
>> files to be able to regress to previous file contents.
>
> Yes, this s really the first rdiff-backup to this location.
>
> A second backup run shortly after the first one completed finished in 55
> minutes.
>
>> What is your /tmp location? rdiff-backup uses this location for some
>> operations though not AFAIK for standard backup runs. Still, if /tmp is
>> on
>> encfs maybe it could be a culprit; you can override rdiff-backup's
>> temporary file location with --tempdir and --remote-tempdir.
>
> If it is truly /tmp then no; /tmp is a ramdisk on the backup server and is
> on the root disk on the target server.  Neither are being run through
> encfs.
>
> If, however, you mean the rdiff-backup-data.tmp files, those ARE being run
> through encfs.
>
>> Might also be worth trying --ssh-no-compression.
>
> I already have "Compression no" set in ~/.ssh/config so I'm not sure what
> this would add?
>
>> Dominic
>> http://www.timedicer.co.uk
>
> -derek
>
> PS: I'm running a raw rsync command now just to see how it behaves -- so
> far I'm only seeing about 2MB/s, but it's only been running for 10 minutes
> or so.

My rsync backup just finished.  It copied 202688912 KB in 1599m55.255s  so
about 2.1MB/s.  Still significantly slower than SCP, but faster than
rdiff-backup.

The command I ran was:

rsync -art -e "ssh -l root -i /dev/null -o Compression=no"
root@server:/var/www/ /backups/server

:-(

-derek

>
>>
>> On 28 March 2016 at 14:37, Derek Atkins <[hidden email]> wrote:
>>
>>> Just a quick update:  I tried making these changes on both sides and it
>>> really didn't make a difference.  Full backup of 202852072 Kbytes
>>> required 2267m25.913s (previously it took 2346m57.800s, so it only sped
>>> up a factor of 3%.
>>>
>>> Only thing I have not yet tried is running a raw rsync to see how fast
>>> that runs.  I'll do that next.
>>>
>>> So, back to my orignal question: anyone have any idea how to get
>>> initial
>>> transfers to run faster (or indeed any significant data transfers)?
>>>
>>> Thanks,
>>>
>>> -derek
>>>
>>> "Derek Atkins" <[hidden email]> writes:
>>>
>>> > Hi,
>>> >
>>> > I'm trying to use rdiff-backup to backup a bunch of servers.  One
>>> > particular server contains about 160GB of data, but when I try to
>>> perform
>>> > the rdiff-backup it's saving the data at a measly 1MB/s.
>>> >
>>> > Here's my configuration:
>>> >
>>> >   [server] <--ssh--> [backup-server]{encfs} <--nfs--> [freenas]
>>> >
>>> > I ran a bunch of tests to try to figure out my bottlenecks.
>>> >
>>> > I ran a bunch of dd tests (using dd if=/dev/zero bs=1M count=1000) on
>>> the
>>> > backup server.  Going directly to FreeNAS via NFS (bybassing encfs) I
>>> get
>>> > 50.2MB/s.  If I run dd directly on the backup server (through encfs)
>>> I
>>> get
>>> > 20.1MB/s.  If I go over SSH from the backup server to the target
>>> server
>>> > and run the dd on the target server, then write to FreeNas through
>>> encfs
>>> > declines to 7.6MB/s.
>>> >
>>> > Note that in those SSH tests, however, I forgot to turn off
>>> compression.
>>> > When I do that, the throughput for the dd test reduced to 6.6BM/s.
>>> (Note
>>> > that this is running simultaneously with a running rdiff-backup, so
>>> it's
>>> > possible that they are reducing performance).
>>> >
>>> > Then I ran an scp test to the same target server; copying about 1.4GB
>>> of
>>> > photos.  Files ranged in size from 10KB to 5MB.  When run in standard
>>> mode
>>> > (displaying each file status) I got 4.4MB/s.  Running in quiet mode I
>>> get
>>> > 5.1MB/s.
>>> >
>>> > So clearly the bottleneck is in rdiff-backup -- performance (IMHO)
>>> should
>>> > not be significantly slower than the last dd-over-ssh test.  It
>>> appears
>>> > rdiff-backup is slowing me down by a factor of 5x throughput versus
>>> scp.
>>> >
>>> > I found a message from Ben from 2005 where he suggests increasing the
>>> > blocksize and conn_bufsiz settings in Globals.py:
>>> >
>>> https://lists.gnu.org/archive/html/rdiff-backup-users/2005-10/msg00062.html
>>> >
>>> > What he didn't say was whether this needed to be changed on the
>>> target
>>> > server, the backup server, or both.  Nor do I know if that would
>>> actually
>>> > help this situation.
>>> >
>>> > Do you have any ideas?
>>> >
>>> > Thanks,
>>> >
>>> > -derek
>>> >
>>> > PS: According to rpm, both systems are running version 1.2.8.
>>>
>>> --
>>>        Derek Atkins                 617-623-3745
>>>        [hidden email]             www.ihtfp.com
>>>        Computer and Internet Security Consultant
>>>
>>> _______________________________________________
>>> rdiff-backup-users mailing list at [hidden email]
>>> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>>> Wiki URL:
>>> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>>>
>>
>
>
> --
>        Derek Atkins                 617-623-3745
>        [hidden email]             www.ihtfp.com
>        Computer and Internet Security Consultant
>
>
> _______________________________________________
> rdiff-backup-users mailing list at [hidden email]
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL:
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>


--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup performance -- slow operation on initial backup?

Dominic Raferd-3
It doesn't sound like rdiff-backup is the culprit here. You could try hpn-ssh https://sourceforge.net/projects/hpnssh/ ?

On 29 March 2016 at 21:03, Derek Atkins <[hidden email]> wrote:
Hi,

On Mon, March 28, 2016 10:56 am, Derek Atkins wrote:
> Hi,
>
> Thank you for taking the time to look at this..
>
> On Mon, March 28, 2016 10:41 am, Dominic Raferd wrote:
>> Is this really your first rdiff-backup to this location? If you have any
>> previous rdiff-backup runs to this repository then the situation is
>> complicated by rdiff-backup's need to create a new set of reverse diff
>> files to be able to regress to previous file contents.
>
> Yes, this s really the first rdiff-backup to this location.
>
> A second backup run shortly after the first one completed finished in 55
> minutes.
>
>> What is your /tmp location? rdiff-backup uses this location for some
>> operations though not AFAIK for standard backup runs. Still, if /tmp is
>> on
>> encfs maybe it could be a culprit; you can override rdiff-backup's
>> temporary file location with --tempdir and --remote-tempdir.
>
> If it is truly /tmp then no; /tmp is a ramdisk on the backup server and is
> on the root disk on the target server.  Neither are being run through
> encfs.
>
> If, however, you mean the rdiff-backup-data.tmp files, those ARE being run
> through encfs.
>
>> Might also be worth trying --ssh-no-compression.
>
> I already have "Compression no" set in ~/.ssh/config so I'm not sure what
> this would add?
>
>> Dominic
>> http://www.timedicer.co.uk
>
> -derek
>
> PS: I'm running a raw rsync command now just to see how it behaves -- so
> far I'm only seeing about 2MB/s, but it's only been running for 10 minutes
> or so.

My rsync backup just finished.  It copied 202688912 KB in 1599m55.255s  so
about 2.1MB/s.  Still significantly slower than SCP, but faster than
rdiff-backup.

The command I ran was:

rsync -art -e "ssh -l root -i /dev/null -o Compression=no"
root@server:/var/www/ /backups/server

:-(

-derek

>
>>
>> On 28 March 2016 at 14:37, Derek Atkins <[hidden email]> wrote:
>>
>>> Just a quick update:  I tried making these changes on both sides and it
>>> really didn't make a difference.  Full backup of 202852072 Kbytes
>>> required 2267m25.913s (previously it took 2346m57.800s, so it only sped
>>> up a factor of 3%.
>>>
>>> Only thing I have not yet tried is running a raw rsync to see how fast
>>> that runs.  I'll do that next.
>>>
>>> So, back to my orignal question: anyone have any idea how to get
>>> initial
>>> transfers to run faster (or indeed any significant data transfers)?
>>>
>>> Thanks,
>>>
>>> -derek
>>>
>>> "Derek Atkins" <[hidden email]> writes:
>>>
>>> > Hi,
>>> >
>>> > I'm trying to use rdiff-backup to backup a bunch of servers.  One
>>> > particular server contains about 160GB of data, but when I try to
>>> perform
>>> > the rdiff-backup it's saving the data at a measly 1MB/s.
>>> >
>>> > Here's my configuration:
>>> >
>>> >   [server] <--ssh--> [backup-server]{encfs} <--nfs--> [freenas]
>>> >
>>> > I ran a bunch of tests to try to figure out my bottlenecks.
>>> >
>>> > I ran a bunch of dd tests (using dd if=/dev/zero bs=1M count=1000) on
>>> the
>>> > backup server.  Going directly to FreeNAS via NFS (bybassing encfs) I
>>> get
>>> > 50.2MB/s.  If I run dd directly on the backup server (through encfs)
>>> I
>>> get
>>> > 20.1MB/s.  If I go over SSH from the backup server to the target
>>> server
>>> > and run the dd on the target server, then write to FreeNas through
>>> encfs
>>> > declines to 7.6MB/s.
>>> >
>>> > Note that in those SSH tests, however, I forgot to turn off
>>> compression.
>>> > When I do that, the throughput for the dd test reduced to 6.6BM/s.
>>> (Note
>>> > that this is running simultaneously with a running rdiff-backup, so
>>> it's
>>> > possible that they are reducing performance).
>>> >
>>> > Then I ran an scp test to the same target server; copying about 1.4GB
>>> of
>>> > photos.  Files ranged in size from 10KB to 5MB.  When run in standard
>>> mode
>>> > (displaying each file status) I got 4.4MB/s.  Running in quiet mode I
>>> get
>>> > 5.1MB/s.
>>> >
>>> > So clearly the bottleneck is in rdiff-backup -- performance (IMHO)
>>> should
>>> > not be significantly slower than the last dd-over-ssh test.  It
>>> appears
>>> > rdiff-backup is slowing me down by a factor of 5x throughput versus
>>> scp.
>>> >
>>> > I found a message from Ben from 2005 where he suggests increasing the
>>> > blocksize and conn_bufsiz settings in Globals.py:
>>> >
>>> https://lists.gnu.org/archive/html/rdiff-backup-users/2005-10/msg00062.html
>>> >
>>> > What he didn't say was whether this needed to be changed on the
>>> target
>>> > server, the backup server, or both.  Nor do I know if that would
>>> actually
>>> > help this situation.
>>> >
>>> > Do you have any ideas?
>>> >
>>> > Thanks,
>>> >
>>> > -derek
>>> >
>>> > PS: According to rpm, both systems are running version 1.2.8.
>>>
>>> --
>>>        Derek Atkins                 <a href="tel:617-623-3745" value="+16176233745">617-623-3745
>>>        [hidden email]             www.ihtfp.com
>>>        Computer and Internet Security Consultant
>>>
>>> _______________________________________________
>>> rdiff-backup-users mailing list at [hidden email]
>>> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>>> Wiki URL:
>>> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>>>
>>
>
>
> --
>        Derek Atkins                 <a href="tel:617-623-3745" value="+16176233745">617-623-3745
>        [hidden email]             www.ihtfp.com
>        Computer and Internet Security Consultant
>
>
> _______________________________________________
> rdiff-backup-users mailing list at [hidden email]
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL:
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>


--
       Derek Atkins                 <a href="tel:617-623-3745" value="+16176233745">617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant



_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup performance -- slow operation on initial backup?

Greg Troxel

Dominic Raferd <[hidden email]> writes:

> It doesn't sound like rdiff-backup is the culprit here. You could try
> hpn-ssh https://sourceforge.net/projects/hpnssh/ ?

I would be very surprised if normal people have networks available that
really need hpn-ssh, and 2 MB/s is not that fast.  urely rsync and
rdiff-backup are running over ssh, so that should have the same
transport properties.

I would up the send/receive socket buffers (because it's easy, not
because I think that's the problem), and watch disk/cpu on both sides,
and also run netstat to see if data is piling up in the transmit socket
buffer.

FWIW, I used to use rdiff-backup but found it to be nonrobust on
machines with limited (only a few GB) RAM and hundreds of GB of backup.
I have switched to bup.

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

signature.asc (186 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup performance -- slow operation on initial backup?

Derek Atkins-2
In reply to this post by Dominic Raferd-3
Hi,

You are right that it doesn't seem like rdiff-backup alone is the issue;
but to me it sounds like part of the issue is librsync.

From my original email, I'm getting a good 5 MB/s using raw scp.  But I'm
only seeing 1-2MB/s using rsync and/or rdiff-backup.  So there is clearly
overhead with these two protocols over raw ssh/scp.

Of course scp is still being a bottleneck (compared to 20MB/s capability
on the back-end).  But even if I could get rsync/rdiff-backup up to the
scp-capability of 5MB/s that would help.

I'll take a look at hpn-ssh, but building that for all targets might be a
major PITA.  I might also look at alternate ciphers.

Thanks,

-derek

On Wed, March 30, 2016 3:29 am, Dominic Raferd wrote:

> It doesn't sound like rdiff-backup is the culprit here. You could try
> hpn-ssh https://sourceforge.net/projects/hpnssh/ ?
>
> On 29 March 2016 at 21:03, Derek Atkins <[hidden email]> wrote:
>
>> Hi,
>>
>> On Mon, March 28, 2016 10:56 am, Derek Atkins wrote:
>> > Hi,
>> >
>> > Thank you for taking the time to look at this..
>> >
>> > On Mon, March 28, 2016 10:41 am, Dominic Raferd wrote:
>> >> Is this really your first rdiff-backup to this location? If you have
>> any
>> >> previous rdiff-backup runs to this repository then the situation is
>> >> complicated by rdiff-backup's need to create a new set of reverse
>> diff
>> >> files to be able to regress to previous file contents.
>> >
>> > Yes, this s really the first rdiff-backup to this location.
>> >
>> > A second backup run shortly after the first one completed finished in
>> 55
>> > minutes.
>> >
>> >> What is your /tmp location? rdiff-backup uses this location for some
>> >> operations though not AFAIK for standard backup runs. Still, if /tmp
>> is
>> >> on
>> >> encfs maybe it could be a culprit; you can override rdiff-backup's
>> >> temporary file location with --tempdir and --remote-tempdir.
>> >
>> > If it is truly /tmp then no; /tmp is a ramdisk on the backup server
>> and
>> is
>> > on the root disk on the target server.  Neither are being run through
>> > encfs.
>> >
>> > If, however, you mean the rdiff-backup-data.tmp files, those ARE being
>> run
>> > through encfs.
>> >
>> >> Might also be worth trying --ssh-no-compression.
>> >
>> > I already have "Compression no" set in ~/.ssh/config so I'm not sure
>> what
>> > this would add?
>> >
>> >> Dominic
>> >> http://www.timedicer.co.uk
>> >
>> > -derek
>> >
>> > PS: I'm running a raw rsync command now just to see how it behaves --
>> so
>> > far I'm only seeing about 2MB/s, but it's only been running for 10
>> minutes
>> > or so.
>>
>> My rsync backup just finished.  It copied 202688912 KB in 1599m55.255s
>> so
>> about 2.1MB/s.  Still significantly slower than SCP, but faster than
>> rdiff-backup.
>>
>> The command I ran was:
>>
>> rsync -art -e "ssh -l root -i /dev/null -o Compression=no"
>> root@server:/var/www/ /backups/server
>>
>> :-(
>>
>> -derek
>>
>> >
>> >>
>> >> On 28 March 2016 at 14:37, Derek Atkins <[hidden email]> wrote:
>> >>
>> >>> Just a quick update:  I tried making these changes on both sides and
>> it
>> >>> really didn't make a difference.  Full backup of 202852072 Kbytes
>> >>> required 2267m25.913s (previously it took 2346m57.800s, so it only
>> sped
>> >>> up a factor of 3%.
>> >>>
>> >>> Only thing I have not yet tried is running a raw rsync to see how
>> fast
>> >>> that runs.  I'll do that next.
>> >>>
>> >>> So, back to my orignal question: anyone have any idea how to get
>> >>> initial
>> >>> transfers to run faster (or indeed any significant data transfers)?
>> >>>
>> >>> Thanks,
>> >>>
>> >>> -derek
>> >>>
>> >>> "Derek Atkins" <[hidden email]> writes:
>> >>>
>> >>> > Hi,
>> >>> >
>> >>> > I'm trying to use rdiff-backup to backup a bunch of servers.  One
>> >>> > particular server contains about 160GB of data, but when I try to
>> >>> perform
>> >>> > the rdiff-backup it's saving the data at a measly 1MB/s.
>> >>> >
>> >>> > Here's my configuration:
>> >>> >
>> >>> >   [server] <--ssh--> [backup-server]{encfs} <--nfs--> [freenas]
>> >>> >
>> >>> > I ran a bunch of tests to try to figure out my bottlenecks.
>> >>> >
>> >>> > I ran a bunch of dd tests (using dd if=/dev/zero bs=1M count=1000)
>> on
>> >>> the
>> >>> > backup server.  Going directly to FreeNAS via NFS (bybassing
>> encfs) I
>> >>> get
>> >>> > 50.2MB/s.  If I run dd directly on the backup server (through
>> encfs)
>> >>> I
>> >>> get
>> >>> > 20.1MB/s.  If I go over SSH from the backup server to the target
>> >>> server
>> >>> > and run the dd on the target server, then write to FreeNas through
>> >>> encfs
>> >>> > declines to 7.6MB/s.
>> >>> >
>> >>> > Note that in those SSH tests, however, I forgot to turn off
>> >>> compression.
>> >>> > When I do that, the throughput for the dd test reduced to 6.6BM/s.
>> >>> (Note
>> >>> > that this is running simultaneously with a running rdiff-backup,
>> so
>> >>> it's
>> >>> > possible that they are reducing performance).
>> >>> >
>> >>> > Then I ran an scp test to the same target server; copying about
>> 1.4GB
>> >>> of
>> >>> > photos.  Files ranged in size from 10KB to 5MB.  When run in
>> standard
>> >>> mode
>> >>> > (displaying each file status) I got 4.4MB/s.  Running in quiet
>> mode I
>> >>> get
>> >>> > 5.1MB/s.
>> >>> >
>> >>> > So clearly the bottleneck is in rdiff-backup -- performance (IMHO)
>> >>> should
>> >>> > not be significantly slower than the last dd-over-ssh test.  It
>> >>> appears
>> >>> > rdiff-backup is slowing me down by a factor of 5x throughput
>> versus
>> >>> scp.
>> >>> >
>> >>> > I found a message from Ben from 2005 where he suggests increasing
>> the
>> >>> > blocksize and conn_bufsiz settings in Globals.py:
>> >>> >
>> >>>
>> https://lists.gnu.org/archive/html/rdiff-backup-users/2005-10/msg00062.html
>> >>> >
>> >>> > What he didn't say was whether this needed to be changed on the
>> >>> target
>> >>> > server, the backup server, or both.  Nor do I know if that would
>> >>> actually
>> >>> > help this situation.
>> >>> >
>> >>> > Do you have any ideas?
>> >>> >
>> >>> > Thanks,
>> >>> >
>> >>> > -derek
>> >>> >
>> >>> > PS: According to rpm, both systems are running version 1.2.8.
>> >>>
>> >>> --
>> >>>        Derek Atkins                 617-623-3745
>> >>>        [hidden email]             www.ihtfp.com
>> >>>        Computer and Internet Security Consultant
>> >>>
>> >>> _______________________________________________
>> >>> rdiff-backup-users mailing list at [hidden email]
>> >>> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>> >>> Wiki URL:
>> >>> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>> >>>
>> >>
>> >
>> >
>> > --
>> >        Derek Atkins                 617-623-3745
>> >        [hidden email]             www.ihtfp.com
>> >        Computer and Internet Security Consultant
>> >
>> >
>> > _______________________________________________
>> > rdiff-backup-users mailing list at [hidden email]
>> > https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>> > Wiki URL:
>> > http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>> >
>>
>>
>> --
>>        Derek Atkins                 617-623-3745
>>        [hidden email]             www.ihtfp.com
>>        Computer and Internet Security Consultant
>>
>>
>


--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup performance -- slow operation on initial backup?

Derek Atkins-2
In reply to this post by Greg Troxel
Hey Greg!

On Wed, March 30, 2016 8:13 am, Greg Troxel wrote:

>
> Dominic Raferd <[hidden email]> writes:
>
>> It doesn't sound like rdiff-backup is the culprit here. You could try
>> hpn-ssh https://sourceforge.net/projects/hpnssh/ ?
>
> I would be very surprised if normal people have networks available that
> really need hpn-ssh, and 2 MB/s is not that fast.  urely rsync and
> rdiff-backup are running over ssh, so that should have the same
> transport properties.

Indeed.  Like I said using scp I see 5+ MB/s with the same data set.  Just
using dd over ssh I get 7.   But yes, rsync and rdiff-backup are giving me
the same transport properties.  The fact that it's taking 36-40 hours to
backup a system does not give me confidence in the ability (or timeliness)
to restore!

> I would up the send/receive socket buffers (because it's easy, not
> because I think that's the problem), and watch disk/cpu on both sides,
> and also run netstat to see if data is piling up in the transmit socket
> buffer.

Do you mean within rdiff-backup, or at some other level?

I've already tried increasing the blocksize and conn_blocksize numbers in
Globals.py but didn't see any performance difference.

> FWIW, I used to use rdiff-backup but found it to be nonrobust on
> machines with limited (only a few GB) RAM and hundreds of GB of backup.
> I have switched to bup.

Unfortunately bup is not available on all my target platforms.

Maybe I should consider amanda or bacula?

-derek

--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup performance -- slow operation on initial backup?

Greg Troxel

"Derek Atkins" <[hidden email]> writes:

>> I would up the send/receive socket buffers (because it's easy, not
>> because I think that's the problem), and watch disk/cpu on both sides,
>> and also run netstat to see if data is piling up in the transmit socket
>> buffer.
>
> Do you mean within rdiff-backup, or at some other level?

On NetBSD I mean bumping up these sysctls

net.inet.tcp.sendspace = 131072
net.inet.tcp.recvspace = 131072
net.inet6.tcp6.sendspace = 131072
net.inet6.tcp6.recvspace = 131072

and presumably that's similar on other systems.

But, if your socket buffers aren't full, that's probably not your
problem.

>> FWIW, I used to use rdiff-backup but found it to be nonrobust on
>> machines with limited (only a few GB) RAM and hundreds of GB of backup.
>> I have switched to bup.
>
> Unfortunately bup is not available on all my target platforms.

bup is python with a little C, and thus seems pretty portable.   Where
isn't it working?

There is also attic and borg which are similar to bup.

> Maybe I should consider amanda or bacula?

amanda is basically wrappers around dump and tar.  If you have 50
machines and want to do level 0/1/2 to tape and take tapes offsite, it
works great, after you pay for the LTO tape drive.

Two things to think about:

  do you care about deduplication?  bup does not only per-file but
  within-file deduplication, so if multiple boxes have the same data it
  doesn't take up extra space

  Do you really need to back up all platforms, or could you sync from
  some (Android?) to a machine with more disk and back that up?  I have
  been using syncthing, which seems to be pretty solid, for syncing
  among Android and regular computers (BSD and OS X).  (It is written in
  go so it's not in practice that portable.)

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

signature.asc (186 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup performance -- slow operation on initial backup?

Derek Atkins-2

On Wed, March 30, 2016 12:47 pm, Greg Troxel wrote:

>
> "Derek Atkins" <[hidden email]> writes:
>
>>> I would up the send/receive socket buffers (because it's easy, not
>>> because I think that's the problem), and watch disk/cpu on both sides,
>>> and also run netstat to see if data is piling up in the transmit socket
>>> buffer.
>>
>> Do you mean within rdiff-backup, or at some other level?
>
> On NetBSD I mean bumping up these sysctls
>
> net.inet.tcp.sendspace = 131072
> net.inet.tcp.recvspace = 131072
> net.inet6.tcp6.sendspace = 131072
> net.inet6.tcp6.recvspace = 131072
>
> and presumably that's similar on other systems.

I'll look for the Linux equivalent, however...

> But, if your socket buffers aren't full, that's probably not your
> problem.

... according to repeated netstat the socket buffers are GENERALLY 0,0.  I
did see it get up to about 1000, at one point...  OH, I take that back,
the SEND Queue value was just up to 243816 on the target system....

>>> FWIW, I used to use rdiff-backup but found it to be nonrobust on
>>> machines with limited (only a few GB) RAM and hundreds of GB of backup.
>>> I have switched to bup.
>>
>> Unfortunately bup is not available on all my target platforms.
>
> bup is python with a little C, and thus seems pretty portable.   Where
> isn't it working?

Didn't say it wasn't working.  I said "it is not available", meaning there
is no prepackaged RPM for some of my systems.  I could certainly try to
piece it together, but of course I prefer to use distro-supplied software
wherever possible.  It makes upgrading much easier.

> There is also attic and borg which are similar to bup.
>
>> Maybe I should consider amanda or bacula?
>
> amanda is basically wrappers around dump and tar.  If you have 50
> machines and want to do level 0/1/2 to tape and take tapes offsite, it
> works great, after you pay for the LTO tape drive.

I don't have 50 machines.  I do have ~10-15.  Pretty much all Linux boxes.
 Not using tape; backups are all on spinning media.

> Two things to think about:
>
>   do you care about deduplication?  bup does not only per-file but
>   within-file deduplication, so if multiple boxes have the same data it
>   doesn't take up extra space

No.  At least, not at the block level.  I would care about file-level
dedup, but honestly I only care about that from within one server.  If I
have multiple systems that happen to have the exact same config file I
don't particularly care about dedupping that.  I just don't want a daily
copy of /etc/passwd :)

>   Do you really need to back up all platforms, or could you sync from
>   some (Android?) to a machine with more disk and back that up?  I have
>   been using syncthing, which seems to be pretty solid, for syncing
>   among Android and regular computers (BSD and OS X).  (It is written in
>   go so it's not in practice that portable.)

Pretty much all the systems I want to backup are Linux, but different
vintages and such.

Android is different and I back that up differently.  I'm just trying to
maximize performance.  I've got a GigE network and relatively plenty of
encryption performance; I'd like to leverage that in my backup (and
restore) operations.

Thanks,

-derek

--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup performance -- slow operation on initial backup?

Greg Troxel

"Derek Atkins" <[hidden email]> writes:

> ... according to repeated netstat the socket buffers are GENERALLY 0,0.  I
> did see it get up to about 1000, at one point...  OH, I take that back,
> the SEND Queue value was just up to 243816 on the target system....

If it bursts and comes back, it's probably fine, and many systems
dynamically size the buffers.  This is unlikely to be the issue in a LAN
environment - more like when you have 100 Mb/s pipes and 70ms latencies.

>>> Unfortunately bup is not available on all my target platforms.
>>
>> bup is python with a little C, and thus seems pretty portable.   Where
>> isn't it working?
>
> Didn't say it wasn't working.  I said "it is not available", meaning there
> is no prepackaged RPM for some of my systems.  I could certainly try to
> piece it together, but of course I prefer to use distro-supplied software
> wherever possible.  It makes upgrading much easier.

I see.  (It's in pkgsrc, which builds pretty much everywhere, but that
gets you into a second packaging system.)

>> amanda is basically wrappers around dump and tar.  If you have 50
>> machines and want to do level 0/1/2 to tape and take tapes offsite, it
>> works great, after you pay for the LTO tape drive.
>
> I don't have 50 machines.  I do have ~10-15.  Pretty much all Linux boxes.
>  Not using tape; backups are all on spinning media.

amanda can cope with that (files on disk that are virtual tapes), but
I'm not at all sure it's the right approach for you.

Good luck figuring out where your bottleneck is.

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

signature.asc (186 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup performance -- slow operation on initial backup?

Patrik Dufresne-2
I you want to spent some time debuging this this issue. It might be helpful to run a profiler.

python -m cProfile -o data.prof /usr/bin/rdiff-backup source destination

Then send us the data.prof. I will use `snakeviz data.prof` to visualize the data.

--
Patrik Dufresne Service Logiciel inc.
http://www.patrikdufresne.com/
514-971-6442
1-114 rue des Hautbois,
St-Colomban, QC J5K 2H6

On Wed, Mar 30, 2016 at 1:30 PM, Greg Troxel <[hidden email]> wrote:

"Derek Atkins" <[hidden email]> writes:

> ... according to repeated netstat the socket buffers are GENERALLY 0,0.  I
> did see it get up to about 1000, at one point...  OH, I take that back,
> the SEND Queue value was just up to 243816 on the target system....

If it bursts and comes back, it's probably fine, and many systems
dynamically size the buffers.  This is unlikely to be the issue in a LAN
environment - more like when you have 100 Mb/s pipes and 70ms latencies.

>>> Unfortunately bup is not available on all my target platforms.
>>
>> bup is python with a little C, and thus seems pretty portable.   Where
>> isn't it working?
>
> Didn't say it wasn't working.  I said "it is not available", meaning there
> is no prepackaged RPM for some of my systems.  I could certainly try to
> piece it together, but of course I prefer to use distro-supplied software
> wherever possible.  It makes upgrading much easier.

I see.  (It's in pkgsrc, which builds pretty much everywhere, but that
gets you into a second packaging system.)

>> amanda is basically wrappers around dump and tar.  If you have 50
>> machines and want to do level 0/1/2 to tape and take tapes offsite, it
>> works great, after you pay for the LTO tape drive.
>
> I don't have 50 machines.  I do have ~10-15.  Pretty much all Linux boxes.
>  Not using tape; backups are all on spinning media.

amanda can cope with that (files on disk that are virtual tapes), but
I'm not at all sure it's the right approach for you.

Good luck figuring out where your bottleneck is.

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: rdiff-backup performance -- slow operation on initial backup?

Derek Atkins-2
Should I be profiling on the backup server or the target/client (the
system being backed up)?
-derek

On Wed, March 30, 2016 1:47 pm, Patrik Dufresne wrote:

> I you want to spent some time debuging this this issue. It might be
> helpful
> to run a profiler.
>
> python -m cProfile -o data.prof /usr/bin/rdiff-backup source destination
>
> Then send us the data.prof. I will use `snakeviz data.prof` to visualize
> the data.
>
> --
> Patrik Dufresne Service Logiciel inc.
> http://www.patrikdufresne.com <http://patrikdufresne.com/>/
> 514-971-6442
> 1-114 rue des Hautbois,
> St-Colomban, QC J5K 2H6
>
> On Wed, Mar 30, 2016 at 1:30 PM, Greg Troxel <[hidden email]> wrote:
>
>>
>> "Derek Atkins" <[hidden email]> writes:
>>
>> > ... according to repeated netstat the socket buffers are GENERALLY
>> 0,0.
>> I
>> > did see it get up to about 1000, at one point...  OH, I take that
>> back,
>> > the SEND Queue value was just up to 243816 on the target system....
>>
>> If it bursts and comes back, it's probably fine, and many systems
>> dynamically size the buffers.  This is unlikely to be the issue in a LAN
>> environment - more like when you have 100 Mb/s pipes and 70ms latencies.
>>
>> >>> Unfortunately bup is not available on all my target platforms.
>> >>
>> >> bup is python with a little C, and thus seems pretty portable.
>> Where
>> >> isn't it working?
>> >
>> > Didn't say it wasn't working.  I said "it is not available", meaning
>> there
>> > is no prepackaged RPM for some of my systems.  I could certainly try
>> to
>> > piece it together, but of course I prefer to use distro-supplied
>> software
>> > wherever possible.  It makes upgrading much easier.
>>
>> I see.  (It's in pkgsrc, which builds pretty much everywhere, but that
>> gets you into a second packaging system.)
>>
>> >> amanda is basically wrappers around dump and tar.  If you have 50
>> >> machines and want to do level 0/1/2 to tape and take tapes offsite,
>> it
>> >> works great, after you pay for the LTO tape drive.
>> >
>> > I don't have 50 machines.  I do have ~10-15.  Pretty much all Linux
>> boxes.
>> >  Not using tape; backups are all on spinning media.
>>
>> amanda can cope with that (files on disk that are virtual tapes), but
>> I'm not at all sure it's the right approach for you.
>>
>> Good luck figuring out where your bottleneck is.
>>
>> _______________________________________________
>> rdiff-backup-users mailing list at [hidden email]
>> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>> Wiki URL:
>> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>>
>


--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki