Scalability of duplicity

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Scalability of duplicity

duplicity-talk mailing list
I'm using duplicity via duply and have run into a problem with one of our duply profiles used for postgres database backups. The backups are binary files (uses pgdump --format=custom). The backup duply points to is about 90GB. The duply runs started OK, then hung on 7/10/2017. The only recourse was to remove the files created by duplicity and the lock file. It hung again on 9/16/2017. By hung I mean that the duply and duplicity processes were still running but top didn't show any activity. On 9/20/2017 I removed the 54298 duplicity files. I didn't count the number of files removed on 7/10/2017.

Duply is set up to do incremental backups every day (the pgdump cron is run daily an hour eariler) with a full backup done every week. The retention period is a month.

Now for my questions:

  Has anyone run duply/duplicity with 90GB of binary files?

  The next time this happens, what data should I collect to try and diagnose this problem?

Tom Ekberg
Senior Computer Specialist, Lab Medicine
University of Washington Medical Center
1959 NE Pacific St, MS 357110
Seattle WA 98195
work: (206) 598-8544
email: [hidden email]




_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Scalability of duplicity

duplicity-talk mailing list
Hej,

>
> Now for my questions:
>
>  Has anyone run duply/duplicity with 90GB of binary files?

yes, we do run it with backup sets >300GB, haven't had any issues with stalling backups in the last few years as far as I remember. Running on FreeBSD, though.
Which version of duplicity are you running?

Cheers,

Philip
_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Scalability of duplicity

duplicity-talk mailing list
Philip,

Sorry about the scant environment details. I'm running duplicity 0.7.06 on Ubuntu 16.04.3 with 63GB of RAM and 12TB of disk space. The lscpu command says it has 48 CPUs. Big honkin' server. The command 'gpg --version' displays gpg (GnuPG) 1.4.20. The destination host where duplicity puts the backup files is running Ubuntu 16.04.3 with 8GB of RAM and 4.9TB of disk space. The lscpu command says it has 4 CPUs. 31 days of backups should need about 2.7TB of storage without compression and encryption.

Read the duplicity change log down to the 2015-12-07 line that says 'Prep for 0.7.06'. Didn't see anything specifically there (I didn't look at the bug DB) describing a fix for this problem. I could try running the latest duplicity, 0.7.14, to see if the problem goes away.

What version of duplicity are you running?

Tom Ekberg
Senior Computer Specialist, Lab Medicine
University of Washington Medical Center
1959 NE Pacific St, MS 357110
Seattle WA 98195
work: (206) 598-8544
email: [hidden email]

On Thu, 5 Oct 2017, Philip Jocks wrote:

> Date: Thu, 5 Oct 2017 11:06:31 +0200
> From: Philip Jocks <[hidden email]>
> To: Discussion about duplicity backup <[hidden email]>
> Cc: Tom Ekberg <[hidden email]>
> Subject: Re: [Duplicity-talk] Scalability of duplicity
>
> Hej,
>
>>
>> Now for my questions:
>>
>>  Has anyone run duply/duplicity with 90GB of binary files?
>
> yes, we do run it with backup sets >300GB, haven't had any issues with stalling backups in the last few years as far as I remember. Running on FreeBSD, though.
> Which version of duplicity are you running?
>
> Cheers,
>
> Philip


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Scalability of duplicity

duplicity-talk mailing list
Hi,


Sorry about the scant environment details. I'm running duplicity 0.7.06 on Ubuntu 16.04.3 with 63GB of RAM and 12TB of disk space. The lscpu command says it has 48 CPUs. Big honkin' server. The command 'gpg --version' displays gpg (GnuPG) 1.4.20. The destination host where duplicity puts the backup files is running Ubuntu 16.04.3 with 8GB of RAM and 4.9TB of disk space. The lscpu command says it has 4 CPUs. 31 days of backups should need about 2.7TB of storage without compression and encryption.

If there is no CPU activity and Duplicity is still running it might be that the underlying network  protocol has stalled. Are you moving files over SFTP or SCP? If the underlying network hangs and does not properly timeout then it might cause symptoms like this.

You can use strace system command the check what a process is waiting if it does nothing.

Hope this gives clues,
-Mikko
 

Read the duplicity change log down to the 2015-12-07 line that says 'Prep for 0.7.06'. Didn't see anything specifically there (I didn't look at the bug DB) describing a fix for this problem. I could try running the latest duplicity, 0.7.14, to see if the problem goes away.

What version of duplicity are you running?

Tom Ekberg
Senior Computer Specialist, Lab Medicine
University of Washington Medical Center
1959 NE Pacific St, MS 357110
Seattle WA 98195
work: <a href="tel:%28206%29%20598-8544" value="+12065988544" target="_blank">(206) 598-8544
email: [hidden email]

On Thu, 5 Oct 2017, Philip Jocks wrote:

Date: Thu, 5 Oct 2017 11:06:31 +0200
From: Philip Jocks <[hidden email]>
To: Discussion about duplicity backup <[hidden email]>
Cc: Tom Ekberg <[hidden email]>
Subject: Re: [Duplicity-talk] Scalability of duplicity


Hej,


Now for my questions:

 Has anyone run duply/duplicity with 90GB of binary files?

yes, we do run it with backup sets >300GB, haven't had any issues with stalling backups in the last few years as far as I remember. Running on FreeBSD, though.
Which version of duplicity are you running?

Cheers,

Philip


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk



--

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Scalability of duplicity

duplicity-talk mailing list
Mikko,

Thanks for the suggestion. Duplicity is using scp to transfer the files.

Tom Ekberg
Senior Computer Specialist, Lab Medicine
University of Washington Medical Center
1959 NE Pacific St, MS 357110
Seattle WA 98195
work: (206) 598-8544
email: [hidden email]

On Thu, 5 Oct 2017, Mikko Ohtamaa wrote:

> Date: Thu, 5 Oct 2017 20:19:23 +0300
> From: Mikko Ohtamaa <[hidden email]>
> To: Discussion about duplicity backup <[hidden email]>
> Cc: Philip Jocks <[hidden email]>, Tom Ekberg <[hidden email]>
> Subject: Re: [Duplicity-talk] Scalability of duplicity
>
> Hi,
>
>
>       Sorry about the scant environment details. I'm running duplicity 0.7.06 on Ubuntu 16.04.3 with 63GB of RAM and 12TB of disk space. The lscpu command says
>       it has 48 CPUs. Big honkin' server. The command 'gpg --version' displays gpg (GnuPG) 1.4.20. The destination host where duplicity puts the backup files is
>       running Ubuntu 16.04.3 with 8GB of RAM and 4.9TB of disk space. The lscpu command says it has 4 CPUs. 31 days of backups should need about 2.7TB of storage
>       without compression and encryption.
>
>
> If there is no CPU activity and Duplicity is still running it might be that the underlying network  protocol has stalled. Are you moving files over SFTP or SCP? If the
> underlying network hangs and does not properly timeout then it might cause symptoms like this.
>
> You can use strace system command the check what a process is waiting if it does nothing.
>
> Hope this gives clues,
> -Mikko
>  
>
>       Read the duplicity change log down to the 2015-12-07 line that says 'Prep for 0.7.06'. Didn't see anything specifically there (I didn't look at the bug DB)
>       describing a fix for this problem. I could try running the latest duplicity, 0.7.14, to see if the problem goes away.
>
>       What version of duplicity are you running?
>
>       Tom Ekberg
>       Senior Computer Specialist, Lab Medicine
>       University of Washington Medical Center
>       1959 NE Pacific St, MS 357110
>       Seattle WA 98195
>       work: (206) 598-8544
>       email: [hidden email]
>
>       On Thu, 5 Oct 2017, Philip Jocks wrote:
>
>             Date: Thu, 5 Oct 2017 11:06:31 +0200
>             From: Philip Jocks <[hidden email]>
>             To: Discussion about duplicity backup <[hidden email]>
>             Cc: Tom Ekberg <[hidden email]>
>             Subject: Re: [Duplicity-talk] Scalability of duplicity
>
>             Hej,
>
>
>                   Now for my questions:
>
>                    Has anyone run duply/duplicity with 90GB of binary files?
>
>
>             yes, we do run it with backup sets >300GB, haven't had any issues with stalling backups in the last few years as far as I remember. Running on
>             FreeBSD, though.
>             Which version of duplicity are you running?
>
>             Cheers,
>
>             Philip
>
>
>
> _______________________________________________
> Duplicity-talk mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>
>
>
>
> --
> Mikko Ohtamaa
> http://opensourcehacker.com
> http://twitter.com/moo9000
>
>
>

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk