how to tune duplicity for bigger files

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

how to tune duplicity for bigger files

duplicity-talk mailing list
Hello everybody,

My duplicity is really slow with bigger files and runs on 100% CPU load
for ages. I am doing backup of large files from 1GB to 100GB encrypted.

What can I do to speed this up?

Can I change the code how it calculate block size?
/usr/lib/python2.7/dist-packages/duplicity/diffdir.py

Are there any options that I can tune?
--max-blocksize=number

Kind regards,

Jelle de Jong

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: how to tune duplicity for bigger files

duplicity-talk mailing list
Have you tried the newer version?

There are multiple options both stable and daily:


NOTE: UNinstall duplicity first if it was installed via the distribution repository.  For Ubuntu that would be "sudo apt-get purge duplicity".

On Mon, Mar 30, 2020 at 2:36 PM Jelle de Jong via Duplicity-talk <[hidden email]> wrote:
Hello everybody,

My duplicity is really slow with bigger files and runs on 100% CPU load
for ages. I am doing backup of large files from 1GB to 100GB encrypted.

What can I do to speed this up?

Can I change the code how it calculate block size?
/usr/lib/python2.7/dist-packages/duplicity/diffdir.py

Are there any options that I can tune?
--max-blocksize=number

Kind regards,

Jelle de Jong

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: how to tune duplicity for bigger files

duplicity-talk mailing list
Hello everybody,

Yes I tried newer versions, I am still getting constant at X5680 @
3.33GHz and very slow 2MB/s upload. Works good with lots of small files
though, but with big files it comes to a halt.

Any way I can I do to let duplicity work well with multi GB files?

# duplicity --version
duplicity 0.8.12

# gpg --version
gpg (GnuPG) 2.2.12

# python3 --version
Python 3.7.3

Kind regards,

Jelle de Jong

On 2020-03-30 21:46, Kenneth Loafman wrote:

> Have you tried the newer version?
>
> There are multiple options both stable and daily:
>
>   *
>     Stable tarball install - https://launchpad.net/duplicity/+download
>   *
>     Daily duplicity builds -
>     https://launchpad.net/~duplicity-team/+archive/ubuntu/daily-dev-trunk
>   *
>     Stable snap builds - “sudo snap install duplicity —classic"
>   *
>     Latest snap builds - “sudo snap install duplicity —classic —edge"
>   *
>     Latest pip builds - “sudo pip install duplicity"
>
>
> NOTE: UNinstall duplicity first if it was installed via the distribution
> repository. For Ubuntu that would be "sudo apt-get purge duplicity".
>
> On Mon, Mar 30, 2020 at 2:36 PM Jelle de Jong via Duplicity-talk
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Hello everybody,
>
>     My duplicity is really slow with bigger files and runs on 100% CPU load
>     for ages. I am doing backup of large files from 1GB to 100GB encrypted.
>
>     What can I do to speed this up?
>
>     Can I change the code how it calculate block size?
>     /usr/lib/python2.7/dist-packages/duplicity/diffdir.py
>
>     Are there any options that I can tune?
>     --max-blocksize=number
>
>     Kind regards,
>
>     Jelle de Jong
>
>     _______________________________________________
>     Duplicity-talk mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: how to tune duplicity for bigger files

duplicity-talk mailing list
might be a bug. maybe somewhere memory is not released until the file is closed and hogs memory until the system starts paging. can you monitor memory usage development during such a slow backup?

..ede/duply.net

On 06.04.2020 19:57, Jelle de Jong via Duplicity-talk wrote:

> Hello everybody,
>
> Yes I tried newer versions, I am still getting constant at X5680 @ 3.33GHz and very slow 2MB/s upload. Works good with lots of small files though, but with big files it comes to a halt.
>
> Any way I can I do to let duplicity work well with multi GB files?
>
> # duplicity --version
> duplicity 0.8.12
>
> # gpg --version
> gpg (GnuPG) 2.2.12
>
> # python3 --version
> Python 3.7.3
>
> Kind regards,
>
> Jelle de Jong
>
> On 2020-03-30 21:46, Kenneth Loafman wrote:
>> Have you tried the newer version?
>>
>> There are multiple options both stable and daily:
>>
>>   *
>>     Stable tarball install - https://launchpad.net/duplicity/+download
>>   *
>>     Daily duplicity builds -
>>     https://launchpad.net/~duplicity-team/+archive/ubuntu/daily-dev-trunk
>>   *
>>     Stable snap builds - “sudo snap install duplicity —classic"
>>   *
>>     Latest snap builds - “sudo snap install duplicity —classic —edge"
>>   *
>>     Latest pip builds - “sudo pip install duplicity"
>>
>> NOTE: UNinstall duplicity first if it was installed via the distribution repository. For Ubuntu that would be "sudo apt-get purge duplicity".
>>
>> On Mon, Mar 30, 2020 at 2:36 PM Jelle de Jong via Duplicity-talk <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>     Hello everybody,
>>
>>     My duplicity is really slow with bigger files and runs on 100% CPU load
>>     for ages. I am doing backup of large files from 1GB to 100GB encrypted.
>>
>>     What can I do to speed this up?
>>
>>     Can I change the code how it calculate block size?
>>     /usr/lib/python2.7/dist-packages/duplicity/diffdir.py
>>
>>     Are there any options that I can tune?
>>     --max-blocksize=number
>>
>>     Kind regards,
>>
>>     Jelle de Jong
>>
>>     _______________________________________________
>>     Duplicity-talk mailing list
>>     [hidden email] <mailto:[hidden email]>
>>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>
> _______________________________________________
> Duplicity-talk mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: how to tune duplicity for bigger files

duplicity-talk mailing list
Hi,

ede is quite right about memory usage.  duplicity keeps the signatures of the file in process in memory, so it could be filling memory.

A suggestion would be to set --max-blocksize=8192, 4x the current default, and move it higher or lower depending on need.

The man page also has info that may help.

...Ken


On Mon, Apr 6, 2020 at 1:44 PM edgar.soldin--- via Duplicity-talk <[hidden email]> wrote:
might be a bug. maybe somewhere memory is not released until the file is closed and hogs memory until the system starts paging. can you monitor memory usage development during such a slow backup?

..ede/duply.net

On 06.04.2020 19:57, Jelle de Jong via Duplicity-talk wrote:
> Hello everybody,
>
> Yes I tried newer versions, I am still getting constant at X5680 @ 3.33GHz and very slow 2MB/s upload. Works good with lots of small files though, but with big files it comes to a halt.
>
> Any way I can I do to let duplicity work well with multi GB files?
>
> # duplicity --version
> duplicity 0.8.12
>
> # gpg --version
> gpg (GnuPG) 2.2.12
>
> # python3 --version
> Python 3.7.3
>
> Kind regards,
>
> Jelle de Jong
>
> On 2020-03-30 21:46, Kenneth Loafman wrote:
>> Have you tried the newer version?
>>
>> There are multiple options both stable and daily:
>>
>>   *
>>     Stable tarball install - https://launchpad.net/duplicity/+download
>>   *
>>     Daily duplicity builds -
>>     https://launchpad.net/~duplicity-team/+archive/ubuntu/daily-dev-trunk
>>   *
>>     Stable snap builds - “sudo snap install duplicity —classic"
>>   *
>>     Latest snap builds - “sudo snap install duplicity —classic —edge"
>>   *
>>     Latest pip builds - “sudo pip install duplicity"
>>
>> NOTE: UNinstall duplicity first if it was installed via the distribution repository. For Ubuntu that would be "sudo apt-get purge duplicity".
>>
>> On Mon, Mar 30, 2020 at 2:36 PM Jelle de Jong via Duplicity-talk <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>     Hello everybody,
>>
>>     My duplicity is really slow with bigger files and runs on 100% CPU load
>>     for ages. I am doing backup of large files from 1GB to 100GB encrypted.
>>
>>     What can I do to speed this up?
>>
>>     Can I change the code how it calculate block size?
>>     /usr/lib/python2.7/dist-packages/duplicity/diffdir.py
>>
>>     Are there any options that I can tune?
>>     --max-blocksize=number
>>
>>     Kind regards,
>>
>>     Jelle de Jong
>>
>>     _______________________________________________
>>     Duplicity-talk mailing list
>>     [hidden email] <mailto:[hidden email]>
>>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>
> _______________________________________________
> Duplicity-talk mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Does Duplicity Remove files?

duplicity-talk mailing list
Hello,

I've got a simple question that I can't seem to find an answer to.

Does duplicity remove files from the backup that have been removed from
the source?

I'm running a simple duplicity shell script that backs up files to a
backblaze b2 server.

I'm wondering if deleted files from the source (being backed up) are
removed from the backup on the b2 server by duplicity?

I understand that files wouldn't be removed from the full backup but
would a record of that file removal be included in the differential so
that when I restore everything, the deleted files would not be restored?

thanks,

Jared





_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Does Duplicity Remove files?

duplicity-talk mailing list
On Wed, 8 Apr 2020, Jared Larsen via Duplicity-talk wrote:

> I understand that files wouldn't be removed from the full backup but
> would a record of that file removal be included in the differential so
> that when I restore everything, the deleted files would not be restored?

Yes, the incremental records that the file has been deleted, and if you
restore it, the file will not be restored.  You can still restore them
from an older backup using --restore-time.

If you run the backup with a sufficiently high -v level, you'll see
deleted files indicated with a D in the output.

--
Nate Eldredge
[hidden email]


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk