Duplicity is slow on many small files

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Duplicity is slow on many small files

duplicity-talk mailing list
Hi!

I'm using duplicity to backup my webhosting server. My www directory is
17G, and it contains many many small files. A full backup to amazon S3
is >1 day. Is this a normal running time for a full backup? Creating tar
from the www folder is 12min. Is it a good idea to make a tar before
backup? Will it speed up the backup process? Will duplicity store only
modifications of the tar? Or have you any other idea to speeding up the
backup?

Ty

Best regards,
   Laszlo

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Duplicity is slow on many small files

duplicity-talk mailing list
On 07.02.2017 11:46, Fazekas László via Duplicity-talk wrote:
> Hi!
>
> I'm using duplicity to backup my webhosting server. My www directory is 17G, and it contains many many small files. A full backup to amazon S3 is >1 day. Is this a normal running time for a full backup?

what's your duplicity version?

try a test backup to a local file:// target and see how long that takes in comparison.

>Creating tar from the www folder is 12min. Is it a good idea to make a tar before backup?

only if you are willing to untar manually on restore of course

>Will it speed up the backup process?

if the small files are the issue, then probably yes.

>Will duplicity store only modifications of the tar?

yes, but as we are using librsync it will look at it chunk by chunk and if say in the beginning a file got bigger all content afterwards is offset and will be regarded as changed.

>Or have you any other idea to speeding up the backup?

primarily using the latest duplicity from the website. there was an issue some month ago that slowed down backups.

..ede/duply.net

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Duplicity is slow on many small files

duplicity-talk mailing list
solved ...

I've done a test with the file:// protocol, and duplicity was really
fast, but when I've tried to upload to s3 through awscli, the uload
seepd was only ~100K/sec. This is why the backup was slow. I don't know
it is a problem with my connection or it is the normal S3 upload speed,
but not a duplicity issue.

Ty for your help

2017-02-07 11:56 keltezéssel, edgar.soldin--- via Duplicity-talk írta:

> On 07.02.2017 11:46, Fazekas László via Duplicity-talk wrote:
>> Hi!
>>
>> I'm using duplicity to backup my webhosting server. My www directory is 17G, and it contains many many small files. A full backup to amazon S3 is >1 day. Is this a normal running time for a full backup?
> what's your duplicity version?
>
> try a test backup to a local file:// target and see how long that takes in comparison.
>
>> Creating tar from the www folder is 12min. Is it a good idea to make a tar before backup?
> only if you are willing to untar manually on restore of course
>
>> Will it speed up the backup process?
> if the small files are the issue, then probably yes.
>
>> Will duplicity store only modifications of the tar?
> yes, but as we are using librsync it will look at it chunk by chunk and if say in the beginning a file got bigger all content afterwards is offset and will be regarded as changed.
>
>> Or have you any other idea to speeding up the backup?
> primarily using the latest duplicity from the website. there was an issue some month ago that slowed down backups.
>
> ..ede/duply.net
>
> _______________________________________________
> Duplicity-talk mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk