Increasing number of asynchronously created temp files?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Increasing number of asynchronously created temp files?

duplicity-talk mailing list
Hello,

I just started using duplicity (0.7.15 from epel) and it seems to be
working fine. I'm using the asynchronous-upload option to make better
use of the CPU and network resources. I can see the two temporary gpg
files of volsize being used.

The stuff I'm backing up varies, there are collections of large files,
but also some areas of the file system have a lot of small files. When
the backup process comes across areas of small files, the upload
"pipeline" stalls. Meaning that the archive file upload is happening
faster than the archive file creation. Thus changing the backup process
from being mostly network speed bound to CPU(/disk) bound when it hits
these small files.

I tried to look for a setting in the duplicity man page for increasing
the number of temporary archive files to use from two, but couldn't find
one. Would it be possible to add such a user-tunable feature? This way
when duplicity is going through sections of the filesystem containing
large files it could build up a bigger queue of upload work. This queue
could then keep the upload process better occupied when the archiving
part runs into those areas of smaller files.

   Tapio

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Increasing number of asynchronously created temp files?

duplicity-talk mailing list
For now, the limit is one async thread.  We've not had time to take it further.

...Ken


On Sun, Jan 28, 2018 at 4:21 AM, Tapio Sokura via Duplicity-talk <[hidden email]> wrote:
Hello,

I just started using duplicity (0.7.15 from epel) and it seems to be working fine. I'm using the asynchronous-upload option to make better use of the CPU and network resources. I can see the two temporary gpg files of volsize being used.

The stuff I'm backing up varies, there are collections of large files, but also some areas of the file system have a lot of small files. When the backup process comes across areas of small files, the upload "pipeline" stalls. Meaning that the archive file upload is happening faster than the archive file creation. Thus changing the backup process from being mostly network speed bound to CPU(/disk) bound when it hits these small files.

I tried to look for a setting in the duplicity man page for increasing the number of temporary archive files to use from two, but couldn't find one. Would it be possible to add such a user-tunable feature? This way when duplicity is going through sections of the filesystem containing large files it could build up a bigger queue of upload work. This queue could then keep the upload process better occupied when the archiving part runs into those areas of smaller files.

  Tapio

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Increasing number of asynchronously created temp files?

duplicity-talk mailing list
hey Tapio,

Ken is correct and i'd like to add if you want to contribute/sponsor this feature though you will be more than welcome.

..ede/duply.net

On 1/28/2018 12:43, Kenneth Loafman via Duplicity-talk wrote:

> For now, the limit is one async thread.  We've not had time to take it further.
>
> ...Ken
>
>
> On Sun, Jan 28, 2018 at 4:21 AM, Tapio Sokura via Duplicity-talk <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Hello,
>
>     I just started using duplicity (0.7.15 from epel) and it seems to be working fine. I'm using the asynchronous-upload option to make better use of the CPU and network resources. I can see the two temporary gpg files of volsize being used.
>
>     The stuff I'm backing up varies, there are collections of large files, but also some areas of the file system have a lot of small files. When the backup process comes across areas of small files, the upload "pipeline" stalls. Meaning that the archive file upload is happening faster than the archive file creation. Thus changing the backup process from being mostly network speed bound to CPU(/disk) bound when it hits these small files.
>
>     I tried to look for a setting in the duplicity man page for increasing the number of temporary archive files to use from two, but couldn't find one. Would it be possible to add such a user-tunable feature? This way when duplicity is going through sections of the filesystem containing large files it could build up a bigger queue of upload work. This queue could then keep the upload process better occupied when the archiving part runs into those areas of smaller files.
>
>       Tapio
>
>     _______________________________________________
>     Duplicity-talk mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>
>
>
>
>
> _______________________________________________
> Duplicity-talk mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk