Hanging just before completion

Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Hanging just before completion

duplicity-talk mailing list
Hi All,

Duplicity is a great program and I appreciate this community and the development process.

Several duplicity jobs have hung after putting the final archive file up using the rclone backend.  The total archive size has been >=30TB.  The manifest and sigtar files are complete, but are left in the archive-dir with the .part suffix.  I've tried to look into this with gdb but the process forking makes that difficult.

Debug logging shows this as the point where it hangs:


DEBUG 1

. AsyncScheduler: active workers = 0



Centos Linux tethys.atmos.ucla.edu 3.10.0-1127.19.1.el7.x86_64
python 3.6.8 (virtual env)
duplicity-0.8.16
Options: --asynchronous-upload --no-compression --no-encryption --volsize 20000

I've been able to save the jobs by manually renaming and transferring the manifest and sigtar.  

Best regards,
Cody

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Hanging just before completion

duplicity-talk mailing list
Hi Cody,

What are the sizes of the manifest and sigtar files?  They both get uploaded last and take a while.  Check for network activity.

...Thanks,
...Ken
 

On Wed, Oct 14, 2020 at 4:42 PM Cody Ashe-McNalley via Duplicity-talk <[hidden email]> wrote:
Hi All,

Duplicity is a great program and I appreciate this community and the development process.

Several duplicity jobs have hung after putting the final archive file up using the rclone backend.  The total archive size has been >=30TB.  The manifest and sigtar files are complete, but are left in the archive-dir with the .part suffix.  I've tried to look into this with gdb but the process forking makes that difficult.

Debug logging shows this as the point where it hangs:


DEBUG 1

. AsyncScheduler: active workers = 0



Centos Linux tethys.atmos.ucla.edu 3.10.0-1127.19.1.el7.x86_64
python 3.6.8 (virtual env)
duplicity-0.8.16
Options: --asynchronous-upload --no-compression --no-encryption --volsize 20000

I've been able to save the jobs by manually renaming and transferring the manifest and sigtar.  

Best regards,
Cody
_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Hanging just before completion

duplicity-talk mailing list
Hi Ken,

The hanging seems to happen before upload would take place.  The sigtar files are huge, sometimes almost a terabyte.  But they will sit there for weeks with no network or process activity.  It only takes me a few hours to rename, gzip, and upload them.

It seems related to finishing the async process forking loop, but perhaps that does finish and it hangs immediately after.  The final difftar is uploaded.

Many thanks,
Cody



On Thu, Oct 15, 2020 at 6:36 AM Kenneth Loafman via Duplicity-talk <[hidden email]> wrote:
Hi Cody,

What are the sizes of the manifest and sigtar files?  They both get uploaded last and take a while.  Check for network activity.

...Thanks,
...Ken
 

On Wed, Oct 14, 2020 at 4:42 PM Cody Ashe-McNalley via Duplicity-talk <[hidden email]> wrote:
Hi All,

Duplicity is a great program and I appreciate this community and the development process.

Several duplicity jobs have hung after putting the final archive file up using the rclone backend.  The total archive size has been >=30TB.  The manifest and sigtar files are complete, but are left in the archive-dir with the .part suffix.  I've tried to look into this with gdb but the process forking makes that difficult.

Debug logging shows this as the point where it hangs:


DEBUG 1

. AsyncScheduler: active workers = 0



Centos Linux tethys.atmos.ucla.edu 3.10.0-1127.19.1.el7.x86_64
python 3.6.8 (virtual env)
duplicity-0.8.16
Options: --asynchronous-upload --no-compression --no-encryption --volsize 20000

I've been able to save the jobs by manually renaming and transferring the manifest and sigtar.  

Best regards,
Cody
_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Hanging just before completion

duplicity-talk mailing list
Hi Cody,

I'm having trouble reproducing this.

What kind of target schema do you use?  Is it the file:// schema or something else?  Please list the full command line.

I'd like a log file, please. This is going big and I only need a portion. Please run with -v9, redirect to a log file, then run:

$ head -n 200 < log > log.head
$ tail -n 200 < log > log.tail

Attach, don't copy/paste, log.head and log.tail to your reply.

...Thanks,
...Ken

On Thu, Oct 15, 2020 at 11:56 AM Cody Ashe-McNalley via Duplicity-talk <[hidden email]> wrote:
Hi Ken,

The hanging seems to happen before upload would take place.  The sigtar files are huge, sometimes almost a terabyte.  But they will sit there for weeks with no network or process activity.  It only takes me a few hours to rename, gzip, and upload them.

It seems related to finishing the async process forking loop, but perhaps that does finish and it hangs immediately after.  The final difftar is uploaded.

Many thanks,
Cody



On Thu, Oct 15, 2020 at 6:36 AM Kenneth Loafman via Duplicity-talk <[hidden email]> wrote:
Hi Cody,

What are the sizes of the manifest and sigtar files?  They both get uploaded last and take a while.  Check for network activity.

...Thanks,
...Ken
 

On Wed, Oct 14, 2020 at 4:42 PM Cody Ashe-McNalley via Duplicity-talk <[hidden email]> wrote:
Hi All,

Duplicity is a great program and I appreciate this community and the development process.

Several duplicity jobs have hung after putting the final archive file up using the rclone backend.  The total archive size has been >=30TB.  The manifest and sigtar files are complete, but are left in the archive-dir with the .part suffix.  I've tried to look into this with gdb but the process forking makes that difficult.

Debug logging shows this as the point where it hangs:


DEBUG 1

. AsyncScheduler: active workers = 0



Centos Linux tethys.atmos.ucla.edu 3.10.0-1127.19.1.el7.x86_64
python 3.6.8 (virtual env)
duplicity-0.8.16
Options: --asynchronous-upload --no-compression --no-encryption --volsize 20000

I've been able to save the jobs by manually renaming and transferring the manifest and sigtar.  

Best regards,
Cody
_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk