large upload "ETA stalled"?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

large upload "ETA stalled"?

duplicity-talk mailing list
I'm doing a big backup. It'll take days/weeks to complete. With --progress turned on, I see this:

0.0KB 15:00:22 [0.0B/s] [>                                        ] 0% ETA Stalled!


I know it isn't stalled because I can see the volume size on the remote end getting bigger. Is this to be expected?


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: large upload "ETA stalled"?

duplicity-talk mailing list
Hi,

I've used this chance to explore the source code and it's seems that reporting progress is the "responsibility" of the backend. The S3 or Dropbox backends, for example, report progress but some (most?) don't. You haven't mentioned the version of duplicity you are using but it should apply.

In any case, the progress tracker uses a 5s window to determine if the upload is stalled. So, if the backend does not report progress you will always get "stalled". For S3 you can "stall" if the upload rate is bellow 25KB/s and for Dropbox you may "stall" if the upload rate is bellow 3MB/s, I believe.

Cheers,
Cláudio

2017-01-12 1:01 GMT+00:00 Ted Timmons via Duplicity-talk <[hidden email]>:
I'm doing a big backup. It'll take days/weeks to complete. With --progress turned on, I see this:

0.0KB 15:00:22 [0.0B/s] [>                                        ] 0% ETA Stalled!


I know it isn't stalled because I can see the volume size on the remote end getting bigger. Is this to be expected?


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk



_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: large upload "ETA stalled"?

duplicity-talk mailing list
Thanks. I'm using the B2 backend (backblaze) and am on the latest gzip (0.7.11).

Maybe I'll look into the b2 connector and compare it to s3.

On Thu, Jan 12, 2017 at 4:22 PM Cláudio Gil <[hidden email]> wrote:
Hi,

I've used this chance to explore the source code and it's seems that reporting progress is the "responsibility" of the backend. The S3 or Dropbox backends, for example, report progress but some (most?) don't. You haven't mentioned the version of duplicity you are using but it should apply.

In any case, the progress tracker uses a 5s window to determine if the upload is stalled. So, if the backend does not report progress you will always get "stalled". For S3 you can "stall" if the upload rate is bellow 25KB/s and for Dropbox you may "stall" if the upload rate is bellow 3MB/s, I believe.

Cheers,
Cláudio

2017-01-12 1:01 GMT+00:00 Ted Timmons via Duplicity-talk <[hidden email]>:
I'm doing a big backup. It'll take days/weeks to complete. With --progress turned on, I see this:

0.0KB 15:00:22 [0.0B/s] [>                                        ] 0% ETA Stalled!


I know it isn't stalled because I can see the volume size on the remote end getting bigger. Is this to be expected?


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk



_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: large upload "ETA stalled"?

duplicity-talk mailing list
If your talking about looking at the code, just look for "report_progress".

Ted Timmons <[hidden email]> escreveu em sex, 13/01/2017 às 00:36 :
Thanks. I'm using the B2 backend (backblaze) and am on the latest gzip (0.7.11).

Maybe I'll look into the b2 connector and compare it to s3.

On Thu, Jan 12, 2017 at 4:22 PM Cláudio Gil <[hidden email]> wrote:
Hi,

I've used this chance to explore the source code and it's seems that reporting progress is the "responsibility" of the backend. The S3 or Dropbox backends, for example, report progress but some (most?) don't. You haven't mentioned the version of duplicity you are using but it should apply.

In any case, the progress tracker uses a 5s window to determine if the upload is stalled. So, if the backend does not report progress you will always get "stalled". For S3 you can "stall" if the upload rate is bellow 25KB/s and for Dropbox you may "stall" if the upload rate is bellow 3MB/s, I believe.

Cheers,
Cláudio

2017-01-12 1:01 GMT+00:00 Ted Timmons via Duplicity-talk <[hidden email]>:
I'm doing a big backup. It'll take days/weeks to complete. With --progress turned on, I see this:













0.0KB 15:00:22 [0.0B/s] [>                                        ] 0% ETA Stalled!


I know it isn't stalled because I can see the volume size on the remote end getting bigger. Is this to be expected?




_______________________________________________


Duplicity-talk mailing list


[hidden email]


https://lists.nongnu.org/mailman/listinfo/duplicity-talk









_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: large upload "ETA stalled"?

duplicity-talk mailing list
Neat. Looks like "from duplicity import progress" is the key, I added a little code. Perhaps you could merge it? (consider the changes PD for licensing if needed).
https://gist.github.com/tedder/b6af9b01fd422450fc3dea37dd982fd4

2.5GB 00:38:51 [36.3MB/s] [> ] 0% ETA 1d 7h 17min





On Thu, Jan 12, 2017 at 4:41 PM Cláudio Gil <[hidden email]> wrote:
If your talking about looking at the code, just look for "report_progress".

Ted Timmons <[hidden email]> escreveu em sex, 13/01/2017 às 00:36 :
Thanks. I'm using the B2 backend (backblaze) and am on the latest gzip (0.7.11).

Maybe I'll look into the b2 connector and compare it to s3.

On Thu, Jan 12, 2017 at 4:22 PM Cláudio Gil <[hidden email]> wrote:
Hi,

I've used this chance to explore the source code and it's seems that reporting progress is the "responsibility" of the backend. The S3 or Dropbox backends, for example, report progress but some (most?) don't. You haven't mentioned the version of duplicity you are using but it should apply.

In any case, the progress tracker uses a 5s window to determine if the upload is stalled. So, if the backend does not report progress you will always get "stalled". For S3 you can "stall" if the upload rate is bellow 25KB/s and for Dropbox you may "stall" if the upload rate is bellow 3MB/s, I believe.

Cheers,
Cláudio

2017-01-12 1:01 GMT+00:00 Ted Timmons via Duplicity-talk <[hidden email]>:
I'm doing a big backup. It'll take days/weeks to complete. With --progress turned on, I see this:













0.0KB 15:00:22 [0.0B/s] [>                                        ] 0% ETA Stalled!


I know it isn't stalled because I can see the volume size on the remote end getting bigger. Is this to be expected?




_______________________________________________


Duplicity-talk mailing list


[hidden email]


https://lists.nongnu.org/mailman/listinfo/duplicity-talk









_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: large upload "ETA stalled"?

duplicity-talk mailing list

Hi,

I'm not a maintainer. But I've submitted a pull request, some time ago, and it was accepted.

Since the patch is simple enough I will fire up my dev VM, when I get the chance, and submit that patch as a new pull request.

Neat. Looks like "from duplicity import progress" is the key, I added a little code. Perhaps you could merge it? (consider the changes PD for licensing if needed).
https://gist.github.com/tedder/b6af9b01fd422450fc3dea37dd982fd4

2.5GB 00:38:51 [36.3MB/s] [> ] 0% ETA 1d 7h 17min





On Thu, Jan 12, 2017 at 4:41 PM Cláudio Gil <[hidden email]> wrote:
If your talking about looking at the code, just look for "report_progress".

Ted Timmons <[hidden email]> escreveu em sex, 13/01/2017 às 00:36 :
Thanks. I'm using the B2 backend (backblaze) and am on the latest gzip (0.7.11).

Maybe I'll look into the b2 connector and compare it to s3.

On Thu, Jan 12, 2017 at 4:22 PM Cláudio Gil <[hidden email]> wrote:
Hi,

I've used this chance to explore the source code and it's seems that reporting progress is the "responsibility" of the backend. The S3 or Dropbox backends, for example, report progress but some (most?) don't. You haven't mentioned the version of duplicity you are using but it should apply.

In any case, the progress tracker uses a 5s window to determine if the upload is stalled. So, if the backend does not report progress you will always get "stalled". For S3 you can "stall" if the upload rate is bellow 25KB/s and for Dropbox you may "stall" if the upload rate is bellow 3MB/s, I believe.

Cheers,
Cláudio

2017-01-12 1:01 GMT+00:00 Ted Timmons via Duplicity-talk <[hidden email]>:
I'm doing a big backup. It'll take days/weeks to complete. With --progress turned on, I see this:













0.0KB 15:00:22 [0.0B/s] [>                                        ] 0% ETA Stalled!


I know it isn't stalled because I can see the volume size on the remote end getting bigger. Is this to be expected?




_______________________________________________


Duplicity-talk mailing list


[hidden email]


https://lists.nongnu.org/mailman/listinfo/duplicity-talk









_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: large upload "ETA stalled"?

duplicity-talk mailing list
If you look at the diff you can see that I've split the import into two. The rest remains the same.

This kind of progress reporting does not require any support from the backend's API so it would work for all other backends, I suppose. I don't use progress but if others feel important I can try to add this kind of rough progress into the other backends. Maybe this would be 0.8-series material...

Cheers,
Cláudio

2017-01-14 9:48 GMT+00:00 Cláudio Gil <[hidden email]>:

Hi,

I'm not a maintainer. But I've submitted a pull request, some time ago, and it was accepted.

Since the patch is simple enough I will fire up my dev VM, when I get the chance, and submit that patch as a new pull request.

Neat. Looks like "from duplicity import progress" is the key, I added a little code. Perhaps you could merge it? (consider the changes PD for licensing if needed).
https://gist.github.com/tedder/b6af9b01fd422450fc3dea37dd982fd4

2.5GB 00:38:51 [36.3MB/s] [> ] 0% ETA 1d 7h 17min





On Thu, Jan 12, 2017 at 4:41 PM Cláudio Gil <[hidden email]> wrote:
If your talking about looking at the code, just look for "report_progress".

Ted Timmons <[hidden email]> escreveu em sex, 13/01/2017 às 00:36 :
Thanks. I'm using the B2 backend (backblaze) and am on the latest gzip (0.7.11).

Maybe I'll look into the b2 connector and compare it to s3.

On Thu, Jan 12, 2017 at 4:22 PM Cláudio Gil <[hidden email]> wrote:
Hi,

I've used this chance to explore the source code and it's seems that reporting progress is the "responsibility" of the backend. The S3 or Dropbox backends, for example, report progress but some (most?) don't. You haven't mentioned the version of duplicity you are using but it should apply.

In any case, the progress tracker uses a 5s window to determine if the upload is stalled. So, if the backend does not report progress you will always get "stalled". For S3 you can "stall" if the upload rate is bellow 25KB/s and for Dropbox you may "stall" if the upload rate is bellow 3MB/s, I believe.

Cheers,
Cláudio

2017-01-12 1:01 GMT+00:00 Ted Timmons via Duplicity-talk <[hidden email]>:
I'm doing a big backup. It'll take days/weeks to complete. With --progress turned on, I see this:













0.0KB 15:00:22 [0.0B/s] [>                                        ] 0% ETA Stalled!


I know it isn't stalled because I can see the volume size on the remote end getting bigger. Is this to be expected?




_______________________________________________


Duplicity-talk mailing list


[hidden email]


https://lists.nongnu.org/mailman/listinfo/duplicity-talk










_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: large upload "ETA stalled"?

duplicity-talk mailing list
On 15.01.2017 12:15, Cláudio Gil via Duplicity-talk wrote:
> Hi,
>
> https://code.launchpad.net/~m4ktub/duplicity/0.7-fixes/+merge/314778 <https://code.launchpad.net/%7Em4ktub/duplicity/0.7-fixes/+merge/314778>
> If you look at the diff you can see that I've split the import into two. The rest remains the same.
>
> This kind of progress reporting does not require any support from the backend's API so it would work for all other backends, I suppose. I don't use progress but if others feel important I can try to add this kind of rough progress into the other backends. Maybe this would be 0.8-series material...
>

the progress reporting in _boto_{multi,single}.py looks a bit more complex than that.

is it possible that this fix only handles the progress for uploads but ignores downloads _get() ?

..ede/duply.net

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: large upload "ETA stalled"?

duplicity-talk mailing list
The boto API supports a progress callback which means you can get progress updates during the upload. Dropbox API also supports chunks/appends but I don't think other APIs do. I haven't really looked into it.

In any case, you are right, the code only introduces progress report for the upload part. It can be easily added to the get (and other backend) because it is a very rough progress report which don't requires any special support.

Cheers,
Cláudio 

edgar.soldin--- via Duplicity-talk <[hidden email]> escreveu em dom, 15/01/2017 às 11:30 :
On 15.01.2017 12:15, Cláudio Gil via Duplicity-talk wrote:

> Hi,

>

> https://code.launchpad.net/~m4ktub/duplicity/0.7-fixes/+merge/314778 <https://code.launchpad.net/%7Em4ktub/duplicity/0.7-fixes/+merge/314778>

> If you look at the diff you can see that I've split the import into two. The rest remains the same.

>

> This kind of progress reporting does not require any support from the backend's API so it would work for all other backends, I suppose. I don't use progress but if others feel important I can try to add this kind of rough progress into the other backends. Maybe this would be 0.8-series material...

>



the progress reporting in _boto_{multi,single}.py looks a bit more complex than that.



is it possible that this fix only handles the progress for uploads but ignores downloads _get() ?



..ede/duply.net



_______________________________________________

Duplicity-talk mailing list

[hidden email]

https://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: large upload "ETA stalled"?

duplicity-talk mailing list
hey Claudio,

if it's really that easy you might consider writing a decorator that can be added easily to the remaining backends, that do not need special handling.

..ede/duply.net

On 15.01.2017 13:41, Cláudio Gil wrote:

> The boto API supports a progress callback which means you can get progress updates during the upload. Dropbox API also supports chunks/appends but I don't think other APIs do. I haven't really looked into it.
>
> In any case, you are right, the code only introduces progress report for the upload part. It can be easily added to the get (and other backend) because it is a very rough progress report which don't requires any special support.
>
> Cheers,
> Cláudio
>
> edgar.soldin--- via Duplicity-talk <[hidden email] <mailto:[hidden email]>> escreveu em dom, 15/01/2017 às 11:30 :
>
>     On 15.01.2017 12:15, Cláudio Gil via Duplicity-talk wrote:
>
>     > Hi,
>
>     >
>
>     > https://code.launchpad.net/~m4ktub/duplicity/0.7-fixes/+merge/314778 <https://code.launchpad.net/%7Em4ktub/duplicity/0.7-fixes/+merge/314778> <https://code.launchpad.net/%7Em4ktub/duplicity/0.7-fixes/+merge/314778>
>
>     > If you look at the diff you can see that I've split the import into two. The rest remains the same.
>
>     >
>
>     > This kind of progress reporting does not require any support from the backend's API so it would work for all other backends, I suppose. I don't use progress but if others feel important I can try to add this kind of rough progress into the other backends. Maybe this would be 0.8-series material...
>
>     >
>
>
>
>     the progress reporting in _boto_{multi,single}.py looks a bit more complex than that.
>
>
>
>     is it possible that this fix only handles the progress for uploads but ignores downloads _get() ?
>
>
>
>     ..ede/duply.net <http://duply.net>
>
>
>
>     _______________________________________________
>
>     Duplicity-talk mailing list
>
>     [hidden email] <mailto:[hidden email]>
>
>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: large upload "ETA stalled"?

duplicity-talk mailing list
The previous branch now includes the "autoprogress" decorator and uses it in the b2 backend. I've also added it to the local backend.

I've done a few tests with the local backend and it works as expected. Since it only provides progress at the end of the upload/download then it still shows stalled when volumes take longer than 5s (by default) to transfer. Using "--progress-rate=<sec>" can avoid that. For example, with the default volume size of 200MB and --progress-rate=60 you will see "Stalled", one in a while, when the bandwidth is bellow 1.7MB/s (bandwidth = volsize/(rate*2)).

Cheers,
Cláudio

2017-01-15 12:46 GMT+00:00 <[hidden email]>:
hey Claudio,

if it's really that easy you might consider writing a decorator that can be added easily to the remaining backends, that do not need special handling.

..ede/duply.net

On 15.01.2017 13:41, Cláudio Gil wrote:
> The boto API supports a progress callback which means you can get progress updates during the upload. Dropbox API also supports chunks/appends but I don't think other APIs do. I haven't really looked into it.
>
> In any case, you are right, the code only introduces progress report for the upload part. It can be easily added to the get (and other backend) because it is a very rough progress report which don't requires any special support.
>
> Cheers,
> Cláudio
>
> edgar.soldin--- via Duplicity-talk <[hidden email] <mailto:[hidden email]>> escreveu em dom, 15/01/2017 às 11:30 :
>
>     On 15.01.2017 12:15, Cláudio Gil via Duplicity-talk wrote:
>
>     > Hi,
>
>     >
>
>     > https://code.launchpad.net/~m4ktub/duplicity/0.7-fixes/+merge/314778 <https://code.launchpad.net/%7Em4ktub/duplicity/0.7-fixes/+merge/314778> <https://code.launchpad.net/%7Em4ktub/duplicity/0.7-fixes/+merge/314778>
>
>     > If you look at the diff you can see that I've split the import into two. The rest remains the same.
>
>     >
>
>     > This kind of progress reporting does not require any support from the backend's API so it would work for all other backends, I suppose. I don't use progress but if others feel important I can try to add this kind of rough progress into the other backends. Maybe this would be 0.8-series material...
>
>     >
>
>
>
>     the progress reporting in _boto_{multi,single}.py looks a bit more complex than that.
>
>
>
>     is it possible that this fix only handles the progress for uploads but ignores downloads _get() ?
>
>
>
>     ..ede/duply.net <http://duply.net>
>
>
>
>     _______________________________________________
>
>     Duplicity-talk mailing list
>
>     [hidden email] <mailto:[hidden email]>
>
>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: large upload "ETA stalled"?

duplicity-talk mailing list
i wonder if it'd make sense to somehow mark/tag backends that support progress and throw an error when --progress is enabled on a backend that currently does not support it?

local file progress should probably properly implement via
  http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/duplicity/path.py#L622
to avoid "stalling" on local targets. users might be confused when their otherwise outperforming system claims to be "stalling".

..ede/duply.net

On 16.01.2017 03:08, Cláudio Gil wrote:

> The previous branch now includes the "autoprogress" decorator and uses it
> in the b2 backend. I've also added it to the local backend.
>
> I've done a few tests with the local backend and it works as expected.
> Since it only provides progress at the end of the upload/download then it
> still shows stalled when volumes take longer than 5s (by default) to
> transfer. Using "--progress-rate=<sec>" can avoid that. For example, with
> the default volume size of 200MB and --progress-rate=60 you will see
> "Stalled", one in a while, when the bandwidth is bellow 1.7MB/s (bandwidth
> = volsize/(rate*2)).
>
> Cheers,
> Cláudio
>
> 2017-01-15 12:46 GMT+00:00 <[hidden email]>:
>
>> hey Claudio,
>>
>> if it's really that easy you might consider writing a decorator that can
>> be added easily to the remaining backends, that do not need special
>> handling.
>>
>> ..ede/duply.net
>>
>> On 15.01.2017 13:41, Cláudio Gil wrote:
>>> The boto API supports a progress callback which means you can get
>> progress updates during the upload. Dropbox API also supports
>> chunks/appends but I don't think other APIs do. I haven't really looked
>> into it.
>>>
>>> In any case, you are right, the code only introduces progress report for
>> the upload part. It can be easily added to the get (and other backend)
>> because it is a very rough progress report which don't requires any special
>> support.
>>>
>>> Cheers,
>>> Cláudio
>>>
>>> edgar.soldin--- via Duplicity-talk <[hidden email] <mailto:
>> [hidden email]>> escreveu em dom, 15/01/2017 às 11:30 :
>>>
>>>     On 15.01.2017 12:15, Cláudio Gil via Duplicity-talk wrote:
>>>
>>>     > Hi,
>>>
>>>     >
>>>
>>>     > https://code.launchpad.net/~m4ktub/duplicity/0.7-fixes/+
>> merge/314778 <https://code.launchpad.net/%7Em4ktub/duplicity/0.7-fixes/+
>> merge/314778> <https://code.launchpad.net/%7Em4ktub/duplicity/0.7-fixes/+
>> merge/314778>
>>>
>>>     > If you look at the diff you can see that I've split the import
>> into two. The rest remains the same.
>>>
>>>     >
>>>
>>>     > This kind of progress reporting does not require any support from
>> the backend's API so it would work for all other backends, I suppose. I
>> don't use progress but if others feel important I can try to add this kind
>> of rough progress into the other backends. Maybe this would be 0.8-series
>> material...
>>>
>>>     >
>>>
>>>
>>>
>>>     the progress reporting in _boto_{multi,single}.py looks a bit more
>> complex than that.
>>>
>>>
>>>
>>>     is it possible that this fix only handles the progress for uploads
>> but ignores downloads _get() ?
>>>
>>>
>>>
>>>     ..ede/duply.net <http://duply.net>
>>>
>>>
>>>
>>>     _______________________________________________
>>>
>>>     Duplicity-talk mailing list
>>>
>>>     [hidden email] <mailto:[hidden email]>
>>>
>>>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>
>

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: large upload "ETA stalled"?

duplicity-talk mailing list
This is cool, Claudio and Edgar, thanks.

On Mon, Jan 16, 2017 at 2:01 AM edgar.soldin--- via Duplicity-talk <[hidden email]> wrote:
i wonder if it'd make sense to somehow mark/tag backends that support progress and throw an error when --progress is enabled on a backend that currently does not support it?

local file progress should probably properly implement via
  http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/duplicity/path.py#L622
to avoid "stalling" on local targets. users might be confused when their otherwise outperforming system claims to be "stalling".

..ede/duply.net

On 16.01.2017 03:08, Cláudio Gil wrote:
> The previous branch now includes the "autoprogress" decorator and uses it
> in the b2 backend. I've also added it to the local backend.
>
> I've done a few tests with the local backend and it works as expected.
> Since it only provides progress at the end of the upload/download then it
> still shows stalled when volumes take longer than 5s (by default) to
> transfer. Using "--progress-rate=<sec>" can avoid that. For example, with
> the default volume size of 200MB and --progress-rate=60 you will see
> "Stalled", one in a while, when the bandwidth is bellow 1.7MB/s (bandwidth
> = volsize/(rate*2)).
>
> Cheers,
> Cláudio
>
> 2017-01-15 12:46 GMT+00:00 <[hidden email]>:
>
>> hey Claudio,
>>
>> if it's really that easy you might consider writing a decorator that can
>> be added easily to the remaining backends, that do not need special
>> handling.
>>
>> ..ede/duply.net
>>
>> On 15.01.2017 13:41, Cláudio Gil wrote:
>>> The boto API supports a progress callback which means you can get
>> progress updates during the upload. Dropbox API also supports
>> chunks/appends but I don't think other APIs do. I haven't really looked
>> into it.
>>>
>>> In any case, you are right, the code only introduces progress report for
>> the upload part. It can be easily added to the get (and other backend)
>> because it is a very rough progress report which don't requires any special
>> support.
>>>
>>> Cheers,
>>> Cláudio
>>>
>>> edgar.soldin--- via Duplicity-talk <[hidden email] <mailto:
>> [hidden email]>> escreveu em dom, 15/01/2017 às 11:30 :
>>>
>>>     On 15.01.2017 12:15, Cláudio Gil via Duplicity-talk wrote:
>>>
>>>     > Hi,
>>>
>>>     >
>>>
>>>     > https://code.launchpad.net/~m4ktub/duplicity/0.7-fixes/+
>> merge/314778 <https://code.launchpad.net/%7Em4ktub/duplicity/0.7-fixes/+
>> merge/314778> <https://code.launchpad.net/%7Em4ktub/duplicity/0.7-fixes/+
>> merge/314778>
>>>
>>>     > If you look at the diff you can see that I've split the import
>> into two. The rest remains the same.
>>>
>>>     >
>>>
>>>     > This kind of progress reporting does not require any support from
>> the backend's API so it would work for all other backends, I suppose. I
>> don't use progress but if others feel important I can try to add this kind
>> of rough progress into the other backends. Maybe this would be 0.8-series
>> material...
>>>
>>>     >
>>>
>>>
>>>
>>>     the progress reporting in _boto_{multi,single}.py looks a bit more
>> complex than that.
>>>
>>>
>>>
>>>     is it possible that this fix only handles the progress for uploads
>> but ignores downloads _get() ?
>>>
>>>
>>>
>>>     ..ede/duply.net <http://duply.net>
>>>
>>>
>>>
>>>     _______________________________________________
>>>
>>>     Duplicity-talk mailing list
>>>
>>>     [hidden email] <mailto:[hidden email]>
>>>
>>>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>
>

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: large upload "ETA stalled"?

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list

I don't have a strong feeling regarding a mark/tag but it does not seem very useful. We could as well add this rough progress reporting to the main backend.py logic and ensure all current and future backends have it. Each backend would then only need to add intermediary progress if it's possible. For backends like boto or dpbx the extra cost would be, more or less, the cost of an extra os.path.getsize(...).

Cheers,
Cláudio

Em 16/01/2017 10:00, <[hidden email]> escreveu:
i wonder if it'd make sense to somehow mark/tag backends that support progress and throw an error when --progress is enabled on a backend that currently does not support it?

local file progress should probably properly implement via
  http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/duplicity/path.py#L622
to avoid "stalling" on local targets. users might be confused when their otherwise outperforming system claims to be "stalling".

..ede/duply.net

On 16.01.2017 03:08, Cláudio Gil wrote:
> The previous branch now includes the "autoprogress" decorator and uses it
> in the b2 backend. I've also added it to the local backend.
>
> I've done a few tests with the local backend and it works as expected.
> Since it only provides progress at the end of the upload/download then it
> still shows stalled when volumes take longer than 5s (by default) to
> transfer. Using "--progress-rate=<sec>" can avoid that. For example, with
> the default volume size of 200MB and --progress-rate=60 you will see
> "Stalled", one in a while, when the bandwidth is bellow 1.7MB/s (bandwidth
> = volsize/(rate*2)).
>
> Cheers,
> Cláudio
>
> 2017-01-15 12:46 GMT+00:00 <[hidden email]>:
>
>> hey Claudio,
>>
>> if it's really that easy you might consider writing a decorator that can
>> be added easily to the remaining backends, that do not need special
>> handling.
>>
>> ..ede/duply.net
>>
>> On 15.01.2017 13:41, Cláudio Gil wrote:
>>> The boto API supports a progress callback which means you can get
>> progress updates during the upload. Dropbox API also supports
>> chunks/appends but I don't think other APIs do. I haven't really looked
>> into it.
>>>
>>> In any case, you are right, the code only introduces progress report for
>> the upload part. It can be easily added to the get (and other backend)
>> because it is a very rough progress report which don't requires any special
>> support.
>>>
>>> Cheers,
>>> Cláudio
>>>
>>> edgar.soldin--- via Duplicity-talk <[hidden email] <mailto:
>> [hidden email]>> escreveu em dom, 15/01/2017 às 11:30 :
>>>
>>>     On 15.01.2017 12:15, Cláudio Gil via Duplicity-talk wrote:
>>>
>>>     > Hi,
>>>
>>>     >
>>>
>>>     > https://code.launchpad.net/~m4ktub/duplicity/0.7-fixes/+
>> merge/314778 <https://code.launchpad.net/%7Em4ktub/duplicity/0.7-fixes/+
>> merge/314778> <https://code.launchpad.net/%7Em4ktub/duplicity/0.7-fixes/+
>> merge/314778>
>>>
>>>     > If you look at the diff you can see that I've split the import
>> into two. The rest remains the same.
>>>
>>>     >
>>>
>>>     > This kind of progress reporting does not require any support from
>> the backend's API so it would work for all other backends, I suppose. I
>> don't use progress but if others feel important I can try to add this kind
>> of rough progress into the other backends. Maybe this would be 0.8-series
>> material...
>>>
>>>     >
>>>
>>>
>>>
>>>     the progress reporting in _boto_{multi,single}.py looks a bit more
>> complex than that.
>>>
>>>
>>>
>>>     is it possible that this fix only handles the progress for uploads
>> but ignores downloads _get() ?
>>>
>>>
>>>
>>>     ..ede/duply.net <http://duply.net>
>>>
>>>
>>>
>>>     _______________________________________________
>>>
>>>     Duplicity-talk mailing list
>>>
>>>     [hidden email] <mailto:[hidden email]>
>>>
>>>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>
>

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk