Python error in Webdav backend

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Python error in Webdav backend

duplicity-talk mailing list
Hi all,

I am in the process of replacing my NAS with a new one. In that process, I am moving from a working Duplicity 0.7.18.1 on Python 2.7.12 to Duplicity 0.8.04 on Python 3.7.3.

And I am getting the following error in the Webdav backend, using credentials that work on the existing installation (as well as using a different webdav client).

I have set env variables as follows:

FTP_PASSWORD=MyWebDavPassword
PASSPHRASE=MyBackupPassword

Who knows how to move forward from this?

$ duplicity -v9 full --volsize 64 --name tmp /tmp $TARGET/tmp
Using archive dir: /home/sven/.cache/duplicity/tmp
Using backup name: tmp
GPG binary is gpg, version (2, 2, 12)
Import of duplicity.backends.adbackend Succeeded
Import of duplicity.backends.azurebackend Succeeded
Import of duplicity.backends.b2backend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.dpbxbackend Failed: No module named 'requests'
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.hubicbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.jottacloudbackend Succeeded
Import of duplicity.backends.lftpbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.mediafirebackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.multibackend Succeeded
Import of duplicity.backends.ncftpbackend Succeeded
Import of duplicity.backends.onedrivebackend Succeeded
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.pcabackend Succeeded
Import of duplicity.backends.pydrivebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.ssh_paramiko_backend Succeeded
Import of duplicity.backends.ssh_pexpect_backend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Using WebDAV protocol http
Using WebDAV host svenhz.stackstorage.com port None
Using WebDAV directory /remote.php/webdav/backup/tmp/
Main action: full
Acquiring lockfile b'/home/sven/.cache/duplicity/tmp/lockfile'
================================================================================
duplicity $version ($reldate)
Args: /usr/local/bin/duplicity -v9 full --volsize 64 --name tmp /tmp webdavs://svenhz@.../remote.php/webdav/backup/tmp
Linux helios4 4.19.63-mvebu #5.91 SMP Wed Jul 31 15:58:43 CEST 2019 armv7l
/usr/bin/python 3.7.3 (default, Apr  3 2019, 05:39:12)
[GCC 8.3.0]
================================================================================
Using temporary directory /tmp/duplicity-nxtddn9t-tempdir
Registering (mkstemp) temporary file /tmp/duplicity-nxtddn9t-tempdir/mkstemp-h85_trch-1
Temp has 1056886784 available, backup will use approx 87241523.
WebDAV create connection on 'svenhz.stackstorage.com'
WebDAV PROPFIND /remote.php/webdav/backup/tmp/ request with headers: {'Connection': 'keep-alive', 'Depth': '1'}
WebDAV data length: 95
WebDAV response status 401 with reason 'Unauthorized'.
Backtrace of previous error: Traceback (innermost last):
  File "/usr/local/lib/python3.7/dist-packages/duplicity/backend.py", line 371, in inner_retry
    return fn(self, *args)
  File "/usr/local/lib/python3.7/dist-packages/duplicity/backend.py", line 580, in list
    return [tobytes(x) for x in self.backend._list()]
  File "/usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py", line 343, in _list
    raise e
  File "/usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py", line 317, in _list
    response = self.request(u"PROPFIND", self.directory, self.listbody)
  File "/usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py", line 240, in request
    self.headers[u'Authorization'] = self.get_authorization(response, quoted_path)
  File "/usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py", line 271, in get_authorization
    return self.get_basic_authorization()
  File "/usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py", line 291, in get_basic_authorization
    return u'Basic %s' % base64.encodestring(auth_string).strip()
  File "/usr/lib/python3.7/base64.py", line 540, in encodestring
    return encodebytes(s)
  File "/usr/lib/python3.7/base64.py", line 527, in encodebytes
    _input_type_check(s)
  File "/usr/lib/python3.7/base64.py", line 513, in _input_type_check
    raise TypeError(msg) from err
 TypeError: expected bytes-like object, not str

Attempt 1 failed. TypeError: expected bytes-like object, not str

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Python error in Webdav backend

duplicity-talk mailing list
hey Sven, see below

On 20.09.2019 12:04, Sven Hazejager via Duplicity-talk wrote:

> Hi all,
>
> I am in the process of replacing my NAS with a new one. In that process, I am moving from a working Duplicity 0.7.18.1 on Python 2.7.12 to Duplicity 0.8.04 on Python 3.7.3.
>
> And I am getting the following error in the Webdav backend, using credentials that work on the existing installation (as well as using a different webdav client).
>
> I have set env variables as follows:
>
> FTP_PASSWORD=MyWebDavPassword
> PASSPHRASE=MyBackupPassword
> TARGET=webdavs://[hidden email]/remote.php/webdav/backup <http://svenhz@.../remote.php/webdav/backup>
>
> Who knows how to move forward from this?
>
> $ duplicity -v9 full --volsize 64 --name tmp /tmp $TARGET/tmp
> Using archive dir: /home/sven/.cache/duplicity/tmp
> Using backup name: tmp
> GPG binary is gpg, version (2, 2, 12)
> Import of duplicity.backends.adbackend Succeeded
> Import of duplicity.backends.azurebackend Succeeded
> Import of duplicity.backends.b2backend Succeeded
> Import of duplicity.backends.botobackend Succeeded
> Import of duplicity.backends.cfbackend Succeeded
> Import of duplicity.backends.dpbxbackend Failed: No module named 'requests'
> Import of duplicity.backends.gdocsbackend Succeeded
> Import of duplicity.backends.giobackend Succeeded
> Import of duplicity.backends.hsibackend Succeeded
> Import of duplicity.backends.hubicbackend Succeeded
> Import of duplicity.backends.imapbackend Succeeded
> Import of duplicity.backends.jottacloudbackend Succeeded
> Import of duplicity.backends.lftpbackend Succeeded
> Import of duplicity.backends.localbackend Succeeded
> Import of duplicity.backends.mediafirebackend Succeeded
> Import of duplicity.backends.megabackend Succeeded
> Import of duplicity.backends.multibackend Succeeded
> Import of duplicity.backends.ncftpbackend Succeeded
> Import of duplicity.backends.onedrivebackend Succeeded
> Import of duplicity.backends.par2backend Succeeded
> Import of duplicity.backends.pcabackend Succeeded
> Import of duplicity.backends.pydrivebackend Succeeded
> Import of duplicity.backends.rsyncbackend Succeeded
> Import of duplicity.backends.ssh_paramiko_backend Succeeded
> Import of duplicity.backends.ssh_pexpect_backend Succeeded
> Import of duplicity.backends.swiftbackend Succeeded
> Import of duplicity.backends.sxbackend Succeeded
> Import of duplicity.backends.tahoebackend Succeeded
> Import of duplicity.backends.webdavbackend Succeeded
> Using WebDAV protocol http
> Using WebDAV host svenhz.stackstorage.com <http://svenhz.stackstorage.com> port None
> Using WebDAV directory /remote.php/webdav/backup/tmp/
> Main action: full
> Acquiring lockfile b'/home/sven/.cache/duplicity/tmp/lockfile'
> ================================================================================
> duplicity $version ($reldate)
> Args: /usr/local/bin/duplicity -v9 full --volsize 64 --name tmp /tmp webdavs://[hidden email]/remote.php/webdav/backup/tmp <http://svenhz@.../remote.php/webdav/backup/tmp>
> Linux helios4 4.19.63-mvebu #5.91 SMP Wed Jul 31 15:58:43 CEST 2019 armv7l
> /usr/bin/python 3.7.3 (default, Apr  3 2019, 05:39:12)
> [GCC 8.3.0]
> ================================================================================
> Using temporary directory /tmp/duplicity-nxtddn9t-tempdir
> Registering (mkstemp) temporary file /tmp/duplicity-nxtddn9t-tempdir/mkstemp-h85_trch-1
> Temp has 1056886784 available, backup will use approx 87241523.
> WebDAV create connection on 'svenhz.stackstorage.com <http://svenhz.stackstorage.com>'
> WebDAV PROPFIND /remote.php/webdav/backup/tmp/ request with headers: {'Connection': 'keep-alive', 'Depth': '1'}
> WebDAV data length: 95
> WebDAV response status 401 with reason 'Unauthorized'.
> Backtrace of previous error: Traceback (innermost last):
>   File "/usr/local/lib/python3.7/dist-packages/duplicity/backend.py", line 371, in inner_retry
>     return fn(self, *args)
>   File "/usr/local/lib/python3.7/dist-packages/duplicity/backend.py", line 580, in list
>     return [tobytes(x) for x in self.backend._list()]
>   File "/usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py", line 343, in _list
>     raise e
>   File "/usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py", line 317, in _list
>     response = self.request(u"PROPFIND", self.directory, self.listbody)
>   File "/usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py", line 240, in request
>     self.headers[u'Authorization'] = self.get_authorization(response, quoted_path)
>   File "/usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py", line 271, in get_authorization
>     return self.get_basic_authorization()
>   File "/usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py", line 291, in get_basic_authorization
>     return u'Basic %s' % base64.encodestring(auth_string).strip()
>   File "/usr/lib/python3.7/base64.py", line 540, in encodestring
>     return encodebytes(s)
>   File "/usr/lib/python3.7/base64.py", line 527, in encodebytes
>     _input_type_check(s)
>   File "/usr/lib/python3.7/base64.py", line 513, in _input_type_check
>     raise TypeError(msg) from err
>  TypeError: expected bytes-like object, not str
>
> Attempt 1 failed. TypeError: expected bytes-like object, not str

probably this issue
  https://stackoverflow.com/questions/8908287/why-do-i-need-b-to-encode-a-string-with-base64

try patching /usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py", line 291
from
  return u'Basic %s' % base64.encodestring(auth_string).strip()
to
  return u'Basic %s' % base64.encodestring( bytes(auth_string, "utf-8") ).strip()

and see if that works. if so please tell us so we can patch this bug accordingly.

..ede/duply.net


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Python error in Webdav backend

duplicity-talk mailing list
Hi Edgar,

On Fri, 20 Sep 2019 at 15:12, edgar.soldin--- via Duplicity-talk <[hidden email]> wrote:

probably this issue
  https://stackoverflow.com/questions/8908287/why-do-i-need-b-to-encode-a-string-with-base64

try patching /usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py", line 291
from
  return u'Basic %s' % base64.encodestring(auth_string).strip()
to
  return u'Basic %s' % base64.encodestring( bytes(auth_string, "utf-8") ).strip()

and see if that works. if so please tell us so we can patch this bug accordingly.


Thanks, not solved yet, but we're close. Below is the current output. The call actually goes through to the server. BUT, where the asterisks are is the Base64 encoded password... and it has a leading " b' ", which my working duplicity install does NOT have. I am not sure if that is a part of the newly created byte array, are that it gets added somewhere else when the byte array is printed.

WebDAV create connection on 'svenhz.stackstorage.com'
WebDAV PROPFIND /remote.php/webdav/backup/tmp/ request with headers: {'Connection': 'keep-alive', 'Depth': '1'}
WebDAV data length: 95
WebDAV response status 401 with reason 'Unauthorized'.
WebDAV retry request with authentification headers.
WebDAV PROPFIND /remote.php/webdav/backup/tmp/ request2 with headers: {'Connection': 'keep-alive', 'Depth': '1', 'Authorization': "Basic b'*****************'"}
WebDAV data length: 95
WebDAV response2 status 401 with reason 'Unauthorized'.
Backtrace of previous error: Traceback (innermost last):
  File "/usr/local/lib/python3.7/dist-packages/duplicity/backend.py", line 371, in inner_retry
    return fn(self, *args)
  File "/usr/local/lib/python3.7/dist-packages/duplicity/backend.py", line 580, in list
    return [tobytes(x) for x in self.backend._list()]
  File "/usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py", line 343, in _list
    raise e
  File "/usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py", line 332, in _list
    raise BackendException(u"Bad status code %s reason %s." % (status, reason))
 duplicity.errors.BackendException: Bad status code 401 reason Unauthorized.

Attempt 1 failed. BackendException: Bad status code 401 reason Unauthorized.

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Python error in Webdav backend

duplicity-talk mailing list


On Fri, 20 Sep 2019 at 19:04, Sven Hazejager <[hidden email]> wrote:

Thanks, not solved yet, but we're close. Below is the current output. The call actually goes through to the server. BUT, where the asterisks are is the Base64 encoded password... and it has a leading " b' ", which my working duplicity install does NOT have. I am not sure if that is a part of the newly created byte array, are that it gets added somewhere else when the byte array is printed.


Actually, it looks like this solves this error, thanks to https://stackoverflow.com/questions/31144988/base64-encodestring-failing-in-python-3.

Change line 291 to:

 return u'Basic %s' % base64.encodestring(auth_string.encode()).decode().strip()

Now a backup starts, and we hit the next error once the first volume is uploaded:

Writing duplicity-full.20190920T171053Z.vol1.difftar.gpg
Backtrace of previous error: Traceback (innermost last):
  File "/usr/local/lib/python3.7/dist-packages/duplicity/backend.py", line 371, in inner_retry
    return fn(self, *args)
  File "/usr/local/lib/python3.7/dist-packages/duplicity/backend.py", line 531, in put
    self.__do_put(source_path, remote_filename)
  File "/usr/local/lib/python3.7/dist-packages/duplicity/backend.py", line 517, in __do_put
    self.backend._put(source_path, remote_filename)
  File "/usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py", line 437, in _put
    url = self.directory + remote_filename
 TypeError: can only concatenate str (not "bytes") to str

Attempt 1 failed. TypeError: can only concatenate str (not "bytes") to str

Any ideas? :-)


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Python error in Webdav backend

duplicity-talk mailing list


On Fri, 20 Sep 2019 at 19:11, Sven Hazejager <[hidden email]> wrote:

Attempt 1 failed. TypeError: can only concatenate str (not "bytes") to str

Any ideas? :-)


Again, I am replying to myself. I fixed it and basic backup use cases seem to work (more testing needed still):

/usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py

Line 411 add .decode() to remote_filename
Line 437 add .decode() to remote_filename
Line 459 add .decode() to filename

So far, good news.

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Python error in Webdav backend

duplicity-talk mailing list
This fix will be in the next release, 0.8.05.

Thanks for the help!

...Ken

On Fri, Sep 20, 2019 at 1:25 PM Sven Hazejager via Duplicity-talk <[hidden email]> wrote:


On Fri, 20 Sep 2019 at 19:11, Sven Hazejager <[hidden email]> wrote:

Attempt 1 failed. TypeError: can only concatenate str (not "bytes") to str

Any ideas? :-)


Again, I am replying to myself. I fixed it and basic backup use cases seem to work (more testing needed still):

/usr/local/lib/python3.7/dist-packages/duplicity/backends/webdavbackend.py

Line 411 add .decode() to remote_filename
Line 437 add .decode() to remote_filename
Line 459 add .decode() to filename

So far, good news.
_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk