Fwd: Re: gpg key password asked for backup after verify

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: Re: gpg key password asked for backup after verify

duplicity-talk mailing list
Ken,

how is the state wrt. librsync patch, didn't see it in the repo so far?

*and*, below is once again somebody complaining that suddenly a private key / passphrase is needed. would you agree that we should remove the dirty workaround described in
  https://bugs.launchpad.net/duplicity/+bug/687295
for duplicity 0.8?

it really only "works" with english locale and a backup that does not try to resume. still users risk to do backups where archive dir is not synchronized, which is bad for their data.

..ede

-------- Forwarded Message --------
Subject: Re: [Duplicity-talk] gpg key password asked for backup after verify
Date: Wed, 24 May 2017 16:14:31 +0200
From: Raphael Bauduin <[hidden email]>
To: [hidden email]





On Wed, May 24, 2017 at 1:49 PM, <[hidden email] <mailto:[hidden email]>> wrote:

    whoops, hit send too fast. read on below.

    On 24.05.2017 13 <tel:24.05.2017%2013>:17, Raphael Bauduin wrote:
    >
    >
    > On Wed, May 24, 2017 at 12:19 PM, edgar.soldin--- via Duplicity-talk <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
    >
    >     On 24.05.2017 11 <tel:24.05.2017%2011> <tel:24.05.2017%2011>:28, Raphael Bauduin via Duplicity-talk wrote:
    >     > Hi,
    >     >
    >     > I had encrypted backups working fine for weeks on a server. As the encryption uses the public key, it doesn't ask for a password.
    >     >
    >     > Then I did a duplicity verify, which requires the gpg private key, and asks for a password.
    >     > The verify went fine, but since then the gpg key password is also asked for backups, preventing the automation.... I'm nearly sure this is linked
    >     >
    >     > I have removed the duplicity cache in ~/.cache/duplicity, but to no avail....
    >     >
    >     > Any suggestion?
    >     >
    >
    >     1.
    >     are you using duply?
    >
    >
    > no
    >
    >
    >
    >     2.
    >     what is your backup command line?
    >
    >
    >  LC_ALL=en_US /bin/duplicity   inc --encrypt-key 'XXXX' --exclude /root/.cache/duplicity --exclude  /home/backups --exclude /home/restore --exclude /backups  --include /home/sftp --include /etc --include /home --include /root --exclude '**' / par2+rsync://rsync/duplicity/   --verbosity debug
    >
    >
    >
    >
    >     3.
    >     what's the language locale of your os?
    >
    >
    > I'm forcing it to en_US, which worked fine.
    >
    > Investigating further, I think I might have deleted the cache before I did the verify. So not sure which one causes what.
    > I took a look at the code. Here is the code in question asking for the password when the cache was empty, where I added a print:
    >             if local_missing and (rem_needpass or loc_needpass):
    >                 if decrypt:
    >                     # password for the --encrypt-key
    >                     print "local_missing = %s,--  %s, -- %s" % (local_missing, rem_needpass, loc_needpass)
    >                     globals.gpg_profile.passphrase = get_passphrase(1, "sync")
    >
    > local_missing was a set of .sigtar.gpg files, rem_needpass was True and loc_needpass was False.
    >
    > Now I have done a backup manually (providing the key password), I have the else clause below asking for the password although the action is inc:
    >
    >     elif (action == "inc" and
    >           (globals.gpg_profile.recipients or globals.gpg_profile.hidden_recipients) and not
    >           globals.gpg_profile.sign_key and not globals.restart):
    >         return ""
    >
    >     # Finally, ask the user for the passphrase
    >     else:
    >         print "action = %s" % action
    >         log.Info(_("PASSPHRASE variable not set, asking user."))
    >         use_cache = True
    >
    >
    > globals.gpg_profile.recipients is my encryption key id, globals.gpg_profile.sign_key is None, but globals.restart= <__main__.Restart instance at 0x13a8518>
    >
    > So it seems that the globals.restart is set and makes the code skip the action == "inc" part.
    >
    > Any idea what the problem might be?
    >
    > Thanks
    >

    ok, your backup is restarting. restarting happens when a backup was interrupted. restarting _needs_ to decrypt some information from the backend, which can only be done w/ priv key and passphrase of course.


Just confirm that's what I had. Making a full successful backup makes it run fine without password prompt on subsequent runs.

Thanks a lot for your help!


 


    what you ran into here is essentially this bug
      https://bugs.launchpad.net/duplicity/+bug/687295 <https://bugs.launchpad.net/duplicity/+bug/687295>

    consider using two key pairs in the future. duplicity using gpg can encrypt to multiple keys. place

    A. a sec/pub key for the box
    B. your own pub key

    in your keyring. then backup against both keys and optionally use the machine key to sign your backups. this way the box can decrypt when needed w/o needing your very secret personal private key.

    ..ede/duply.net <http://duply.net>




--
Web database: http://www.myowndb.com
Free Software Developers Meeting: http://www.fosdem.org

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
ede,

1) librsync is good up thru version 2.x.  Not sure exactly what version of duplicity it was fixed in, but the logs should have the bug number and we can track it from there.  It's been a long while since it was fixed, perhaps all the way back to the last of the 0.6-series.  We still use the smaller signatures and I'm going to leave that as an option going forwards.  The larger signatures will double the size of the sigtar file, and will force a full backup if we go to them.

2) The decryption/encryption issues during restart and incrementals are:
  • Do we leave the cache unencrypted?  Without the manifest in plain form, or without the ability to decrypt it, we can't continue.
  • How do we decrypt if the cache is out of sync with the remote?  It happens and the remote manifest needs to be decrypted.
I see no really good reason to encrypt the manifest since it only contains information readily available from the system itself.  However, on the remote it needs to be encrypted, otherwise it could reveal the filenames of the backed up files.  Some folks are sensitive to that.

...Ken



On Wed, May 24, 2017 at 9:47 AM, <[hidden email]> wrote:
Ken,

how is the state wrt. librsync patch, didn't see it in the repo so far?

*and*, below is once again somebody complaining that suddenly a private key / passphrase is needed. would you agree that we should remove the dirty workaround described in
  https://bugs.launchpad.net/duplicity/+bug/687295
for duplicity 0.8?

it really only "works" with english locale and a backup that does not try to resume. still users risk to do backups where archive dir is not synchronized, which is bad for their data.

..ede

-------- Forwarded Message --------
Subject:        Re: [Duplicity-talk] gpg key password asked for backup after verify
Date:   Wed, 24 May 2017 16:14:31 +0200
From:   Raphael Bauduin <[hidden email]>
To:     [hidden email]





On Wed, May 24, 2017 at 1:49 PM, <[hidden email] <mailto:[hidden email]>> wrote:

    whoops, hit send too fast. read on below.

    On <a href="tel:24.05.2017%2013" value="+12405201713">24.05.2017 13 <tel:24.05.2017%2013>:17, Raphael Bauduin wrote:
    >
    >
    > On Wed, May 24, 2017 at 12:19 PM, edgar.soldin--- via Duplicity-talk <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
    >
    >     On <a href="tel:24.05.2017%2011" value="+12405201711">24.05.2017 11 <tel:24.05.2017%2011> <tel:24.05.2017%2011>:28, Raphael Bauduin via Duplicity-talk wrote:
    >     > Hi,
    >     >
    >     > I had encrypted backups working fine for weeks on a server. As the encryption uses the public key, it doesn't ask for a password.
    >     >
    >     > Then I did a duplicity verify, which requires the gpg private key, and asks for a password.
    >     > The verify went fine, but since then the gpg key password is also asked for backups, preventing the automation.... I'm nearly sure this is linked
    >     >
    >     > I have removed the duplicity cache in ~/.cache/duplicity, but to no avail....
    >     >
    >     > Any suggestion?
    >     >
    >
    >     1.
    >     are you using duply?
    >
    >
    > no
    >
    >
    >
    >     2.
    >     what is your backup command line?
    >
    >
    >  LC_ALL=en_US /bin/duplicity   inc --encrypt-key 'XXXX' --exclude /root/.cache/duplicity --exclude  /home/backups --exclude /home/restore --exclude /backups  --include /home/sftp --include /etc --include /home --include /root --exclude '**' / par2+rsync://rsync/duplicity/   --verbosity debug
    >
    >
    >
    >
    >     3.
    >     what's the language locale of your os?
    >
    >
    > I'm forcing it to en_US, which worked fine.
    >
    > Investigating further, I think I might have deleted the cache before I did the verify. So not sure which one causes what.
    > I took a look at the code. Here is the code in question asking for the password when the cache was empty, where I added a print:
    >             if local_missing and (rem_needpass or loc_needpass):
    >                 if decrypt:
    >                     # password for the --encrypt-key
    >                     print "local_missing = %s,--  %s, -- %s" % (local_missing, rem_needpass, loc_needpass)
    >                     globals.gpg_profile.passphrase = get_passphrase(1, "sync")
    >
    > local_missing was a set of .sigtar.gpg files, rem_needpass was True and loc_needpass was False.
    >
    > Now I have done a backup manually (providing the key password), I have the else clause below asking for the password although the action is inc:
    >
    >     elif (action == "inc" and
    >           (globals.gpg_profile.recipients or globals.gpg_profile.hidden_recipients) and not
    >           globals.gpg_profile.sign_key and not globals.restart):
    >         return ""
    >
    >     # Finally, ask the user for the passphrase
    >     else:
    >         print "action = %s" % action
    >         log.Info(_("PASSPHRASE variable not set, asking user."))
    >         use_cache = True
    >
    >
    > globals.gpg_profile.recipients is my encryption key id, globals.gpg_profile.sign_key is None, but globals.restart= <__main__.Restart instance at 0x13a8518>
    >
    > So it seems that the globals.restart is set and makes the code skip the action == "inc" part.
    >
    > Any idea what the problem might be?
    >
    > Thanks
    >

    ok, your backup is restarting. restarting happens when a backup was interrupted. restarting _needs_ to decrypt some information from the backend, which can only be done w/ priv key and passphrase of course.


Just confirm that's what I had. Making a full successful backup makes it run fine without password prompt on subsequent runs.

Thanks a lot for your help!





    what you ran into here is essentially this bug
      https://bugs.launchpad.net/duplicity/+bug/687295 <https://bugs.launchpad.net/duplicity/+bug/687295>

    consider using two key pairs in the future. duplicity using gpg can encrypt to multiple keys. place

    A. a sec/pub key for the box
    B. your own pub key

    in your keyring. then backup against both keys and optionally use the machine key to sign your backups. this way the box can decrypt when needed w/o needing your very secret personal private key.

    ..ede/duply.net <http://duply.net>




--
Web database: http://www.myowndb.com
Free Software Developers Meeting: http://www.fosdem.org


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
I looked it up, it was bug #1416344, and was fixed for 07.02 release on 2015-01-30, a long time ago.

That was going from 0.96 to 1.0.  We're on 2.01 now and all is well.

...Ken

On Wed, May 24, 2017 at 10:34 AM, Kenneth Loafman <[hidden email]> wrote:
ede,

1) librsync is good up thru version 2.x.  Not sure exactly what version of duplicity it was fixed in, but the logs should have the bug number and we can track it from there.  It's been a long while since it was fixed, perhaps all the way back to the last of the 0.6-series.  We still use the smaller signatures and I'm going to leave that as an option going forwards.  The larger signatures will double the size of the sigtar file, and will force a full backup if we go to them.

2) The decryption/encryption issues during restart and incrementals are:
  • Do we leave the cache unencrypted?  Without the manifest in plain form, or without the ability to decrypt it, we can't continue.
  • How do we decrypt if the cache is out of sync with the remote?  It happens and the remote manifest needs to be decrypted.
I see no really good reason to encrypt the manifest since it only contains information readily available from the system itself.  However, on the remote it needs to be encrypted, otherwise it could reveal the filenames of the backed up files.  Some folks are sensitive to that.

...Ken



On Wed, May 24, 2017 at 9:47 AM, <[hidden email]> wrote:
Ken,

how is the state wrt. librsync patch, didn't see it in the repo so far?

*and*, below is once again somebody complaining that suddenly a private key / passphrase is needed. would you agree that we should remove the dirty workaround described in
  https://bugs.launchpad.net/duplicity/+bug/687295
for duplicity 0.8?

it really only "works" with english locale and a backup that does not try to resume. still users risk to do backups where archive dir is not synchronized, which is bad for their data.

..ede

-------- Forwarded Message --------
Subject:        Re: [Duplicity-talk] gpg key password asked for backup after verify
Date:   Wed, 24 May 2017 16:14:31 +0200
From:   Raphael Bauduin <[hidden email]>
To:     [hidden email]





On Wed, May 24, 2017 at 1:49 PM, <[hidden email] <mailto:[hidden email]>> wrote:

    whoops, hit send too fast. read on below.

    On <a href="tel:24.05.2017%2013" value="+12405201713" target="_blank">24.05.2017 13 <tel:24.05.2017%2013>:17, Raphael Bauduin wrote:
    >
    >
    > On Wed, May 24, 2017 at 12:19 PM, edgar.soldin--- via Duplicity-talk <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
    >
    >     On <a href="tel:24.05.2017%2011" value="+12405201711" target="_blank">24.05.2017 11 <tel:24.05.2017%2011> <tel:24.05.2017%2011>:28, Raphael Bauduin via Duplicity-talk wrote:
    >     > Hi,
    >     >
    >     > I had encrypted backups working fine for weeks on a server. As the encryption uses the public key, it doesn't ask for a password.
    >     >
    >     > Then I did a duplicity verify, which requires the gpg private key, and asks for a password.
    >     > The verify went fine, but since then the gpg key password is also asked for backups, preventing the automation.... I'm nearly sure this is linked
    >     >
    >     > I have removed the duplicity cache in ~/.cache/duplicity, but to no avail....
    >     >
    >     > Any suggestion?
    >     >
    >
    >     1.
    >     are you using duply?
    >
    >
    > no
    >
    >
    >
    >     2.
    >     what is your backup command line?
    >
    >
    >  LC_ALL=en_US /bin/duplicity   inc --encrypt-key 'XXXX' --exclude /root/.cache/duplicity --exclude  /home/backups --exclude /home/restore --exclude /backups  --include /home/sftp --include /etc --include /home --include /root --exclude '**' / par2+rsync://rsync/duplicity/   --verbosity debug
    >
    >
    >
    >
    >     3.
    >     what's the language locale of your os?
    >
    >
    > I'm forcing it to en_US, which worked fine.
    >
    > Investigating further, I think I might have deleted the cache before I did the verify. So not sure which one causes what.
    > I took a look at the code. Here is the code in question asking for the password when the cache was empty, where I added a print:
    >             if local_missing and (rem_needpass or loc_needpass):
    >                 if decrypt:
    >                     # password for the --encrypt-key
    >                     print "local_missing = %s,--  %s, -- %s" % (local_missing, rem_needpass, loc_needpass)
    >                     globals.gpg_profile.passphrase = get_passphrase(1, "sync")
    >
    > local_missing was a set of .sigtar.gpg files, rem_needpass was True and loc_needpass was False.
    >
    > Now I have done a backup manually (providing the key password), I have the else clause below asking for the password although the action is inc:
    >
    >     elif (action == "inc" and
    >           (globals.gpg_profile.recipients or globals.gpg_profile.hidden_recipients) and not
    >           globals.gpg_profile.sign_key and not globals.restart):
    >         return ""
    >
    >     # Finally, ask the user for the passphrase
    >     else:
    >         print "action = %s" % action
    >         log.Info(_("PASSPHRASE variable not set, asking user."))
    >         use_cache = True
    >
    >
    > globals.gpg_profile.recipients is my encryption key id, globals.gpg_profile.sign_key is None, but globals.restart= <__main__.Restart instance at 0x13a8518>
    >
    > So it seems that the globals.restart is set and makes the code skip the action == "inc" part.
    >
    > Any idea what the problem might be?
    >
    > Thanks
    >

    ok, your backup is restarting. restarting happens when a backup was interrupted. restarting _needs_ to decrypt some information from the backend, which can only be done w/ priv key and passphrase of course.


Just confirm that's what I had. Making a full successful backup makes it run fine without password prompt on subsequent runs.

Thanks a lot for your help!





    what you ran into here is essentially this bug
      https://bugs.launchpad.net/duplicity/+bug/687295 <https://bugs.launchpad.net/duplicity/+bug/687295>

    consider using two key pairs in the future. duplicity using gpg can encrypt to multiple keys. place

    A. a sec/pub key for the box
    B. your own pub key

    in your keyring. then backup against both keys and optionally use the machine key to sign your backups. this way the box can decrypt when needed w/o needing your very secret personal private key.

    ..ede/duply.net <http://duply.net>




--
Web database: http://www.myowndb.com
Free Software Developers Meeting: http://www.fosdem.org



_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
wow, a misunderstanding on both points, that's a new record. :)) let me try again

wrt. 1.
i meant the "too many open files" bug solved by Howie. didn't see it in trunk so far. ok found it now
  http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/revision/1230#duplicity/patchdir.py
patchdir.py still contains

499            if sys.platform.startswith(('cygwin', 'windows')):
500                tempfp = os.tmpfile()
501            else:

though, which should be removed as it potentially creates temp files outside the given duplicity temp folder.

wrt. 2.
somewhere in the thread "backup without private key"
  https://bugs.launchpad.net/duplicity/+bug/687295
i explain how this can be achieved, but that it is dangerous and therefor generally not advisable. admittedly i also spitball some alternative solutions, but think now that they are no precondition to remove this behaviour.
my point now is, that we agreed accept incompatibility to previous versions when we are incrementing the minor version eg. 0.7->0.8. why not simply remove the "exception" (dirty hack) that ignores decryption error when synchronizing the archive folder? people can achieve the same functionality safely by using an additional machine key pair thus not needing their supersecret private key.

sunny sundaily greetings ..ede/duply.net

On 24.05.2017 22:04, Kenneth Loafman wrote:

> I looked it up, it was bug #1416344
> <https://bugs.launchpad.net/duplicity/+bug/1416344>, and was fixed for
> 07.02 release on 2015-01-30, a long time ago.
>
> That was going from 0.96 to 1.0.  We're on 2.01 now and all is well.
>
> ...Ken
>
> On Wed, May 24, 2017 at 10:34 AM, Kenneth Loafman <[hidden email]>
> wrote:
>
>> ede,
>>
>> 1) librsync is good up thru version 2.x.  Not sure exactly what version of
>> duplicity it was fixed in, but the logs should have the bug number and we
>> can track it from there.  It's been a long while since it was fixed,
>> perhaps all the way back to the last of the 0.6-series.  We still use the
>> smaller signatures and I'm going to leave that as an option going
>> forwards.  The larger signatures will double the size of the sigtar file,
>> and will force a full backup if we go to them.
>>
>> 2) The decryption/encryption issues during restart and incrementals are:
>>
>>    - Do we leave the cache unencrypted?  Without the manifest in plain
>>    form, or without the ability to decrypt it, we can't continue.
>>    - How do we decrypt if the cache is out of sync with the remote?  It
>>    happens and the remote manifest needs to be decrypted.
>>
>> I see no really good reason to encrypt the manifest since it only contains
>> information readily available from the system itself.  However, on the
>> remote it needs to be encrypted, otherwise it could reveal the filenames of
>> the backed up files.  Some folks are sensitive to that.
>>
>> ...Ken
>>
>>
>>
>> On Wed, May 24, 2017 at 9:47 AM, <[hidden email]> wrote:
>>
>>> Ken,
>>>
>>> how is the state wrt. librsync patch, didn't see it in the repo so far?
>>>
>>> *and*, below is once again somebody complaining that suddenly a private
>>> key / passphrase is needed. would you agree that we should remove the dirty
>>> workaround described in
>>>   https://bugs.launchpad.net/duplicity/+bug/687295
>>> for duplicity 0.8?
>>>
>>> it really only "works" with english locale and a backup that does not try
>>> to resume. still users risk to do backups where archive dir is not
>>> synchronized, which is bad for their data.
>>>
>>> ..ede
>>>
>>> -------- Forwarded Message --------
>>> Subject:        Re: [Duplicity-talk] gpg key password asked for backup
>>> after verify
>>> Date:   Wed, 24 May 2017 16:14:31 +0200
>>> From:   Raphael Bauduin <[hidden email]>
>>> To:     [hidden email]
>>>
>>>
>>>
>>>
>>>
>>> On Wed, May 24, 2017 at 1:49 PM, <[hidden email] <mailto:
>>> [hidden email]>> wrote:
>>>
>>>     whoops, hit send too fast. read on below.
>>>
>>>     On 24.05.2017 13 <tel:24.05.2017%2013>:17, Raphael Bauduin wrote:
>>>     >
>>>     >
>>>     > On Wed, May 24, 2017 at 12:19 PM, edgar.soldin--- via
>>> Duplicity-talk <[hidden email] <mailto:duplicity-talk@nongnu.
>>> org> <mailto:[hidden email] <mailto:[hidden email]>>>
>>> wrote:
>>>     >
>>>     >     On 24.05.2017 11 <tel:24.05.2017%2011>
>>> <tel:24.05.2017%2011>:28, Raphael Bauduin via Duplicity-talk wrote:
>>>     >     > Hi,
>>>     >     >
>>>     >     > I had encrypted backups working fine for weeks on a server.
>>> As the encryption uses the public key, it doesn't ask for a password.
>>>     >     >
>>>     >     > Then I did a duplicity verify, which requires the gpg private
>>> key, and asks for a password.
>>>     >     > The verify went fine, but since then the gpg key password is
>>> also asked for backups, preventing the automation.... I'm nearly sure this
>>> is linked
>>>     >     >
>>>     >     > I have removed the duplicity cache in ~/.cache/duplicity, but
>>> to no avail....
>>>     >     >
>>>     >     > Any suggestion?
>>>     >     >
>>>     >
>>>     >     1.
>>>     >     are you using duply?
>>>     >
>>>     >
>>>     > no
>>>     >
>>>     >
>>>     >
>>>     >     2.
>>>     >     what is your backup command line?
>>>     >
>>>     >
>>>     >  LC_ALL=en_US /bin/duplicity   inc --encrypt-key 'XXXX' --exclude
>>> /root/.cache/duplicity --exclude  /home/backups --exclude /home/restore
>>> --exclude /backups  --include /home/sftp --include /etc --include /home
>>> --include /root --exclude '**' / par2+rsync://rsync/duplicity/
>>>  --verbosity debug
>>>     >
>>>     >
>>>     >
>>>     >
>>>     >     3.
>>>     >     what's the language locale of your os?
>>>     >
>>>     >
>>>     > I'm forcing it to en_US, which worked fine.
>>>     >
>>>     > Investigating further, I think I might have deleted the cache
>>> before I did the verify. So not sure which one causes what.
>>>     > I took a look at the code. Here is the code in question asking for
>>> the password when the cache was empty, where I added a print:
>>>     >             if local_missing and (rem_needpass or loc_needpass):
>>>     >                 if decrypt:
>>>     >                     # password for the --encrypt-key
>>>     >                     print "local_missing = %s,--  %s, -- %s" %
>>> (local_missing, rem_needpass, loc_needpass)
>>>     >                     globals.gpg_profile.passphrase =
>>> get_passphrase(1, "sync")
>>>     >
>>>     > local_missing was a set of .sigtar.gpg files, rem_needpass was True
>>> and loc_needpass was False.
>>>     >
>>>     > Now I have done a backup manually (providing the key password), I
>>> have the else clause below asking for the password although the action is
>>> inc:
>>>     >
>>>     >     elif (action == "inc" and
>>>     >           (globals.gpg_profile.recipients or
>>> globals.gpg_profile.hidden_recipients) and not
>>>     >           globals.gpg_profile.sign_key and not globals.restart):
>>>     >         return ""
>>>     >
>>>     >     # Finally, ask the user for the passphrase
>>>     >     else:
>>>     >         print "action = %s" % action
>>>     >         log.Info(_("PASSPHRASE variable not set, asking user."))
>>>     >         use_cache = True
>>>     >
>>>     >
>>>     > globals.gpg_profile.recipients is my encryption key id,
>>> globals.gpg_profile.sign_key is None, but globals.restart=
>>> <__main__.Restart instance at 0x13a8518>
>>>     >
>>>     > So it seems that the globals.restart is set and makes the code skip
>>> the action == "inc" part.
>>>     >
>>>     > Any idea what the problem might be?
>>>     >
>>>     > Thanks
>>>     >
>>>
>>>     ok, your backup is restarting. restarting happens when a backup was
>>> interrupted. restarting _needs_ to decrypt some information from the
>>> backend, which can only be done w/ priv key and passphrase of course.
>>>
>>>
>>> Just confirm that's what I had. Making a full successful backup makes it
>>> run fine without password prompt on subsequent runs.
>>>
>>> Thanks a lot for your help!
>>>
>>>
>>>
>>>
>>>
>>>     what you ran into here is essentially this bug
>>>       https://bugs.launchpad.net/duplicity/+bug/687295 <
>>> https://bugs.launchpad.net/duplicity/+bug/687295>
>>>
>>>     consider using two key pairs in the future. duplicity using gpg can
>>> encrypt to multiple keys. place
>>>
>>>     A. a sec/pub key for the box
>>>     B. your own pub key
>>>
>>>     in your keyring. then backup against both keys and optionally use the
>>> machine key to sign your backups. this way the box can decrypt when needed
>>> w/o needing your very secret personal private key.
>>>
>>>     ..ede/duply.net <http://duply.net>
>>>
>>>
>>>
>>>
>>> --
>>> Web database: http://www.myowndb.com
>>> Free Software Developers Meeting: http://www.fosdem.org
>>>
>>
>>
>


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
OK, let me try again:

1) the TemporaryFile() problems in windows and cygwin have been with us for a long while and it has not been fixed in Python, yet.  The first patch that Howie suggested added darwin to that list, and that was removed by the next commit when he found the problem in _librsyncmodule.c.  I don't know how to fix the problem in Windows, so that issue should probably be handled in the man page and README.  People don't normally partition Windows the same way as Linux, so I doubt it's a real problem for most users.

2) In my response, I outlined why we have problems when we don't have the private key.  Perhaps a solution would be to add a file in the cache called duplicity-md5sums.txt that would contain the md5sum of each file in the cache.  As long as that matched, and the cache was unencrypted, we would not need to hit the remote, except to list filenames.  Thoughts?  As to ignoring errors in the sync between local and remote, I can't even imagine the number of ways we'd get into troubles.

...Ken


On Sun, May 28, 2017 at 7:37 AM, <[hidden email]> wrote:
wow, a misunderstanding on both points, that's a new record. :)) let me try again

wrt. 1.
i meant the "too many open files" bug solved by Howie. didn't see it in trunk so far. ok found it now
  http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/revision/1230#duplicity/patchdir.py
patchdir.py still contains

499            if sys.platform.startswith(('cygwin', 'windows')):
500                tempfp = os.tmpfile()
501            else:

though, which should be removed as it potentially creates temp files outside the given duplicity temp folder.

wrt. 2.
somewhere in the thread "backup without private key"
  https://bugs.launchpad.net/duplicity/+bug/687295
i explain how this can be achieved, but that it is dangerous and therefor generally not advisable. admittedly i also spitball some alternative solutions, but think now that they are no precondition to remove this behaviour.
my point now is, that we agreed accept incompatibility to previous versions when we are incrementing the minor version eg. 0.7->0.8. why not simply remove the "exception" (dirty hack) that ignores decryption error when synchronizing the archive folder? people can achieve the same functionality safely by using an additional machine key pair thus not needing their supersecret private key.

sunny sundaily greetings ..ede/duply.net

On <a href="tel:24.05.2017%2022" value="+12405201722">24.05.2017 22:04, Kenneth Loafman wrote:
> I looked it up, it was bug #1416344
> <https://bugs.launchpad.net/duplicity/+bug/1416344>, and was fixed for
> 07.02 release on 2015-01-30, a long time ago.
>
> That was going from 0.96 to 1.0.  We're on 2.01 now and all is well.
>
> ...Ken
>
> On Wed, May 24, 2017 at 10:34 AM, Kenneth Loafman <[hidden email]>
> wrote:
>
>> ede,
>>
>> 1) librsync is good up thru version 2.x.  Not sure exactly what version of
>> duplicity it was fixed in, but the logs should have the bug number and we
>> can track it from there.  It's been a long while since it was fixed,
>> perhaps all the way back to the last of the 0.6-series.  We still use the
>> smaller signatures and I'm going to leave that as an option going
>> forwards.  The larger signatures will double the size of the sigtar file,
>> and will force a full backup if we go to them.
>>
>> 2) The decryption/encryption issues during restart and incrementals are:
>>
>>    - Do we leave the cache unencrypted?  Without the manifest in plain
>>    form, or without the ability to decrypt it, we can't continue.
>>    - How do we decrypt if the cache is out of sync with the remote?  It
>>    happens and the remote manifest needs to be decrypted.
>>
>> I see no really good reason to encrypt the manifest since it only contains
>> information readily available from the system itself.  However, on the
>> remote it needs to be encrypted, otherwise it could reveal the filenames of
>> the backed up files.  Some folks are sensitive to that.
>>
>> ...Ken
>>
>>
>>
>> On Wed, May 24, 2017 at 9:47 AM, <[hidden email]> wrote:
>>
>>> Ken,
>>>
>>> how is the state wrt. librsync patch, didn't see it in the repo so far?
>>>
>>> *and*, below is once again somebody complaining that suddenly a private
>>> key / passphrase is needed. would you agree that we should remove the dirty
>>> workaround described in
>>>   https://bugs.launchpad.net/duplicity/+bug/687295
>>> for duplicity 0.8?
>>>
>>> it really only "works" with english locale and a backup that does not try
>>> to resume. still users risk to do backups where archive dir is not
>>> synchronized, which is bad for their data.
>>>
>>> ..ede
>>>
>>> -------- Forwarded Message --------
>>> Subject:        Re: [Duplicity-talk] gpg key password asked for backup
>>> after verify
>>> Date:   Wed, 24 May 2017 16:14:31 +0200
>>> From:   Raphael Bauduin <[hidden email]>
>>> To:     [hidden email]
>>>
>>>
>>>
>>>
>>>
>>> On Wed, May 24, 2017 at 1:49 PM, <[hidden email] <mailto:
>>> [hidden email]>> wrote:
>>>
>>>     whoops, hit send too fast. read on below.
>>>
>>>     On <a href="tel:24.05.2017%2013" value="+12405201713">24.05.2017 13 <tel:24.05.2017%2013>:17, Raphael Bauduin wrote:
>>>     >
>>>     >
>>>     > On Wed, May 24, 2017 at 12:19 PM, edgar.soldin--- via
>>> Duplicity-talk <[hidden email] <mailto:[hidden email].
>>> org> <mailto:[hidden email] <mailto:[hidden email]>>>
>>> wrote:
>>>     >
>>>     >     On <a href="tel:24.05.2017%2011" value="+12405201711">24.05.2017 11 <tel:24.05.2017%2011>
>>> <tel:24.05.2017%2011>:28, Raphael Bauduin via Duplicity-talk wrote:
>>>     >     > Hi,
>>>     >     >
>>>     >     > I had encrypted backups working fine for weeks on a server.
>>> As the encryption uses the public key, it doesn't ask for a password.
>>>     >     >
>>>     >     > Then I did a duplicity verify, which requires the gpg private
>>> key, and asks for a password.
>>>     >     > The verify went fine, but since then the gpg key password is
>>> also asked for backups, preventing the automation.... I'm nearly sure this
>>> is linked
>>>     >     >
>>>     >     > I have removed the duplicity cache in ~/.cache/duplicity, but
>>> to no avail....
>>>     >     >
>>>     >     > Any suggestion?
>>>     >     >
>>>     >
>>>     >     1.
>>>     >     are you using duply?
>>>     >
>>>     >
>>>     > no
>>>     >
>>>     >
>>>     >
>>>     >     2.
>>>     >     what is your backup command line?
>>>     >
>>>     >
>>>     >  LC_ALL=en_US /bin/duplicity   inc --encrypt-key 'XXXX' --exclude
>>> /root/.cache/duplicity --exclude  /home/backups --exclude /home/restore
>>> --exclude /backups  --include /home/sftp --include /etc --include /home
>>> --include /root --exclude '**' / par2+rsync://rsync/duplicity/
>>>  --verbosity debug
>>>     >
>>>     >
>>>     >
>>>     >
>>>     >     3.
>>>     >     what's the language locale of your os?
>>>     >
>>>     >
>>>     > I'm forcing it to en_US, which worked fine.
>>>     >
>>>     > Investigating further, I think I might have deleted the cache
>>> before I did the verify. So not sure which one causes what.
>>>     > I took a look at the code. Here is the code in question asking for
>>> the password when the cache was empty, where I added a print:
>>>     >             if local_missing and (rem_needpass or loc_needpass):
>>>     >                 if decrypt:
>>>     >                     # password for the --encrypt-key
>>>     >                     print "local_missing = %s,--  %s, -- %s" %
>>> (local_missing, rem_needpass, loc_needpass)
>>>     >                     globals.gpg_profile.passphrase =
>>> get_passphrase(1, "sync")
>>>     >
>>>     > local_missing was a set of .sigtar.gpg files, rem_needpass was True
>>> and loc_needpass was False.
>>>     >
>>>     > Now I have done a backup manually (providing the key password), I
>>> have the else clause below asking for the password although the action is
>>> inc:
>>>     >
>>>     >     elif (action == "inc" and
>>>     >           (globals.gpg_profile.recipients or
>>> globals.gpg_profile.hidden_recipients) and not
>>>     >           globals.gpg_profile.sign_key and not globals.restart):
>>>     >         return ""
>>>     >
>>>     >     # Finally, ask the user for the passphrase
>>>     >     else:
>>>     >         print "action = %s" % action
>>>     >         log.Info(_("PASSPHRASE variable not set, asking user."))
>>>     >         use_cache = True
>>>     >
>>>     >
>>>     > globals.gpg_profile.recipients is my encryption key id,
>>> globals.gpg_profile.sign_key is None, but globals.restart=
>>> <__main__.Restart instance at 0x13a8518>
>>>     >
>>>     > So it seems that the globals.restart is set and makes the code skip
>>> the action == "inc" part.
>>>     >
>>>     > Any idea what the problem might be?
>>>     >
>>>     > Thanks
>>>     >
>>>
>>>     ok, your backup is restarting. restarting happens when a backup was
>>> interrupted. restarting _needs_ to decrypt some information from the
>>> backend, which can only be done w/ priv key and passphrase of course.
>>>
>>>
>>> Just confirm that's what I had. Making a full successful backup makes it
>>> run fine without password prompt on subsequent runs.
>>>
>>> Thanks a lot for your help!
>>>
>>>
>>>
>>>
>>>
>>>     what you ran into here is essentially this bug
>>>       https://bugs.launchpad.net/duplicity/+bug/687295 <
>>> https://bugs.launchpad.net/duplicity/+bug/687295>
>>>
>>>     consider using two key pairs in the future. duplicity using gpg can
>>> encrypt to multiple keys. place
>>>
>>>     A. a sec/pub key for the box
>>>     B. your own pub key
>>>
>>>     in your keyring. then backup against both keys and optionally use the
>>> machine key to sign your backups. this way the box can decrypt when needed
>>> w/o needing your very secret personal private key.
>>>
>>>     ..ede/duply.net <http://duply.net>
>>>
>>>
>>>
>>>
>>> --
>>> Web database: http://www.myowndb.com
>>> Free Software Developers Meeting: http://www.fosdem.org
>>>
>>
>>
>



_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
hey Ken,

On 28.05.2017 18:48, Kenneth Loafman wrote:
> OK, let me try again:

:)

> 1) the TemporaryFile() problems in windows and cygwin have been with us for a long while and it has not been fixed in Python, yet.  The first patch that Howie suggested added darwin to that list, and that was removed by the next commit when he found the problem in _librsyncmodule.c.  I don't know how to fix the problem in Windows, so that issue should probably be handled in the man page and README.  People don't normally partition Windows the same way as Linux, so I doubt it's a real problem for most users.

can you be more specific what the issue/bug is on windows? do you have a link, bug report or such?

> 2) In my response, I outlined why we have problems when we don't have the private key.  Perhaps a solution would be to add a file in the cache called /duplicity-md5sums.txt/ that would contain the md5sum of each file in the cache.  As long as that matched, and the cache was unencrypted, we would not need to hit the remote, except to list filenames.  Thoughts?  As to ignoring errors in the sync between local and remote, I can't even imagine the number of ways we'd get into troubles.

backups w/o private key currently only work because they _actually do_ "ignoring errors in the sync between local and remote", that's why i want to remove that "functionality"!
as explained, until we implement unencrypted but signed file hashes(or lists) for backend files, users will not miss any functionality as they can already today switch from

bad) backup w/o private key (archive sync decrypt errors are ignored)
  to
good) backup w/o private key by encrypting against a public personal key _and_ a machine specific full key pair

. ..ede


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
1) This was from the README for 0.7.05:
* Fix bug #1494228 CygWin: TypeError: basis_file must be a (true) file
  - The problem that caused the change to tempfile.TemporaryFile was due
    to the fact that os.tmpfile always creates its file in the system
    temp directory, not in the directory specified.  The fix applied was
    to use os.tmpfile in cygwin/windows and tempfile.TemporaryFile in all
    the rest.  This means that cygwin is now broken with respect to temp
    file placement of this one file (deleted automatically on close).

2) I'm not seeing that we ignore errors in the sync between local and remote.  That would produce bad backups if we did.

...Ken


On Mon, May 29, 2017 at 4:15 AM, edgar.soldin--- via Duplicity-talk <[hidden email]> wrote:
hey Ken,

On 28.05.2017 18:48, Kenneth Loafman wrote:
> OK, let me try again:

:)

> 1) the TemporaryFile() problems in windows and cygwin have been with us for a long while and it has not been fixed in Python, yet.  The first patch that Howie suggested added darwin to that list, and that was removed by the next commit when he found the problem in _librsyncmodule.c.  I don't know how to fix the problem in Windows, so that issue should probably be handled in the man page and README.  People don't normally partition Windows the same way as Linux, so I doubt it's a real problem for most users.

can you be more specific what the issue/bug is on windows? do you have a link, bug report or such?

> 2) In my response, I outlined why we have problems when we don't have the private key.  Perhaps a solution would be to add a file in the cache called /duplicity-md5sums.txt/ that would contain the md5sum of each file in the cache.  As long as that matched, and the cache was unencrypted, we would not need to hit the remote, except to list filenames.  Thoughts?  As to ignoring errors in the sync between local and remote, I can't even imagine the number of ways we'd get into troubles.

backups w/o private key currently only work because they _actually do_ "ignoring errors in the sync between local and remote", that's why i want to remove that "functionality"!
as explained, until we implement unencrypted but signed file hashes(or lists) for backend files, users will not miss any functionality as they can already today switch from

bad) backup w/o private key (archive sync decrypt errors are ignored)
  to
good) backup w/o private key by encrypting against a public personal key _and_ a machine specific full key pair

. ..ede


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
Ken,

On 29.05.2017 13:26, Kenneth Loafman wrote:
> 2) I'm not seeing that we ignore errors in the sync between local and remote.  That would produce bad backups if we did.

that's where you are wrong ;)
  http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/annotate/head%3A/duplicity/collections.py#L241

i found it in 2010 in the more detailed thread about this issue
  http://thread.gmane.org/gmane.comp.sysutils.backup.duplicity.general/4245

..ede

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
ede,

Thanks for the links.  I completely forgot about all that.  Yes, we need to fix it.

...Ken


On Mon, May 29, 2017 at 6:40 AM, edgar.soldin--- via Duplicity-talk <[hidden email]> wrote:
Ken,

On 29.05.2017 13:26, Kenneth Loafman wrote:
> 2) I'm not seeing that we ignore errors in the sync between local and remote.  That would produce bad backups if we did.

that's where you are wrong ;)
  http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/annotate/head%3A/duplicity/collections.py#L241

i found it in 2010 in the more detailed thread about this issue
  http://thread.gmane.org/gmane.comp.sysutils.backup.duplicity.general/4245

..ede

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
nP.. as written, 0.8 would be a good chance to simply remove "missing secret key" exception as a first step, as we can argue backward compatibility breakage _but_ more importantly backup security improvement!

..ede

On 29.05.2017 14:05, Kenneth Loafman wrote:

> ede,
>
> Thanks for the links.  I completely forgot about all that.  Yes, we need to fix it.
>
> ...Ken
>
>
> On Mon, May 29, 2017 at 6:40 AM, edgar.soldin--- via Duplicity-talk <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Ken,
>
>     On 29.05.2017 13:26, Kenneth Loafman wrote:
>     > 2) I'm not seeing that we ignore errors in the sync between local and remote.  That would produce bad backups if we did.
>
>     that's where you are wrong ;)
>       http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/annotate/head%3A/duplicity/collections.py#L241 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8-series/annotate/head%3A/duplicity/collections.py#L241>
>
>     i found it in 2010 in the more detailed thread about this issue
>       http://thread.gmane.org/gmane.comp.sysutils.backup.duplicity.general/4245 <http://thread.gmane.org/gmane.comp.sysutils.backup.duplicity.general/4245>
>
>     ..ede
>
>     _______________________________________________
>     Duplicity-talk mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>
>
>


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
On 29.05.2017 13:26, Kenneth Loafman wrote:

> 1) This was from the README for 0.7.05:
>
>         * Fix bug #1494228 CygWin: TypeError: basis_file must be a (true) file
>
>           - The problem that caused the change to tempfile.TemporaryFile was due
>
>             to the fact that os.tmpfile always creates its file in the system
>
>             temp directory, not in the directory specified.  The fix applied was
>
>             to use os.tmpfile in cygwin/windows and tempfile.TemporaryFile in all
>
>             the rest.  This means that cygwin is now broken with respect to temp
>
>             file placement of this one file (deleted automatically on close).

wrt. os.tmpfile() .. afaics. does

   https://hg.python.org/cpython/file/2.7/Lib/tempfile.py#l484

use the os module ( os.open, os.fdopen ) all around to create temp files. aren't these supposed to return a (true) file? the docs say nothing in this regard. https://docs.python.org/2/library/os.html#file-object-creation

..ede


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
Ken,

On 29.05.2017 14:27, edgar.soldin--- via Duplicity-talk wrote:

> On 29.05.2017 13:26, Kenneth Loafman wrote:
>> 1) This was from the README for 0.7.05:
>>
>>         * Fix bug #1494228 CygWin: TypeError: basis_file must be a (true) file
>>
>>           - The problem that caused the change to tempfile.TemporaryFile was due
>>
>>             to the fact that os.tmpfile always creates its file in the system
>>
>>             temp directory, not in the directory specified.  The fix applied was
>>
>>             to use os.tmpfile in cygwin/windows and tempfile.TemporaryFile in all
>>
>>             the rest.  This means that cygwin is now broken with respect to temp
>>
>>             file placement of this one file (deleted automatically on close).
>
> wrt. os.tmpfile() .. afaics. does
>
>    https://hg.python.org/cpython/file/2.7/Lib/tempfile.py#l484
>
> use the os module ( os.open, os.fdopen ) all around to create temp files. aren't these supposed to return a (true) file? the docs say nothing in this regard. https://docs.python.org/2/library/os.html#file-object-creation
>
> ..ede
>

ok answering myself here, after reading the original bug report thoroughly, which says
"
The problem is that, as explained in the Python documentation (http://docs.python.org/library/tempfile.html), tempfile.TemporaryFile returns a *file-like* object, not a file object. Under Linux, it effectively returns a file object but under windows it does not. Hence, the type of the object returned is *not* a types.FileType, causing the exception to be raised.
"
  https://bugs.launchpad.net/duplicity/+bug/670891

further reading shows that the error stems from 'duplicity/librsync.py'
"
  File "/usr/lib/python2.6/site-packages/duplicity/librsync.py", line 167, in __init__
    raise TypeError("basis_file must be a (true) file")
"


so obviously something is done differently in the 'duplicity/_librsyncmodule.c', that needs a delta patch file to be a plain File object. let's see 'duplicity/librsync.py' says
"
class PatchedFile(LikeFile):
    """File-like object which applies a librsync delta incrementally"""
    def __init__(self, basis_file, delta_file):
        """PatchedFile initializer - call with basis delta

        Here basis_file must be a true Python file, because we may
        need to seek() around in it a lot, and this is done in C.
        delta_file only needs read() and close() methods.

        """
        LikeFile.__init__(self, delta_file)
        if not isinstance(basis_file, types.FileType):
            raise TypeError("basis_file must be a (true) file")
        try:
            self.maker = _librsync.new_patchmaker(basis_file)
        except _librsync.librsyncError as e:
            raise librsyncError(str(e))
"
http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/duplicity/librsync.py#L163


and in 'duplicity/_librsyncmodule.c' it tries to utilize 'PyObject_AsFileDescriptor(python_file)'
"
/* Call with the basis file */
static PyObject*
_librsync_new_patchmaker(PyObject* self, PyObject* args)
{
  _librsync_PatchMakerObject* pm;
  PyObject *python_file;
  int fd;

  if (!PyArg_ParseTuple(args, "O:new_patchmaker", &python_file))
    return NULL;
  fd = PyObject_AsFileDescriptor(python_file);
"
http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/duplicity/_librsyncmodule.c#L294


so, if i understand correctly, we just have to implement the method 'fileno()' as described here
  https://docs.python.org/2/c-api/object.html#c.PyObject_AsFileDescriptor
in another wrapper around the tempfile.TemporaryFile() return value
  https://docs.python.org/2/library/tempfile.html#tempfile.TemporaryFile
, utilizing (if necessary) the file-like object's file attribute as mentioned in the python doc and return
"
The returned object is a true file object on POSIX platforms. On other platforms, it is a file-like object whose file attribute is the underlying true file object.
"

implementing that should render the failing TypeError("basis_file must be a (true) file") in 'duplicity/librsync.py' meaningless, so it could be removed or?

alternatively, we could simply extend 'duplicity/librsync.py', to test for a file attribute, in case the "file" object given is just a wrapper. on success we then simply deliver that (pseudo code)
"
self.maker = _librsync.new_patchmaker(basis_file.file)
"

what do you think Ken? would that work? which approach would you prefer? ..ede

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
I don't know what the underlying code looks like for non-POSIX systems, and I don't have a Windows/Cygwin system to test against, so my answer would be that I would prefer it be done in _librsyncmodule.c since that seems to be the only place that needs it.  Failing that, librsync.py is my second preference since it's only one level up.

We would still leave the error check  on line #305 since we could have made a different error.  We would need to set the error message accordingly.

...Ken


On Mon, May 29, 2017 at 8:47 AM, edgar.soldin--- via Duplicity-talk <[hidden email]> wrote:
Ken,

On 29.05.2017 14:27, edgar.soldin--- via Duplicity-talk wrote:
> On 29.05.2017 13:26, Kenneth Loafman wrote:
>> 1) This was from the README for 0.7.05:
>>
>>         * Fix bug #1494228 CygWin: TypeError: basis_file must be a (true) file
>>
>>           - The problem that caused the change to tempfile.TemporaryFile was due
>>
>>             to the fact that os.tmpfile always creates its file in the system
>>
>>             temp directory, not in the directory specified.  The fix applied was
>>
>>             to use os.tmpfile in cygwin/windows and tempfile.TemporaryFile in all
>>
>>             the rest.  This means that cygwin is now broken with respect to temp
>>
>>             file placement of this one file (deleted automatically on close).
>
> wrt. os.tmpfile() .. afaics. does
>
>    https://hg.python.org/cpython/file/2.7/Lib/tempfile.py#l484
>
> use the os module ( os.open, os.fdopen ) all around to create temp files. aren't these supposed to return a (true) file? the docs say nothing in this regard. https://docs.python.org/2/library/os.html#file-object-creation
>
> ..ede
>

ok answering myself here, after reading the original bug report thoroughly, which says
"
The problem is that, as explained in the Python documentation (http://docs.python.org/library/tempfile.html), tempfile.TemporaryFile returns a *file-like* object, not a file object. Under Linux, it effectively returns a file object but under windows it does not. Hence, the type of the object returned is *not* a types.FileType, causing the exception to be raised.
"
  https://bugs.launchpad.net/duplicity/+bug/670891

further reading shows that the error stems from 'duplicity/librsync.py'
"
  File "/usr/lib/python2.6/site-packages/duplicity/librsync.py", line 167, in __init__
    raise TypeError("basis_file must be a (true) file")
"


so obviously something is done differently in the 'duplicity/_librsyncmodule.c', that needs a delta patch file to be a plain File object. let's see 'duplicity/librsync.py' says
"
class PatchedFile(LikeFile):
    """File-like object which applies a librsync delta incrementally"""
    def __init__(self, basis_file, delta_file):
        """PatchedFile initializer - call with basis delta

        Here basis_file must be a true Python file, because we may
        need to seek() around in it a lot, and this is done in C.
        delta_file only needs read() and close() methods.

        """
        LikeFile.__init__(self, delta_file)
        if not isinstance(basis_file, types.FileType):
            raise TypeError("basis_file must be a (true) file")
        try:
            self.maker = _librsync.new_patchmaker(basis_file)
        except _librsync.librsyncError as e:
            raise librsyncError(str(e))
"
http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/duplicity/librsync.py#L163


and in 'duplicity/_librsyncmodule.c' it tries to utilize 'PyObject_AsFileDescriptor(python_file)'
"
/* Call with the basis file */
static PyObject*
_librsync_new_patchmaker(PyObject* self, PyObject* args)
{
  _librsync_PatchMakerObject* pm;
  PyObject *python_file;
  int fd;

  if (!PyArg_ParseTuple(args, "O:new_patchmaker", &python_file))
    return NULL;
  fd = PyObject_AsFileDescriptor(python_file);
"
http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/duplicity/_librsyncmodule.c#L294


so, if i understand correctly, we just have to implement the method 'fileno()' as described here
  https://docs.python.org/2/c-api/object.html#c.PyObject_AsFileDescriptor
in another wrapper around the tempfile.TemporaryFile() return value
  https://docs.python.org/2/library/tempfile.html#tempfile.TemporaryFile
, utilizing (if necessary) the file-like object's file attribute as mentioned in the python doc and return
"
The returned object is a true file object on POSIX platforms. On other platforms, it is a file-like object whose file attribute is the underlying true file object.
"

implementing that should render the failing TypeError("basis_file must be a (true) file") in 'duplicity/librsync.py' meaningless, so it could be removed or?

alternatively, we could simply extend 'duplicity/librsync.py', to test for a file attribute, in case the "file" object given is just a wrapper. on success we then simply deliver that (pseudo code)
"
self.maker = _librsync.new_patchmaker(basis_file.file)
"

what do you think Ken? would that work? which approach would you prefer? ..ede

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
hey Ken,

i tested the following on a local cygwin installation and it patched away fine :)

'duplicity/librsync.py' line 159+
"
class PatchedFile(LikeFile):
    """File-like object which applies a librsync delta incrementally"""
    def __init__(self, basis_file, delta_file):
        """PatchedFile initializer - call with basis delta

        Here basis_file must be a true Python file, because we may
        need to seek() around in it a lot, and this is done in C.
        delta_file only needs read() and close() methods.

        """
        LikeFile.__init__(self, delta_file)
        if not isinstance(basis_file, types.FileType):
            if hasattr(basis_file, 'file') and isinstance(basis_file.file, types.FileType):
                basis_file = basis_file.file
            else:
                raise TypeError("basis_file must be a (true) file or an object whose file attribute is the underlying true file object")
        try:
            self.maker = _librsync.new_patchmaker(basis_file)
        except _librsync.librsyncError as e:
            raise librsyncError(str(e))
"

is that ok w/ you? do you need a branch?  ..ede

On 29.05.2017 16:47, Kenneth Loafman wrote:

> I don't know what the underlying code looks like for non-POSIX systems, and I don't have a Windows/Cygwin system to test against, so my answer would be that I would /prefer/ it be done in /_librsyncmodule.c/ since that seems to be the only place that needs it.  Failing that, /librsync.py/ is my second preference since it's only one level up.
>
> We would still leave the error check  on line #305 since we could have made a different error.  We would need to set the error message accordingly.
>
> ...Ken
>
>
> On Mon, May 29, 2017 at 8:47 AM, edgar.soldin--- via Duplicity-talk <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Ken,
>
>     On 29.05.2017 14:27, edgar.soldin--- via Duplicity-talk wrote:
>     > On 29.05.2017 13:26, Kenneth Loafman wrote:
>     >> 1) This was from the README for 0.7.05:
>     >>
>     >>         * Fix bug #1494228 CygWin: TypeError: basis_file must be a (true) file
>     >>
>     >>           - The problem that caused the change to tempfile.TemporaryFile was due
>     >>
>     >>             to the fact that os.tmpfile always creates its file in the system
>     >>
>     >>             temp directory, not in the directory specified.  The fix applied was
>     >>
>     >>             to use os.tmpfile in cygwin/windows and tempfile.TemporaryFile in all
>     >>
>     >>             the rest.  This means that cygwin is now broken with respect to temp
>     >>
>     >>             file placement of this one file (deleted automatically on close).
>     >
>     > wrt. os.tmpfile() .. afaics. does
>     >
>     >    https://hg.python.org/cpython/file/2.7/Lib/tempfile.py#l484 <https://hg.python.org/cpython/file/2.7/Lib/tempfile.py#l484>
>     >
>     > use the os module ( os.open, os.fdopen ) all around to create temp files. aren't these supposed to return a (true) file? the docs say nothing in this regard. https://docs.python.org/2/library/os.html#file-object-creation <https://docs.python.org/2/library/os.html#file-object-creation>
>     >
>     > ..ede
>     >
>
>     ok answering myself here, after reading the original bug report thoroughly, which says
>     "
>     The problem is that, as explained in the Python documentation (http://docs.python.org/library/tempfile.html <http://docs.python.org/library/tempfile.html>), tempfile.TemporaryFile returns a *file-like* object, not a file object. Under Linux, it effectively returns a file object but under windows it does not. Hence, the type of the object returned is *not* a types.FileType, causing the exception to be raised.
>     "
>       https://bugs.launchpad.net/duplicity/+bug/670891 <https://bugs.launchpad.net/duplicity/+bug/670891>
>
>     further reading shows that the error stems from 'duplicity/librsync.py'
>     "
>       File "/usr/lib/python2.6/site-packages/duplicity/librsync.py", line 167, in __init__
>         raise TypeError("basis_file must be a (true) file")
>     "
>
>
>     so obviously something is done differently in the 'duplicity/_librsyncmodule.c', that needs a delta patch file to be a plain File object. let's see 'duplicity/librsync.py' says
>     "
>     class PatchedFile(LikeFile):
>         """File-like object which applies a librsync delta incrementally"""
>         def __init__(self, basis_file, delta_file):
>             """PatchedFile initializer - call with basis delta
>
>             Here basis_file must be a true Python file, because we may
>             need to seek() around in it a lot, and this is done in C.
>             delta_file only needs read() and close() methods.
>
>             """
>             LikeFile.__init__(self, delta_file)
>             if not isinstance(basis_file, types.FileType):
>                 raise TypeError("basis_file must be a (true) file")
>             try:
>                 self.maker = _librsync.new_patchmaker(basis_file)
>             except _librsync.librsyncError as e:
>                 raise librsyncError(str(e))
>     "
>     http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/duplicity/librsync.py#L163 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8-series/view/head:/duplicity/librsync.py#L163>
>
>
>     and in 'duplicity/_librsyncmodule.c' it tries to utilize 'PyObject_AsFileDescriptor(python_file)'
>     "
>     /* Call with the basis file */
>     static PyObject*
>     _librsync_new_patchmaker(PyObject* self, PyObject* args)
>     {
>       _librsync_PatchMakerObject* pm;
>       PyObject *python_file;
>       int fd;
>
>       if (!PyArg_ParseTuple(args, "O:new_patchmaker", &python_file))
>         return NULL;
>       fd = PyObject_AsFileDescriptor(python_file);
>     "
>     http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/duplicity/_librsyncmodule.c#L294 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8-series/view/head:/duplicity/_librsyncmodule.c#L294>
>
>
>     so, if i understand correctly, we just have to implement the method 'fileno()' as described here
>       https://docs.python.org/2/c-api/object.html#c.PyObject_AsFileDescriptor <https://docs.python.org/2/c-api/object.html#c.PyObject_AsFileDescriptor>
>     in another wrapper around the tempfile.TemporaryFile() return value
>       https://docs.python.org/2/library/tempfile.html#tempfile.TemporaryFile <https://docs.python.org/2/library/tempfile.html#tempfile.TemporaryFile>
>     , utilizing (if necessary) the file-like object's file attribute as mentioned in the python doc and return
>     "
>     The returned object is a true file object on POSIX platforms. On other platforms, it is a file-like object whose file attribute is the underlying true file object.
>     "
>
>     implementing that should render the failing TypeError("basis_file must be a (true) file") in 'duplicity/librsync.py' meaningless, so it could be removed or?
>
>     alternatively, we could simply extend 'duplicity/librsync.py', to test for a file attribute, in case the "file" object given is just a wrapper. on success we then simply deliver that (pseudo code)
>     "
>     self.maker = _librsync.new_patchmaker(basis_file.file)
>     "
>
>     what do you think Ken? would that work? which approach would you prefer? ..ede
>
>     _______________________________________________
>     Duplicity-talk mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>
>
>


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
Thanks ede!

No, I do not need a patch.  Will get it in ASAP.

...Ken


On Wed, May 31, 2017 at 10:32 AM, edgar.soldin--- via Duplicity-talk <[hidden email]> wrote:
hey Ken,

i tested the following on a local cygwin installation and it patched away fine :)

'duplicity/librsync.py' line 159+
"
class PatchedFile(LikeFile):
    """File-like object which applies a librsync delta incrementally"""
    def __init__(self, basis_file, delta_file):
        """PatchedFile initializer - call with basis delta

        Here basis_file must be a true Python file, because we may
        need to seek() around in it a lot, and this is done in C.
        delta_file only needs read() and close() methods.

        """
        LikeFile.__init__(self, delta_file)
        if not isinstance(basis_file, types.FileType):
            if hasattr(basis_file, 'file') and isinstance(basis_file.file, types.FileType):
                basis_file = basis_file.file
            else:
                raise TypeError("basis_file must be a (true) file or an object whose file attribute is the underlying true file object")
        try:
            self.maker = _librsync.new_patchmaker(basis_file)
        except _librsync.librsyncError as e:
            raise librsyncError(str(e))
"

is that ok w/ you? do you need a branch?  ..ede

On 29.05.2017 16:47, Kenneth Loafman wrote:
> I don't know what the underlying code looks like for non-POSIX systems, and I don't have a Windows/Cygwin system to test against, so my answer would be that I would /prefer/ it be done in /_librsyncmodule.c/ since that seems to be the only place that needs it.  Failing that, /librsync.py/ is my second preference since it's only one level up.
>
> We would still leave the error check  on line #305 since we could have made a different error.  We would need to set the error message accordingly.
>
> ...Ken
>
>
> On Mon, May 29, 2017 at 8:47 AM, edgar.soldin--- via Duplicity-talk <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Ken,
>
>     On 29.05.2017 14:27, edgar.soldin--- via Duplicity-talk wrote:
>     > On 29.05.2017 13:26, Kenneth Loafman wrote:
>     >> 1) This was from the README for 0.7.05:
>     >>
>     >>         * Fix bug #1494228 CygWin: TypeError: basis_file must be a (true) file
>     >>
>     >>           - The problem that caused the change to tempfile.TemporaryFile was due
>     >>
>     >>             to the fact that os.tmpfile always creates its file in the system
>     >>
>     >>             temp directory, not in the directory specified.  The fix applied was
>     >>
>     >>             to use os.tmpfile in cygwin/windows and tempfile.TemporaryFile in all
>     >>
>     >>             the rest.  This means that cygwin is now broken with respect to temp
>     >>
>     >>             file placement of this one file (deleted automatically on close).
>     >
>     > wrt. os.tmpfile() .. afaics. does
>     >
>     >    https://hg.python.org/cpython/file/2.7/Lib/tempfile.py#l484 <https://hg.python.org/cpython/file/2.7/Lib/tempfile.py#l484>
>     >
>     > use the os module ( os.open, os.fdopen ) all around to create temp files. aren't these supposed to return a (true) file? the docs say nothing in this regard. https://docs.python.org/2/library/os.html#file-object-creation <https://docs.python.org/2/library/os.html#file-object-creation>
>     >
>     > ..ede
>     >
>
>     ok answering myself here, after reading the original bug report thoroughly, which says
>     "
>     The problem is that, as explained in the Python documentation (http://docs.python.org/library/tempfile.html <http://docs.python.org/library/tempfile.html>), tempfile.TemporaryFile returns a *file-like* object, not a file object. Under Linux, it effectively returns a file object but under windows it does not. Hence, the type of the object returned is *not* a types.FileType, causing the exception to be raised.
>     "
>       https://bugs.launchpad.net/duplicity/+bug/670891 <https://bugs.launchpad.net/duplicity/+bug/670891>
>
>     further reading shows that the error stems from 'duplicity/librsync.py'
>     "
>       File "/usr/lib/python2.6/site-packages/duplicity/librsync.py", line 167, in __init__
>         raise TypeError("basis_file must be a (true) file")
>     "
>
>
>     so obviously something is done differently in the 'duplicity/_librsyncmodule.c', that needs a delta patch file to be a plain File object. let's see 'duplicity/librsync.py' says
>     "
>     class PatchedFile(LikeFile):
>         """File-like object which applies a librsync delta incrementally"""
>         def __init__(self, basis_file, delta_file):
>             """PatchedFile initializer - call with basis delta
>
>             Here basis_file must be a true Python file, because we may
>             need to seek() around in it a lot, and this is done in C.
>             delta_file only needs read() and close() methods.
>
>             """
>             LikeFile.__init__(self, delta_file)
>             if not isinstance(basis_file, types.FileType):
>                 raise TypeError("basis_file must be a (true) file")
>             try:
>                 self.maker = _librsync.new_patchmaker(basis_file)
>             except _librsync.librsyncError as e:
>                 raise librsyncError(str(e))
>     "
>     http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/duplicity/librsync.py#L163 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8-series/view/head:/duplicity/librsync.py#L163>
>
>
>     and in 'duplicity/_librsyncmodule.c' it tries to utilize 'PyObject_AsFileDescriptor(python_file)'
>     "
>     /* Call with the basis file */
>     static PyObject*
>     _librsync_new_patchmaker(PyObject* self, PyObject* args)
>     {
>       _librsync_PatchMakerObject* pm;
>       PyObject *python_file;
>       int fd;
>
>       if (!PyArg_ParseTuple(args, "O:new_patchmaker", &python_file))
>         return NULL;
>       fd = PyObject_AsFileDescriptor(python_file);
>     "
>     http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/duplicity/_librsyncmodule.c#L294 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8-series/view/head:/duplicity/_librsyncmodule.c#L294>
>
>
>     so, if i understand correctly, we just have to implement the method 'fileno()' as described here
>       https://docs.python.org/2/c-api/object.html#c.PyObject_AsFileDescriptor <https://docs.python.org/2/c-api/object.html#c.PyObject_AsFileDescriptor>
>     in another wrapper around the tempfile.TemporaryFile() return value
>       https://docs.python.org/2/library/tempfile.html#tempfile.TemporaryFile <https://docs.python.org/2/library/tempfile.html#tempfile.TemporaryFile>
>     , utilizing (if necessary) the file-like object's file attribute as mentioned in the python doc and return
>     "
>     The returned object is a true file object on POSIX platforms. On other platforms, it is a file-like object whose file attribute is the underlying true file object.
>     "
>
>     implementing that should render the failing TypeError("basis_file must be a (true) file") in 'duplicity/librsync.py' meaningless, so it could be removed or?
>
>     alternatively, we could simply extend 'duplicity/librsync.py', to test for a file attribute, in case the "file" object given is just a wrapper. on success we then simply deliver that (pseudo code)
>     "
>     self.maker = _librsync.new_patchmaker(basis_file.file)
>     "
>
>     what do you think Ken? would that work? which approach would you prefer? ..ede
>
>     _______________________________________________
>     Duplicity-talk mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>
>
>


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
nice.. let me beautify it a bit still _and_ add the patchdir.py change ;) .. the line numbers are from 0.7.12 btw.

'duplicity/librsync.py' line 159+
"
class PatchedFile(LikeFile):
    """File-like object which applies a librsync delta incrementally"""
    def __init__(self, basis_file, delta_file):
        """PatchedFile initializer - call with basis delta

        Here basis_file must be a true Python file, because we may
        need to seek() around in it a lot, and this is done in C.
        delta_file only needs read() and close() methods.

        """
        LikeFile.__init__(self, delta_file)
        if not isinstance(basis_file, types.FileType):
            """ tempfile.TemporaryFile() only guarantees a true file
            object on posix platforms. on cygwin/windows a file-like
            object whose file attribute is the underlying true file
            object is returned.
            """
            if hasattr(basis_file, 'file') and isinstance(basis_file.file, types.FileType):
                basis_file = basis_file.file
            else:
                raise TypeError("basis_file must be a (true) file or an object whose file attribute is the underlying true file object")
        try:
            self.maker = _librsync.new_patchmaker(basis_file)
        except _librsync.librsyncError as e:
            raise librsyncError(str(e))
"

'duplicity/patchdir.py' line 489+
"
    for delta_ropath in patch_seq[1:]:
        assert delta_ropath.difftype == "diff", delta_ropath.difftype
        if not isinstance(current_file, file):
            """
            librsync insists on a real file object, which we create manually
            by using the duplicity.tempdir to tell us where.

            See https://bugs.launchpad.net/duplicity/+bug/670891 for discussion
            of os.tmpfile() vs tempfile.TemporaryFile() w.r.t. Windows / Posix,
            which is worked around in librsync.PatchedFile() now.
            """
            tempfp = tempfile.TemporaryFile(dir=tempdir.default().dir())
            util.copyfileobj(current_file, tempfp)
            assert not current_file.close()
            tempfp.seek(0)
            current_file = tempfp
        current_file = librsync.PatchedFile(current_file,
                                            delta_ropath.open("rb"))
    result = patch_seq[-1].get_ropath()
    result.setfileobj(current_file)
    return result
"

sunny fluffy cloudy greetings.. ede

On 31.05.2017 17:40, Kenneth Loafman wrote:

> Thanks ede!
>
> No, I do not need a patch.  Will get it in ASAP.
>
> ...Ken
>
>
> On Wed, May 31, 2017 at 10:32 AM, edgar.soldin--- via Duplicity-talk <[hidden email] <mailto:[hidden email]>> wrote:
>
>     hey Ken,
>
>     i tested the following on a local cygwin installation and it patched away fine :)
>
>     'duplicity/librsync.py' line 159+
>     "
>     class PatchedFile(LikeFile):
>         """File-like object which applies a librsync delta incrementally"""
>         def __init__(self, basis_file, delta_file):
>             """PatchedFile initializer - call with basis delta
>
>             Here basis_file must be a true Python file, because we may
>             need to seek() around in it a lot, and this is done in C.
>             delta_file only needs read() and close() methods.
>
>             """
>             LikeFile.__init__(self, delta_file)
>             if not isinstance(basis_file, types.FileType):
>                 if hasattr(basis_file, 'file') and isinstance(basis_file.file, types.FileType):
>                     basis_file = basis_file.file
>                 else:
>                     raise TypeError("basis_file must be a (true) file or an object whose file attribute is the underlying true file object")
>             try:
>                 self.maker = _librsync.new_patchmaker(basis_file)
>             except _librsync.librsyncError as e:
>                 raise librsyncError(str(e))
>     "
>
>     is that ok w/ you? do you need a branch?  ..ede
>
>     On 29.05.2017 16:47, Kenneth Loafman wrote:
>     > I don't know what the underlying code looks like for non-POSIX systems, and I don't have a Windows/Cygwin system to test against, so my answer would be that I would /prefer/ it be done in /_librsyncmodule.c/ since that seems to be the only place that needs it.  Failing that, /librsync.py/ <http://librsync.py/> is my second preference since it's only one level up.
>     >
>     > We would still leave the error check  on line #305 since we could have made a different error.  We would need to set the error message accordingly.
>     >
>     > ...Ken
>     >
>     >
>     > On Mon, May 29, 2017 at 8:47 AM, edgar.soldin--- via Duplicity-talk <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>     >
>     >     Ken,
>     >
>     >     On 29.05.2017 14:27, edgar.soldin--- via Duplicity-talk wrote:
>     >     > On 29.05.2017 13:26, Kenneth Loafman wrote:
>     >     >> 1) This was from the README for 0.7.05:
>     >     >>
>     >     >>         * Fix bug #1494228 CygWin: TypeError: basis_file must be a (true) file
>     >     >>
>     >     >>           - The problem that caused the change to tempfile.TemporaryFile was due
>     >     >>
>     >     >>             to the fact that os.tmpfile always creates its file in the system
>     >     >>
>     >     >>             temp directory, not in the directory specified.  The fix applied was
>     >     >>
>     >     >>             to use os.tmpfile in cygwin/windows and tempfile.TemporaryFile in all
>     >     >>
>     >     >>             the rest.  This means that cygwin is now broken with respect to temp
>     >     >>
>     >     >>             file placement of this one file (deleted automatically on close).
>     >     >
>     >     > wrt. os.tmpfile() .. afaics. does
>     >     >
>     >     >    https://hg.python.org/cpython/file/2.7/Lib/tempfile.py#l484 <https://hg.python.org/cpython/file/2.7/Lib/tempfile.py#l484> <https://hg.python.org/cpython/file/2.7/Lib/tempfile.py#l484 <https://hg.python.org/cpython/file/2.7/Lib/tempfile.py#l484>>
>     >     >
>     >     > use the os module ( os.open, os.fdopen ) all around to create temp files. aren't these supposed to return a (true) file? the docs say nothing in this regard. https://docs.python.org/2/library/os.html#file-object-creation <https://docs.python.org/2/library/os.html#file-object-creation> <https://docs.python.org/2/library/os.html#file-object-creation <https://docs.python.org/2/library/os.html#file-object-creation>>
>     >     >
>     >     > ..ede
>     >     >
>     >
>     >     ok answering myself here, after reading the original bug report thoroughly, which says
>     >     "
>     >     The problem is that, as explained in the Python documentation (http://docs.python.org/library/tempfile.html <http://docs.python.org/library/tempfile.html> <http://docs.python.org/library/tempfile.html <http://docs.python.org/library/tempfile.html>>), tempfile.TemporaryFile returns a *file-like* object, not a file object. Under Linux, it effectively returns a file object but under windows it does not. Hence, the type of the object returned is *not* a types.FileType, causing the exception to be raised.
>     >     "
>     >       https://bugs.launchpad.net/duplicity/+bug/670891 <https://bugs.launchpad.net/duplicity/+bug/670891> <https://bugs.launchpad.net/duplicity/+bug/670891 <https://bugs.launchpad.net/duplicity/+bug/670891>>
>     >
>     >     further reading shows that the error stems from 'duplicity/librsync.py'
>     >     "
>     >       File "/usr/lib/python2.6/site-packages/duplicity/librsync.py", line 167, in __init__
>     >         raise TypeError("basis_file must be a (true) file")
>     >     "
>     >
>     >
>     >     so obviously something is done differently in the 'duplicity/_librsyncmodule.c', that needs a delta patch file to be a plain File object. let's see 'duplicity/librsync.py' says
>     >     "
>     >     class PatchedFile(LikeFile):
>     >         """File-like object which applies a librsync delta incrementally"""
>     >         def __init__(self, basis_file, delta_file):
>     >             """PatchedFile initializer - call with basis delta
>     >
>     >             Here basis_file must be a true Python file, because we may
>     >             need to seek() around in it a lot, and this is done in C.
>     >             delta_file only needs read() and close() methods.
>     >
>     >             """
>     >             LikeFile.__init__(self, delta_file)
>     >             if not isinstance(basis_file, types.FileType):
>     >                 raise TypeError("basis_file must be a (true) file")
>     >             try:
>     >                 self.maker = _librsync.new_patchmaker(basis_file)
>     >             except _librsync.librsyncError as e:
>     >                 raise librsyncError(str(e))
>     >     "
>     >     http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/duplicity/librsync.py#L163 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8-series/view/head:/duplicity/librsync.py#L163> <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8-series/view/head:/duplicity/librsync.py#L163 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8-series/view/head:/duplicity/librsync.py#L163>>
>     >
>     >
>     >     and in 'duplicity/_librsyncmodule.c' it tries to utilize 'PyObject_AsFileDescriptor(python_file)'
>     >     "
>     >     /* Call with the basis file */
>     >     static PyObject*
>     >     _librsync_new_patchmaker(PyObject* self, PyObject* args)
>     >     {
>     >       _librsync_PatchMakerObject* pm;
>     >       PyObject *python_file;
>     >       int fd;
>     >
>     >       if (!PyArg_ParseTuple(args, "O:new_patchmaker", &python_file))
>     >         return NULL;
>     >       fd = PyObject_AsFileDescriptor(python_file);
>     >     "
>     >     http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/duplicity/_librsyncmodule.c#L294 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8-series/view/head:/duplicity/_librsyncmodule.c#L294> <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8-series/view/head:/duplicity/_librsyncmodule.c#L294 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8-series/view/head:/duplicity/_librsyncmodule.c#L294>>
>     >
>     >
>     >     so, if i understand correctly, we just have to implement the method 'fileno()' as described here
>     >       https://docs.python.org/2/c-api/object.html#c.PyObject_AsFileDescriptor <https://docs.python.org/2/c-api/object.html#c.PyObject_AsFileDescriptor> <https://docs.python.org/2/c-api/object.html#c.PyObject_AsFileDescriptor <https://docs.python.org/2/c-api/object.html#c.PyObject_AsFileDescriptor>>
>     >     in another wrapper around the tempfile.TemporaryFile() return value
>     >       https://docs.python.org/2/library/tempfile.html#tempfile.TemporaryFile <https://docs.python.org/2/library/tempfile.html#tempfile.TemporaryFile> <https://docs.python.org/2/library/tempfile.html#tempfile.TemporaryFile <https://docs.python.org/2/library/tempfile.html#tempfile.TemporaryFile>>
>     >     , utilizing (if necessary) the file-like object's file attribute as mentioned in the python doc and return
>     >     "
>     >     The returned object is a true file object on POSIX platforms. On other platforms, it is a file-like object whose file attribute is the underlying true file object.
>     >     "
>     >
>     >     implementing that should render the failing TypeError("basis_file must be a (true) file") in 'duplicity/librsync.py' meaningless, so it could be removed or?
>     >
>     >     alternatively, we could simply extend 'duplicity/librsync.py', to test for a file attribute, in case the "file" object given is just a wrapper. on success we then simply deliver that (pseudo code)
>     >     "
>     >     self.maker = _librsync.new_patchmaker(basis_file.file)
>     >     "
>     >
>     >     what do you think Ken? would that work? which approach would you prefer? ..ede
>     >
>     >     _______________________________________________
>     >     Duplicity-talk mailing list
>     >     [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>
>     >     https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>>
>     >
>     >
>
>
>     _______________________________________________
>     Duplicity-talk mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>
>
>


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
Ken,

On 20.07.2017 14:03, Kenneth Loafman wrote:
> Hi ede,
>
> OK, I implemented it so it will output a non-fatal error message on gpg fails with remote manifest files, return None, and continue.  I ignored the code from the previous message because the try/except was removed completely.  Tests are passing and I'll get it released soon.  

it's a start, but we need to decide if this warrants a fatal failure or not!
as long as there is no way to compare the remote manifest w/ the local one w/o decryption i'd say it is. actually it is already fatal to anyone not using an english locale ;)

btw. my suggestion was to make it fatal in 0.8 as the version raise might warrant major changes.

generally we need to realize that backing up to dumb backends either needs decryption for sync from time to time or implement a way to signal both location's files are identical. how to do that safely w/o proper decryption/signing is beyond me.
that's why i say, duplicity usage needs at least one secret key/message and if it is only a secret machine key used to sign stuff.

> We need to continue this discussion on the list

done. cc'd the list.

>We keep introducing potentially fatal issues.

exactly where do we "keep introducing" them? :)

sunny greetings ..ede/duply.net


>
> ...Ken
>
>
> On Wed, Jul 19, 2017 at 4:44 AM, <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Ken,
>
>     because now the try/except has a condition that leads it _not to_ fail at all [1], but to silently return no manifest, even though there is one, but it cannot be decrypted w/o the secret key.
>
>     seen?.. ede
>
>     [1]  when gpg spits out "secret key not available" in english
>
>     On 18.07.2017 23:02, Kenneth Loafman wrote:
>     > ede,
>     >
>     > I still don't understand... removing the exception does exactly nothing except cause it to fail in a different and more confusing manner, so why bother?
>     >
>     > ...Ken
>     >
>     >
>     > On Tue, Jul 18, 2017 at 3:40 PM, <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>     >
>     >     Ken,
>     >
>     >     kind of academic, but ok,
>     >
>     >     1. the local cache might have been deleted
>     >     2. it might have been tampered with
>     >     3. some file system condition might have reverted the cache to a previous state
>     >     4. ...
>     >
>     >     stuff like that. the solution to use a local machine secret/public key seems to be much more appealing than the assumption that we are synchronized in case we do not have a secret key to decrypt.
>     >
>     >     also agn, we will need the secret key in case of resuming anyway or the backup will fail indefinitely until a new chain is started or a key is given.
>     >
>     >     ..ede
>     >
>     >     On 7/18/2017 10:32 PM, Kenneth Loafman wrote:
>     >     > ede,
>     >     >
>     >     > It's going to raise an error if it can't read/decrypt the manifest, either
>     >     > in a handled exception, or an unhandled one.
>     >     >
>     >     > How could duplicity write to a backend with a more recent manifest?  The
>     >     > lockfile guarantees there is only one writer at a time.
>     >     >
>     >     > ...Ken
>     >     >
>     >     >
>     >     > On Tue, Jul 18, 2017 at 3:23 PM, <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>     >     >
>     >     >> Ken,
>     >     >>
>     >     >> how do you imagine to resolve this cleanly w/o raising an error? the
>     >     >> moment a user writes a backup to a backend where a more recent manifest
>     >     >> already resides, the chain is corrupted, no?
>     >     >>
>     >     >> ..ede
>     >     >>
>     >     >> On 7/18/2017 10:18 PM, Kenneth Loafman wrote:
>     >     >>> ede,
>     >     >>>
>     >     >>> Without going back through this convoluted email, why are we removing the
>     >     >>> exception rather than fixing it?  gpg has defined return codes and it
>     >     >> seems
>     >     >>> that would be the way to go?
>     >     >>>
>     >     >>> ...Ken
>     >     >>>
>     >     >>> On Tue, Jul 18, 2017 at 10:07 AM, <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>     >     >>>
>     >     >>>> you could edit it manually
>     >     >>>>   >collections.py#get_remote_manifest() L232
>     >     >>>>
>     >     >>>>     def get_remote_manifest(self):
>     >     >>>>         """
>     >     >>>>         Return manifest by reading remote manifest on backend
>     >     >>>>         """
>     >     >>>>         assert self.remote_manifest_name
>     >     >>>>         manifest_buffer = self.backend.get_data(self.
>     >     >> remote_manifest_name)
>     >     >>>>         log.Info(_("Processing remote manifest %s (%s)") %
>     >     >>>> (self.remote_manifest_name, len(manifest_buffer)))
>     >     >>>>         return manifest.Manifest().from_string(manifest_buffer)
>     >     >>>>
>     >     >>>> sufficient? ..ede
>     >     >>>>
>     >     >>>> On 18.07.2017 17:01, Kenneth Loafman wrote:
>     >     >>>>> Hi ede,
>     >     >>>>>
>     >     >>>>> I thought this was fixed earlier?
>     >     >>>>>
>     >     >>>>> Could you provide a "bzr diff collections.py > collections.py.patch".
>     >     >> I
>     >     >>>> don't need a branch for something like this.
>     >     >>>>>
>     >     >>>>> ...Thanks,
>     >     >>>>> ...Ken
>     >     >>>>>
>     >     >>>>>
>     >     >>>>> On Tue, Jul 18, 2017 at 9:13 AM, <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:
>     >     >>>> [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>> wrote:
>     >     >>>>>
>     >     >>>>>     hey Ken,
>     >     >>>>>
>     >     >>>>>     do you need a branch? or can you remove it on the fly? afaics the
>     >     >>>> hack is still in there
>     >     >>>>>       http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0> <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0>>.
>     >     >>>> 8-series/annotate/head%3A/duplicity/collections.py#L241 <
>     >     >>>> http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0> <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0>>.
>     >     >>>> 8-series/annotate/head%3A/duplicity/collections.py#L241>
>     >     >>>>>
>     >     >>>>>     ..ede
>     >     >>>>>
>     >     >>>>>     -------- Forwarded Message --------
>     >     >>>>>     Subject: Re: [Duplicity-talk] gpg key password asked for backup
>     >     >>>> after verify
>     >     >>>>>     Date: Mon, 29 May 2017 14:08:14 +0200
>     >     >>>>>     From: edgar.soldin--- via Duplicity-talk <
>     >     >> [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>
>     >     >>>> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>>
>     >     >>>>>     Reply-To: Discussion about duplicity backup <
>     >     >>>> [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>>
>     >     >>>>>     To: Kenneth Loafman <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:
>     >     >> [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>>,
>     >     >>>> Discussion about duplicity backup <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:
>     >     >>>> [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>>
>     >     >>>>>
>     >     >>>>>     nP.. as written, 0.8 would be a good chance to simply remove
>     >     >>>> "missing secret key" exception as a first step, as we can argue backward
>     >     >>>> compatibility breakage _but_ more importantly backup security
>     >     >> improvement!
>     >     >>>>>
>     >     >>>>>     ..ede
>     >     >>>>>
>     >     >>>>>     On 29.05.2017 14:05, Kenneth Loafman wrote:
>     >     >>>>>     > ede,
>     >     >>>>>     >
>     >     >>>>>     > Thanks for the links.  I completely forgot about all that.  Yes,
>     >     >>>> we need to fix it.
>     >     >>>>>     >
>     >     >>>>>     > ...Ken
>     >     >>>>>     >
>     >     >>>>>     >
>     >     >>>>>     > On Mon, May 29, 2017 at 6:40 AM, edgar.soldin--- via
>     >     >>>> Duplicity-talk <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:duplicity-talk@nongnu <mailto:duplicity-talk@nongnu> <mailto:duplicity-talk@nongnu <mailto:duplicity-talk@nongnu>>
>     >     >> .
>     >     >>>> org> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:duplicity-talk@nongnu <mailto:duplicity-talk@nongnu> <mailto:duplicity-talk@nongnu <mailto:duplicity-talk@nongnu>>.
>     >     >> org>>>
>     >     >>>> wrote:
>     >     >>>>>     >
>     >     >>>>>     >     Ken,
>     >     >>>>>     >
>     >     >>>>>     >     On 29.05.2017 13:26, Kenneth Loafman wrote:
>     >     >>>>>     >     > 2) I'm not seeing that we ignore errors in the sync between
>     >     >>>> local and remote.  That would produce bad backups if we did.
>     >     >>>>>     >
>     >     >>>>>     >     that's where you are wrong ;)
>     >     >>>>>     >       http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8- <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8-> <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8- <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8->>
>     >     >>>> series/annotate/head%3A/duplicity/collections.py#L241 <
>     >     >>>> http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0> <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0>>.
>     >     >>>> 8-series/annotate/head%3A/duplicity/collections.py#L241> <
>     >     >>>> http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0> <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0>>.
>     >     >>>> 8-series/annotate/head%3A/duplicity/collections.py#L241 <
>     >     >>>> http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0> <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0>>.
>     >     >>>> 8-series/annotate/head%3A/duplicity/collections.py#L241>>
>     >     >>>>>     >
>     >     >>>>>     >     i found it in 2010 in the more detailed thread about this
>     >     >> issue
>     >     >>>>>     >       http://thread.gmane.org/gmane.comp.sysutils.backup <http://thread.gmane.org/gmane.comp.sysutils.backup> <http://thread.gmane.org/gmane.comp.sysutils.backup <http://thread.gmane.org/gmane.comp.sysutils.backup>>.
>     >     >>>> duplicity.general/4245 <http://thread.gmane.org/
>     >     >>>> gmane.comp.sysutils.backup.duplicity.general/4245> <
>     >     >>>> http://thread.gmane.org/gmane.comp.sysutils.backup <http://thread.gmane.org/gmane.comp.sysutils.backup> <http://thread.gmane.org/gmane.comp.sysutils.backup <http://thread.gmane.org/gmane.comp.sysutils.backup>>.
>     >     >> duplicity.general/4245
>     >     >>>> <http://thread.gmane.org/gmane.comp.sysutils.backup <http://thread.gmane.org/gmane.comp.sysutils.backup> <http://thread.gmane.org/gmane.comp.sysutils.backup <http://thread.gmane.org/gmane.comp.sysutils.backup>>.
>     >     >> duplicity.general/4245
>     >     >>>>>>
>     >     >>>>>     >
>     >     >>>>>     >     ..ede
>     >     >>>>>     >
>     >     >>>>>     >     _______________________________________________
>     >     >>>>>     >     Duplicity-talk mailing list
>     >     >>>>>     >     [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>
>     >     >>>> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>>
>     >     >>>>>     >     https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>> <
>     >     >>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>>> <
>     >     >>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>> <
>     >     >>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>>>>
>     >     >>>>>     >
>     >     >>>>>     >
>     >     >>>>>
>     >     >>>>>
>     >     >>>>>     _______________________________________________
>     >     >>>>>     Duplicity-talk mailing list
>     >     >>>>>     [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>
>     >     >>>>>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>> <
>     >     >>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>>>
>     >     >>>>>
>     >     >>>>>
>     >     >>>>
>     >     >>>>
>     >     >>>
>     >     >>
>     >     >>
>     >     >
>     >
>     >
>
>


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg key password asked for backup after verify

duplicity-talk mailing list
ede,

To me, the absence of the private key on the backup machine is worthy of a fatal error, however, from the gist of this message, people were asking for it to not be so.  If we adopted the policy of a master key and a machine key we could force the issue to be fatal, but users won't like it.  I like your solution for multiple machines, but it's not simple enough for the average user.

The reason I made it non-fatal was to get rid of the English-only check we had in place.  I'm thinking of going to GPGME library and doing away with the external gpg.  We can rewrite the gpg.py module in duplicity and have a much cleaner environment and use return codes rather than error messages.

...Ken


On Thu, Jul 20, 2017 at 8:01 AM, <[hidden email]> wrote:
Ken,

On 20.07.2017 14:03, Kenneth Loafman wrote:
> Hi ede,
>
> OK, I implemented it so it will output a non-fatal error message on gpg fails with remote manifest files, return None, and continue.  I ignored the code from the previous message because the try/except was removed completely.  Tests are passing and I'll get it released soon.

it's a start, but we need to decide if this warrants a fatal failure or not!
as long as there is no way to compare the remote manifest w/ the local one w/o decryption i'd say it is. actually it is already fatal to anyone not using an english locale ;)

btw. my suggestion was to make it fatal in 0.8 as the version raise might warrant major changes.

generally we need to realize that backing up to dumb backends either needs decryption for sync from time to time or implement a way to signal both location's files are identical. how to do that safely w/o proper decryption/signing is beyond me.
that's why i say, duplicity usage needs at least one secret key/message and if it is only a secret machine key used to sign stuff.

> We need to continue this discussion on the list

done. cc'd the list.

>We keep introducing potentially fatal issues.

exactly where do we "keep introducing" them? :)

sunny greetings ..ede/duply.net


>
> ...Ken
>
>
> On Wed, Jul 19, 2017 at 4:44 AM, <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Ken,
>
>     because now the try/except has a condition that leads it _not to_ fail at all [1], but to silently return no manifest, even though there is one, but it cannot be decrypted w/o the secret key.
>
>     seen?.. ede
>
>     [1]  when gpg spits out "secret key not available" in english
>
>     On 18.07.2017 23:02, Kenneth Loafman wrote:
>     > ede,
>     >
>     > I still don't understand... removing the exception does exactly nothing except cause it to fail in a different and more confusing manner, so why bother?
>     >
>     > ...Ken
>     >
>     >
>     > On Tue, Jul 18, 2017 at 3:40 PM, <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>     >
>     >     Ken,
>     >
>     >     kind of academic, but ok,
>     >
>     >     1. the local cache might have been deleted
>     >     2. it might have been tampered with
>     >     3. some file system condition might have reverted the cache to a previous state
>     >     4. ...
>     >
>     >     stuff like that. the solution to use a local machine secret/public key seems to be much more appealing than the assumption that we are synchronized in case we do not have a secret key to decrypt.
>     >
>     >     also agn, we will need the secret key in case of resuming anyway or the backup will fail indefinitely until a new chain is started or a key is given.
>     >
>     >     ..ede
>     >
>     >     On 7/18/2017 10:32 PM, Kenneth Loafman wrote:
>     >     > ede,
>     >     >
>     >     > It's going to raise an error if it can't read/decrypt the manifest, either
>     >     > in a handled exception, or an unhandled one.
>     >     >
>     >     > How could duplicity write to a backend with a more recent manifest?  The
>     >     > lockfile guarantees there is only one writer at a time.
>     >     >
>     >     > ...Ken
>     >     >
>     >     >
>     >     > On Tue, Jul 18, 2017 at 3:23 PM, <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>     >     >
>     >     >> Ken,
>     >     >>
>     >     >> how do you imagine to resolve this cleanly w/o raising an error? the
>     >     >> moment a user writes a backup to a backend where a more recent manifest
>     >     >> already resides, the chain is corrupted, no?
>     >     >>
>     >     >> ..ede
>     >     >>
>     >     >> On 7/18/2017 10:18 PM, Kenneth Loafman wrote:
>     >     >>> ede,
>     >     >>>
>     >     >>> Without going back through this convoluted email, why are we removing the
>     >     >>> exception rather than fixing it?  gpg has defined return codes and it
>     >     >> seems
>     >     >>> that would be the way to go?
>     >     >>>
>     >     >>> ...Ken
>     >     >>>
>     >     >>> On Tue, Jul 18, 2017 at 10:07 AM, <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>     >     >>>
>     >     >>>> you could edit it manually
>     >     >>>>   >collections.py#get_remote_manifest() L232
>     >     >>>>
>     >     >>>>     def get_remote_manifest(self):
>     >     >>>>         """
>     >     >>>>         Return manifest by reading remote manifest on backend
>     >     >>>>         """
>     >     >>>>         assert self.remote_manifest_name
>     >     >>>>         manifest_buffer = self.backend.get_data(self.
>     >     >> remote_manifest_name)
>     >     >>>>         log.Info(_("Processing remote manifest %s (%s)") %
>     >     >>>> (self.remote_manifest_name, len(manifest_buffer)))
>     >     >>>>         return manifest.Manifest().from_string(manifest_buffer)
>     >     >>>>
>     >     >>>> sufficient? ..ede
>     >     >>>>
>     >     >>>> On 18.07.2017 17:01, Kenneth Loafman wrote:
>     >     >>>>> Hi ede,
>     >     >>>>>
>     >     >>>>> I thought this was fixed earlier?
>     >     >>>>>
>     >     >>>>> Could you provide a "bzr diff collections.py > collections.py.patch".
>     >     >> I
>     >     >>>> don't need a branch for something like this.
>     >     >>>>>
>     >     >>>>> ...Thanks,
>     >     >>>>> ...Ken
>     >     >>>>>
>     >     >>>>>
>     >     >>>>> On Tue, Jul 18, 2017 at 9:13 AM, <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:
>     >     >>>> [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>> wrote:
>     >     >>>>>
>     >     >>>>>     hey Ken,
>     >     >>>>>
>     >     >>>>>     do you need a branch? or can you remove it on the fly? afaics the
>     >     >>>> hack is still in there
>     >     >>>>>       http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0> <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0>>.
>     >     >>>> 8-series/annotate/head%3A/duplicity/collections.py#L241 <
>     >     >>>> http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0> <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0>>.
>     >     >>>> 8-series/annotate/head%3A/duplicity/collections.py#L241>
>     >     >>>>>
>     >     >>>>>     ..ede
>     >     >>>>>
>     >     >>>>>     -------- Forwarded Message --------
>     >     >>>>>     Subject: Re: [Duplicity-talk] gpg key password asked for backup
>     >     >>>> after verify
>     >     >>>>>     Date: Mon, 29 May 2017 14:08:14 +0200
>     >     >>>>>     From: edgar.soldin--- via Duplicity-talk <
>     >     >> [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>
>     >     >>>> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>>
>     >     >>>>>     Reply-To: Discussion about duplicity backup <
>     >     >>>> [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>>
>     >     >>>>>     To: Kenneth Loafman <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:
>     >     >> [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>>,
>     >     >>>> Discussion about duplicity backup <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:
>     >     >>>> [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>>
>     >     >>>>>
>     >     >>>>>     nP.. as written, 0.8 would be a good chance to simply remove
>     >     >>>> "missing secret key" exception as a first step, as we can argue backward
>     >     >>>> compatibility breakage _but_ more importantly backup security
>     >     >> improvement!
>     >     >>>>>
>     >     >>>>>     ..ede
>     >     >>>>>
>     >     >>>>>     On 29.05.2017 14:05, Kenneth Loafman wrote:
>     >     >>>>>     > ede,
>     >     >>>>>     >
>     >     >>>>>     > Thanks for the links.  I completely forgot about all that.  Yes,
>     >     >>>> we need to fix it.
>     >     >>>>>     >
>     >     >>>>>     > ...Ken
>     >     >>>>>     >
>     >     >>>>>     >
>     >     >>>>>     > On Mon, May 29, 2017 at 6:40 AM, edgar.soldin--- via
>     >     >>>> Duplicity-talk <[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>
>     >     >> .
>     >     >>>> org> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>.
>     >     >> org>>>
>     >     >>>> wrote:
>     >     >>>>>     >
>     >     >>>>>     >     Ken,
>     >     >>>>>     >
>     >     >>>>>     >     On 29.05.2017 13:26, Kenneth Loafman wrote:
>     >     >>>>>     >     > 2) I'm not seeing that we ignore errors in the sync between
>     >     >>>> local and remote.  That would produce bad backups if we did.
>     >     >>>>>     >
>     >     >>>>>     >     that's where you are wrong ;)
>     >     >>>>>     >       http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8- <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8-> <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8- <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0.8->>
>     >     >>>> series/annotate/head%3A/duplicity/collections.py#L241 <
>     >     >>>> http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0> <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0>>.
>     >     >>>> 8-series/annotate/head%3A/duplicity/collections.py#L241> <
>     >     >>>> http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0> <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0>>.
>     >     >>>> 8-series/annotate/head%3A/duplicity/collections.py#L241 <
>     >     >>>> http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0> <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0 <http://bazaar.launchpad.net/%7Eduplicity-team/duplicity/0>>.
>     >     >>>> 8-series/annotate/head%3A/duplicity/collections.py#L241>>
>     >     >>>>>     >
>     >     >>>>>     >     i found it in 2010 in the more detailed thread about this
>     >     >> issue
>     >     >>>>>     >       http://thread.gmane.org/gmane.comp.sysutils.backup <http://thread.gmane.org/gmane.comp.sysutils.backup> <http://thread.gmane.org/gmane.comp.sysutils.backup <http://thread.gmane.org/gmane.comp.sysutils.backup>>.
>     >     >>>> duplicity.general/4245 <http://thread.gmane.org/
>     >     >>>> gmane.comp.sysutils.backup.duplicity.general/4245> <
>     >     >>>> http://thread.gmane.org/gmane.comp.sysutils.backup <http://thread.gmane.org/gmane.comp.sysutils.backup> <http://thread.gmane.org/gmane.comp.sysutils.backup <http://thread.gmane.org/gmane.comp.sysutils.backup>>.
>     >     >> duplicity.general/4245
>     >     >>>> <http://thread.gmane.org/gmane.comp.sysutils.backup <http://thread.gmane.org/gmane.comp.sysutils.backup> <http://thread.gmane.org/gmane.comp.sysutils.backup <http://thread.gmane.org/gmane.comp.sysutils.backup>>.
>     >     >> duplicity.general/4245
>     >     >>>>>>
>     >     >>>>>     >
>     >     >>>>>     >     ..ede
>     >     >>>>>     >
>     >     >>>>>     >     _______________________________________________
>     >     >>>>>     >     Duplicity-talk mailing list
>     >     >>>>>     >     [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>
>     >     >>>> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>>
>     >     >>>>>     >     https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>> <
>     >     >>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>>> <
>     >     >>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>> <
>     >     >>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>>>>
>     >     >>>>>     >
>     >     >>>>>     >
>     >     >>>>>
>     >     >>>>>
>     >     >>>>>     _______________________________________________
>     >     >>>>>     Duplicity-talk mailing list
>     >     >>>>>     [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> <mailto:[hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>>>
>     >     >>>>>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>> <
>     >     >>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>>>
>     >     >>>>>
>     >     >>>>>
>     >     >>>>
>     >     >>>>
>     >     >>>
>     >     >>
>     >     >>
>     >     >
>     >
>     >
>
>



_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk