Problem with --exclude-if-present in version 0.7.10

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with --exclude-if-present in version 0.7.10

duplicity-talk mailing list
Hello,

I want to send a backup with --exclude-if-present=TAG and
--include-filelist.
The folder /src/folder is not accessible (mode 750, not owner, not
member of group). This folder is excluded by way of '- **/folder' in
the include filelist. Does the selection process resolve
exclude-if-present=TAG before it looks at include/exclude filelists
anyway ?

duplicity 0.7.10 (python 2.7.12) fails with

OSError: [Errno 13] Permission denied: '/src/folder/TAG'

duplicity 0.7.06 prints

Error accessing possibly locked file /src/folder

but it isn't a showstopper.

--
  Richard

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Problem with --exclude-if-present in version 0.7.10

duplicity-talk mailing list
Hello Richard,

On 2016-08-29 09:50, Richard via Duplicity-talk wrote:
> I want to send a backup with --exclude-if-present=TAG and
> --include-filelist.
> The folder /src/folder is not accessible (mode 750, not owner, not
> member of group). This folder is excluded by way of '- **/folder' in
> the include filelist.

> duplicity 0.7.10 (python 2.7.12) fails with
> OSError: [Errno 13] Permission denied: '/src/folder/TAG'
>
> duplicity 0.7.06 prints
> Error accessing possibly locked file /src/folder

That makes sense. 0.7.06 threw read errors if any folders within the
backup path were locked/unreadable, even if those were explicitly
excluded. In 0.7.10, it only gives the “Error accessing possibly locked
file” if the file/folder is not excluded. You shouldn’t receive this
error because you have excluded the folder. If you remove the -
**/folder in the include filelist (and anything else in your include
filelist that would exclude that folder), this error should come back
again (please let me know if that isn’t the case). The only difference
that should be fixed, therefore, is that the OSError should be
suppressed if the folder is separately excluded in a different selection
function. This difference should not have any impact on what is backed
up in this case (or any I can think of), but I will make a bug for this
and add it to my list.

> Does the selection process resolve
> exclude-if-present=TAG before it looks at include/exclude filelists
> anyway ?

As per the manual:
--exclude-if-present filename
Exclude directories if filename is present. Allows the user to specify
folders that they do not wish to backup by adding a specified file (e.g.
".nobackup") instead of maintaining a comprehensive exclude/include
list. This option needs to come before any other include or exclude
options.

So you are supposed to put in that option before any other options (and
it is therefore supposed to take priority over any other options). I’m
not quite sure why this is mandated, as from a very quick look at the
code it looks like it is just evaluated in order like the rest of the
options — any ideas why this is in the manual, Kenneth (or anyone else)?

Kind regards,

Aaron


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Problem with --exclude-if-present in version 0.7.10

duplicity-talk mailing list
On Tue, 30 Aug 2016 11:59:35 +0100
Aaron <[hidden email]> wrote:

Hello,

> Hello Richard,
>
> On 2016-08-29 09:50, Richard via Duplicity-talk wrote:
> > I want to send a backup with --exclude-if-present=TAG and
> > --include-filelist.
> > The folder /src/folder is not accessible (mode 750, not owner, not
> > member of group). This folder is excluded by way of '- **/folder' in
> > the include filelist.  
>
> > duplicity 0.7.10 (python 2.7.12) fails with
> > OSError: [Errno 13] Permission denied: '/src/folder/TAG'
> >
> > duplicity 0.7.06 prints
> > Error accessing possibly locked file /src/folder  
>
> That makes sense. 0.7.06 threw read errors if any folders within the
> backup path were locked/unreadable, even if those were explicitly
> excluded. In 0.7.10, it only gives the “Error accessing possibly
> locked file” if the file/folder is not excluded. You shouldn’t
> receive this error because you have excluded the folder. If you
> remove the - **/folder in the include filelist (and anything else in
> your include filelist that would exclude that folder), this error
> should come back again (please let me know if that isn’t the case).

With 0.7.10 I get the error in both cases: when '**/folder' is
excluded, and when it isn't.
Here is a stacktrace when '**/folder' is excluded in include-filelist:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1546, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1540, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1391, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1522, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 662, in incremental_backup
    bytes_written = dummy_backup(tarblock_iter)
  File "/usr/bin/duplicity", line 238, in dummy_backup
    while tarblock_iter.next():
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line
523, in next
    result = self.process(self.input_iter.next())
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line
195, in get_delta_iter
    for new_path, sig_path in collated:
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line
276, in collate2iters
    relem1 = riter1.next()
  File "/usr/lib/python2.7/dist-packages/duplicity/selection.py", line
179, in Iterate
    subpath, val = diryield_stack[-1].next()
  File "/usr/lib/python2.7/dist-packages/duplicity/selection.py", line
146, in diryield
    s = self.Select(new_path)
  File "/usr/lib/python2.7/dist-packages/duplicity/selection.py", line
208, in Select
    result = sf(path)
  File "/usr/lib/python2.7/dist-packages/duplicity/selection.py", line
450, in exclude_sel_func
    if path.isdir() and path.append(filename).exists():
  File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 535,
in append
    return self.__class__(self.base, self.index + (ext,))
  File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 515,
in __init__
    self.setdata()
  File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 520,
in setdata
    self.stat = os.lstat(self.name)
OSError: [Errno 13] Permission denied: '/src/folder/TAG'


> The only difference that should be fixed, therefore, is that the
> OSError should be suppressed if the folder is separately excluded in
> a different selection function. This difference should not have any
> impact on what is backed up in this case (or any I can think of), but
> I will make a bug for this and add it to my list.
>
> > Does the selection process resolve
> > exclude-if-present=TAG before it looks at include/exclude filelists
> > anyway ?  
>
> As per the manual:
> --exclude-if-present filename
> Exclude directories if filename is present. Allows the user to
> specify folders that they do not wish to backup by adding a specified
> file (e.g. ".nobackup") instead of maintaining a comprehensive
> exclude/include list. This option needs to come before any other
> include or exclude options.
>
> So you are supposed to put in that option before any other options
> (and it is therefore supposed to take priority over any other
> options). I’m not quite sure why this is mandated, as from a very
> quick look at the code it looks like it is just evaluated in order
> like the rest of the options — any ideas why this is in the manual,
> Kenneth (or anyone else)?
>
> Kind regards,
>
> Aaron
>


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Problem with --exclude-if-present in version 0.7.10

duplicity-talk mailing list
Thanks Richard,

On 2016-08-30 14:23, Richard McGraw via Duplicity-talk wrote:
>> [...] If you
>> remove the - **/folder in the include filelist (and anything else in
>> your include filelist that would exclude that folder), this error
>> should come back again (please let me know if that isn’t the case).
>
> With 0.7.10 I get the error in both cases: when '**/folder' is
> excluded, and when it isn't.

When you refer to "the error", do you mean the OSError or the "possibly
locked file" error?

> Here is a stacktrace when '**/folder' is excluded in include-filelist:

Can you please copy-paste the command you are running? I am interested
in which of the two options (--exclude-if-present or --include-filelist)
are first in the command you are running.

Kind regards,

Aaron

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Problem with --exclude-if-present in version 0.7.10

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
Hello Richard,

On 2016-08-29 09:50, Richard via Duplicity-talk wrote:

>> I want to send a backup with --exclude-if-present=TAG and
>> --include-filelist.
>> The folder /src/folder is not accessible (mode 750, not owner, not
>> member of group). This folder is excluded by way of '- **/folder' in
>> the include filelist.
>
>> duplicity 0.7.10 (python 2.7.12) fails with
>> OSError: [Errno 13] Permission denied: '/src/folder/TAG'
>>
>> duplicity 0.7.06 prints
>> Error accessing possibly locked file /src/folder
>

I have created a bug for this here:
https://bugs.launchpad.net/duplicity/+bug/1620085
so please subscribe to that if you are interested in further progress.

I have also reproduced the issue in a test case, created a fix for it
and submitted the fix for merging, so you should see this working in a
future release shortly. It would be great if you could please keep an
eye out for that release and let us know that the fix worked for you.

Many thanks for bringing this issue to our attention.

Kind regards,

Aaron

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Problem with --exclude-if-present in version 0.7.10

duplicity-talk mailing list
On Sun, 4 Sep 2016 23:44:26 +0100
Aaron <[hidden email]> wrote:

> I have created a bug for this here:
> https://bugs.launchpad.net/duplicity/+bug/1620085
> so please subscribe to that if you are interested in further progress.
>
> I have also reproduced the issue in a test case, created a fix for it
> and submitted the fix for merging, so you should see this working in
> a future release shortly. It would be great if you could please keep
> an eye out for that release and let us know that the fix worked for
> you.
>
> Many thanks for bringing this issue to our attention.
>

Thanks for taking care of it !

> Kind regards,


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Problem with --exclude-if-present in version 0.7.10

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
On Sun, 4 Sep 2016 23:44:26 +0100
Aaron <[hidden email]> wrote:

> Hello Richard,
>
> On 2016-08-29 09:50, Richard via Duplicity-talk wrote:
> >> I want to send a backup with --exclude-if-present=TAG and
> >> --include-filelist.
> >> The folder /src/folder is not accessible (mode 750, not owner, not
> >> member of group). This folder is excluded by way of '- **/folder'
> >> in the include filelist.  
> >  
> >> duplicity 0.7.10 (python 2.7.12) fails with
> >> OSError: [Errno 13] Permission denied: '/src/folder/TAG'
> >>
> >> duplicity 0.7.06 prints
> >> Error accessing possibly locked file /src/folder  
> >  
>
> I have created a bug for this here:
> https://bugs.launchpad.net/duplicity/+bug/1620085
> so please subscribe to that if you are interested in further progress.
>
> I have also reproduced the issue in a test case, created a fix for it
> and submitted the fix for merging, so you should see this working in
> a future release shortly. It would be great if you could please keep
> an eye out for that release and let us know that the fix worked for
> you.
>
> Many thanks for bringing this issue to our attention.
>
> Kind regards,
>
> Aaron

Since my update to duplicity 0.7.11, I haven't reproduced the problem.

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk