filelist glob usage frustration

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
RL
Reply | Threaded
Open this post in threaded view
|

filelist glob usage frustration

RL
Cut to the bottom for my problem. The rest is background.

I have searched and read everything I could find, especially with
explicit examples, on rdiff control for the last 2 days. All examples
seemed very simple, which is fine to start with. But they have not
helped me get started with the most basic tests.

I have little/no familiarity with "regular expressions"; that language
looks like very evil gobbledy-gook to me. However, I have been using simple wild card filters with *, ? format forever.

rdiff seems to use some kind of hybrid, in fact _two_ hybrids because
the "filelist" files format appears to operate differently than the
command line format. I use filelists because it is considerably easier to view and understand multiple file selection specs in that format.

Clearly I have a very basic misunderstanding, at the lowest
level. Here are console printouts which I hope show what I have done and
make it easy for somebody experienced to immediately point out problems.


== test directory:
rufus@Air-PC:~/WK/testin> dir
total 600
-rw-r--r-- 1 rufus users     67 Feb 16 17:24 aud-link.txt
-rw-r--r-- 1 rufus users   3759 May  3 23:39 bad-world_rant.txt
-rw-r--r-- 1 rufus users   3981 Feb  9 13:00 bernie-rant.txt
-rw-r--r-- 1 rufus users 584635 Feb  3 16:13
bldg_bath_fans_FineHomeBuilding.pdf
drwxr-xr-x 3 rufus users   4096 Apr 17 17:01 dir1
drwxr-xr-x 2 rufus users   4096 May 11 08:03 dir2
drwxr-xr-x 6 rufus users   4096 May 12 07:55 dir3
drwxr-xr-x 2 rufus users   4096 May 12 08:16 dir4
rufus@Air-PC:~/WK/testin>


== test directory, 2nd level - #a
rufus@Air-PC:~/WK/testin/dir3> dir
total 40
drwxr-xr-x 4 rufus users 4096 May 11 08:07 CL_pics
-rw-r--r-- 1 rufus users    2 May 11 09:37 f-test
drwxr-xr-x 3 rufus users 4096 May 11 09:16 maps
-rw-r--r-- 1 rufus users    2 May 12 07:48 m_test-1.txt
-rw-r--r-- 1 rufus users    2 May 12 07:48 m_test-2.txt
-rw-r--r-- 1 rufus users    6 May  7 08:43 M_test2.txt
-rw-r--r-- 1 rufus users    2 May 12 07:48 m_test.next.txt
-rw-r--r-- 1 rufus users    6 May  7 08:44  m_test.txt
drwxr-xr-x 2 rufus users 4096 May 12 07:55 Next-Dir
drwxr-xr-x 4 rufus users 4096 May 11 09:16 pics
rufus@Air-PC:~/WK/testin/dir3>

== test directory, 2nd level - #b
rufus@Air-PC:~/WK/testin/dir4> dir
total 24
-rw-r--r-- 1 rufus users 2 May 12 07:48 m_test-2.txt
-rw-r--r-- 1 rufus users 2 May 12 07:48 m_test-41.txt
-rw-r--r-- 1 rufus users 2 May 12 07:48 m_test-42.txt
-rw-r--r-- 1 rufus users 6 May  7 08:43 M_test42.txt
-rw-r--r-- 1 rufus users 2 May 12 07:48 m_test.4next.txt
-rw-r--r-- 1 rufus users 6 May  7 08:44  m_test.txt
rufus@Air-PC:~/WK/testin/dir4>

There are other dirs, but they don't factor until I can at least get started.

== Here is the script I use to run rdiff:
rufus@Air-PC:~/bin> cat rdiff_test2.sh
#!/bin/bash

rdiff-backup -v5 --include-globbing-filelist rdiff_filelist-test_2.txt \
    ./testin \
    testout2

== Here is the filelist:
rufus@Air-PC:~/WK> cat rdiff_filelist-test_2.txt
+ /home/rufus/WK/testin/**
- **
rufus@Air-PC:~/WK>

==Here is my test run:
rufus@Air-PC:~/WK> rdiff_test2.sh
Using rdiff-backup version 2.0.0
        with cpython /usr/bin/python3 version 3.6.10
        on Linux-4.12.14-lp151.28.48-default-x86_64-with-glibc2.3.4, fs encoding utf-8
Found interrupted initial backup. Removing...
Unable to import module posix1e from pylibacl package.
POSIX ACLs not supported on filesystem at testin
Unable to import win32security module. Windows ACLs
not supported by filesystem at testin
-----------------------------------------------------------------
Detected abilities for source (read only) file system:
  Access control lists                         Off
  Extended attributes                          On
  Windows access control lists                 Off
  Case sensitivity                             On
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Unable to import module posix1e from pylibacl package.
POSIX ACLs not supported on filesystem at testout2/rdiff-backup-data/rdiff-backup.tmp.0
Unable to import win32security module. Windows ACLs
not supported by filesystem at testout2/rdiff-backup-data/rdiff-backup.tmp.0
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
  Ownership changing                           Off
  Hard linking                                 On
  fsync() directories                          On
  Directory inc permissions                    On
  High-bit permissions                         On
  Symlink permissions                          Off
  Extended filenames                           On
  Windows reserved filenames                   Off
  Access control lists                         Off
  Extended attributes                          On
  Windows access control lists                 Off
  Case sensitivity                             On
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Backup: escape_dos_devices = 0
Backup: escape_trailing_spaces = 0
Reading globbing filelist rdiff_filelist-test_2.txt
Fatal Error: Fatal Error: The file specification
    'b'/home/rufus/WK/testin/**''
cannot match any files in the base directory
    'b'testin''
Useful file specifications begin with the base directory or some
pattern (such as '**') which matches the base directory.
rufus@Air-PC:~/WK>

==Here is the test using "*" instead of "**" (and with -v3)
rufus@Air-PC:~/WK> rdiff_test2.sh
Fatal Error: Fatal Error: The file specification
    'b'/home/rufus/WK/testin/*''
cannot match any files in the base directory
    'b'testin''
Useful file specifications begin with the base directory or some
pattern (such as '**') which matches the base directory.
rufus@Air-PC:~/WK>

The above two test results make absolutely no sense to me. And yes, I am including and then excluding the whole tree. This is a test to figure out just how the heck rdiff works, not a production run. Based on the docs, prior lines take precedence and I would expect one or both or those tests to copy the whole tree. It's likely something extremely simple, but I'm going in circles and don't see it.

Thanks for thoughts.


Regards,
Rufus

RL
Reply | Threaded
Open this post in threaded view
|

Re: filelist glob usage frustration

RL
Eric

Appreciate your prompt help.

> AFAICJ your main error is to mix absolute paths in the glob list with relative paths on the command line.

I had thought that the relative addresses resolved to absolute addresses before execution. IOW, that the relative paths in the command line appear to rdiff as absolute paths or lists of absolute values when it is executing because the shell resolves them before passing them as arguments to the program. Not?

Ah. It looks like there is a sinkhole in my understanding of shell behavior. The trivial tests now run as expected. Onward. Thank you very much for your help.

(Bookkeeping)

I am replying to the list in response to your email but not copying you. Is that the usual best way? Or should I copy you as well? (I have not been active in lists for a very long while.)

I normally top-post and leave the minimum quote needed for clarity. From dial-up days. Not sure if that is best on a list now. Better to always quote fully, or maybe interleave relevant snippets and crop less germane bits? Or some other form?

Regards,
Rufus

Reply | Threaded
Open this post in threaded view
|

Re: filelist glob usage frustration

Eric Lavarde
Hi,

On 12/05/2020 23:30, RL wrote:
> Ah. It looks like there is a sinkhole in my understanding of shell behavior. The trivial tests now run as expected. Onward. Thank you very much for your help.

Yep, shells generally only resolve wildcards and variables (and a few
other similar things), and not relative paths. You can see it with e.g.
`echo ./relativepath * ${HOME} $(date)`, you'll see what get resolved by
the shell and what not.

>
> (Bookkeeping)
>
> I am replying to the list in response to your email but not copying you. Is that the usual best way? Or should I copy you as well? (I have not been active in lists for a very long while.)

Do not copy me, please. I read the list and else it confuses my filter.

> I normally top-post and leave the minimum quote needed for clarity. From dial-up days. Not sure if that is best on a list now. Better to always quote fully, or maybe interleave relevant snippets and crop less germane bits? Or some other form?

Whatever is reasonable for the specific case :-)

I personally prefer to crop useless parts, the thread has the history,
not each e-mail, and top-post or interleave, depending on the complexity
of my answer. If something is worth documenting, it anyway belongs in
the docs...

KR, Eric