Failure running tests.

Next Topic
 
classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Failure running tests.

duplicity-talk mailing list
Duplicity wizards,

I think I have successfully build duplicity version 0.8.04 on Mac OS X (python 3.7 and Mac OS X 10.11).  This was upgrading from my previous installation using python 2.7.

However the testing phase seems to fail  a  lot of tests.  The first group I understand since there is no tahoe backend installed.  I can live with that and will get around to adding backend support later.

But a lot of tests show that duplicity does not find my lftp installation.  The binary is at a “non-standard” install, or at least non-standard for linux folks.  It is at /sw/bin/lftp.  My normal path should include that and be inherited by the testing routines.  However, it may be that the path to the lftp binary is hardwired to somewhere else?

I have also installed the lftp dynamic libraries:
/sw//lib/lftp
/sw//lib/lftp/4.8.3/liblftp-network.dylib
/sw//lib/lftp/4.8.3/liblftp-pty.dylib
/sw//lib/liblftp-jobs.0.dylib
/sw//lib/liblftp-tasks.0.dylib

The second issue is that I get a lot of errors complaining that reprlib is also missing.
b'    __import__(newmodname)'
b'ImportError: No module named reprlib’

The module exists in my build for python 3.7 and is at the install location I expect
/sw/lib/python3.7/reprlib.py

Again is that module path hardwired somewhere that I should edit?

Any guidance would be helpful.

-Scott


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
I am going to reply to my own message.. :-)  Most of the errors were due to my not reading my own build notes. :-(  I have run the tests again and seem to have several errors due to unicode issues which was the big advantage of moving to duplicity 8.  I am down to 17 errors and 3 warnings:
======== 17 failed, 409 passed, 2 skipped, 3 warnings in 894.00 seconds ========


These two errors have me stumped not being a python or unicode expert.  os.py looks reasonable to me and should return a string or path.
b'  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/duplicity/path.py", line 476, in <lambda>'
b'    util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))'
b"PermissionError: [Errno 1] Operation not permitted: b'testfiles/cache/3f6f90580e457c90e7202bc33cd9e363/duplicity-full-signatures.19700102T034640Z.sigtar.gz'"
b'During handling of the above exception, another exception occurred:'
b'Traceback (most recent call last):'
b'  File "../bin/duplicity", line 157, in <module>'
b'    if u"Forced assertion for testing" in util.uexc(e):'
b'  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/duplicity/util.py", line 120, in uexc'
b'    return fsdecode(m)'
b'  File "/sw/lib/python3.7/os.py", line 821, in fsdecode'
b'    filename = fspath(filename)  # Does type-checking of `filename`.'
b'TypeError: expected str, bytes or os.PathLike object, not int'
...return_val: 1

So there is a permission error which may be normal since the testing is done in a non-admin account and cannot chown files.  But then there is a secondary error where fsdecode fails.  It looks like os.py does return the path and not an int but someone thinks it is getting a wrong type.

The other errors are also unicode related.
________________ TestUnicode.test_unicode_paths_square_brackets ________________

self = <testing.functional.test_selection.TestUnicode testMethod=test_unicode_paths_square_brackets>

    def test_unicode_paths_square_brackets(self):
        u""" Test --include and --exclude work with unicode paths with character options in []s and [!]s"""
        p = u"testfiles/select-unicode/"
        self.backup(u"full", u"testfiles/select-unicode",
                    options=[u"--exclude", p + u"прыклад/пример/例/Παράδειγμα/उदाहरण.txt",
                             u"--exclude", p + u"пры[к,и,р]лад/пример/例/Παράδειγμα/דוגמא.txt",
                             u"--exclude", p + u"прыклад/пр[!a,b,c]мер/例/მაგალითი/",
                             u"--include", p + u"прыклад/при[g,м,д]ер/例/",
                             u"--exclude", p + u"прыклад/пример/",
                             u"--include", p + u"прыклад/",
                             u"--include", p + u"օրինակ.txt",
                             u"--exclude", p + u"**"])
        self.restore()
        restore_dir = u"testfiles/restore_out"
        restored = self.directory_tree_to_list_of_lists(restore_dir)
        self.assertEqual(restored, [[u"прыклад", u"օրինակ.txt"],
>                                   [u"пример", u"উদাহরণ"], [u"例"], [u"Παράδειγμα"], [u"ઉદાહરણ.log"]])
E       AssertionError: Lists differ: [['пр[48 chars]'], ['Παράδειγμα'], ['דוגמא.txt', 'उदाहरण.txt', 'ઉદાહરણ.log']] != [['пр[48 chars]'], ['Παράδειγμα'], ['ઉદાહરણ.log']]
E       
E       First differing element 3:
E       ['Παράδειγμα']
E       ['Παράδειγμα']
E       
E         [['прыклад', 'օրինակ.txt'],
E          ['пример', 'উদাহরণ'],
E          ['例'],
E       -  ['Παράδειγμα'],
E       ?       ^^
E       
E       +  ['Παράδειγμα'],
E       ?       ^
E       
E       -  ['דוגמא.txt', 'उदाहरण.txt', 'ઉદાહરણ.log']]
E       +  ['ઉદાહરણ.log']]
The strings that are listed as different (element 3) but I am not quite sure what the issue is.  Unicode should be completely cross platform…  That is the point! :-)

-Scott


On Sep 1, 2019, at 5:49 PM, Scott Hannahs via Duplicity-talk <[hidden email]> wrote:

Duplicity wizards,

I think I have successfully build duplicity version 0.8.04 on Mac OS X (python 3.7 and Mac OS X 10.11).  This was upgrading from my previous installation using python 2.7.

However the testing phase seems to fail  a  lot of tests.  The first group I understand since there is no tahoe backend installed.  I can live with that and will get around to adding backend support later.



_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Hi Scott,

How are you running the tests?  We recommend just using "tox" as the test driver.

Are you using python37 from homebrew or macports.  I've tested using the homebrew install under 10.14.

You should not be getting any errors on py27 or py37.  There are 4 known errors on py36.

A bit more detail, e.g. full test logs, would help when reporting.

...Thanks,
...Ken


On Mon, Sep 2, 2019 at 3:58 PM Scott Hannahs via Duplicity-talk <[hidden email]> wrote:
I am going to reply to my own message.. :-)  Most of the errors were due to my not reading my own build notes. :-(  I have run the tests again and seem to have several errors due to unicode issues which was the big advantage of moving to duplicity 8.  I am down to 17 errors and 3 warnings:
======== 17 failed, 409 passed, 2 skipped, 3 warnings in 894.00 seconds ========


These two errors have me stumped not being a python or unicode expert.  os.py looks reasonable to me and should return a string or path.
b'  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/duplicity/path.py", line 476, in <lambda>'
b'    util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))'
b"PermissionError: [Errno 1] Operation not permitted: b'testfiles/cache/3f6f90580e457c90e7202bc33cd9e363/duplicity-full-signatures.19700102T034640Z.sigtar.gz'"
b'During handling of the above exception, another exception occurred:'
b'Traceback (most recent call last):'
b'  File "../bin/duplicity", line 157, in <module>'
b'    if u"Forced assertion for testing" in util.uexc(e):'
b'  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/duplicity/util.py", line 120, in uexc'
b'    return fsdecode(m)'
b'  File "/sw/lib/python3.7/os.py", line 821, in fsdecode'
b'    filename = fspath(filename)  # Does type-checking of `filename`.'
b'TypeError: expected str, bytes or os.PathLike object, not int'
...return_val: 1

So there is a permission error which may be normal since the testing is done in a non-admin account and cannot chown files.  But then there is a secondary error where fsdecode fails.  It looks like os.py does return the path and not an int but someone thinks it is getting a wrong type.

The other errors are also unicode related.
________________ TestUnicode.test_unicode_paths_square_brackets ________________

self = <testing.functional.test_selection.TestUnicode testMethod=test_unicode_paths_square_brackets>

    def test_unicode_paths_square_brackets(self):
        u""" Test --include and --exclude work with unicode paths with character options in []s and [!]s"""
        p = u"testfiles/select-unicode/"
        self.backup(u"full", u"testfiles/select-unicode",
                    options=[u"--exclude", p + u"прыклад/пример/例/Παράδειγμα/उदाहरण.txt",
                             u"--exclude", p + u"пры[к,и,р]лад/пример/例/Παράδειγμα/דוגמא.txt",
                             u"--exclude", p + u"прыклад/пр[!a,b,c]мер/例/მაგალითი/",
                             u"--include", p + u"прыклад/при[g,м,д]ер/例/",
                             u"--exclude", p + u"прыклад/пример/",
                             u"--include", p + u"прыклад/",
                             u"--include", p + u"օրինակ.txt",
                             u"--exclude", p + u"**"])
        self.restore()
        restore_dir = u"testfiles/restore_out"
        restored = self.directory_tree_to_list_of_lists(restore_dir)
        self.assertEqual(restored, [[u"прыклад", u"օրինակ.txt"],
>                                   [u"пример", u"উদাহরণ"], [u"例"], [u"Παράδειγμα"], [u"ઉદાહરણ.log"]])
E       AssertionError: Lists differ: [['пр[48 chars]'], ['Παράδειγμα'], ['דוגמא.txt', 'उदाहरण.txt', 'ઉદાહરણ.log']] != [['пр[48 chars]'], ['Παράδειγμα'], ['ઉદાહરણ.log']]
E       
E       First differing element 3:
E       ['Παράδειγμα']
E       ['Παράδειγμα']
E       
E         [['прыклад', 'օրինակ.txt'],
E          ['пример', 'উদাহরণ'],
E          ['例'],
E       -  ['Παράδειγμα'],
E       ?       ^^
E       
E       +  ['Παράδειγμα'],
E       ?       ^
E       
E       -  ['דוגמא.txt', 'उदाहरण.txt', 'ઉદાહરણ.log']]
E       +  ['ઉદાહરણ.log']]
The strings that are listed as different (element 3) but I am not quite sure what the issue is.  Unicode should be completely cross platform…  That is the point! :-)

-Scott


On Sep 1, 2019, at 5:49 PM, Scott Hannahs via Duplicity-talk <[hidden email]> wrote:

Duplicity wizards,

I think I have successfully build duplicity version 0.8.04 on Mac OS X (python 3.7 and Mac OS X 10.11).  This was upgrading from my previous installation using python 2.7.

However the testing phase seems to fail  a  lot of tests.  The first group I understand since there is no tahoe backend installed.  I can live with that and will get around to adding backend support later.


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Ken,

I am running them under the test phase of “fink”.  I use fink rather than homebrew.

so the test script is the following.  I am not sure if the chmod is still necessary but it used to be.  The build is done while running as an unprivileged user called “fink-bld”. This script should call tox directly and not be an issue.  I am fairly happy that it passed 409 different tests!

TestScript: <<
        #!/bin/sh -ev
        ulimit -n 5120
        chmod 0700 `pwd`/testing/gnupg
        %p/bin/python3.7 setup.py test
<<

I am using the install of python from the 3.7 sources.  I patched duplicity so that every line “#!/bin/env python” was replaced with ‘#!/bin/env python3.7” as I have done in the past for python 2.7.

The python version is 3.7.4.
% python3.7 --version
Python 3.7.4

I heard about the errors in 3.6 and so chose to build under 3.7.  I am running a fairly old OS but python 2.7 is also getting a bit old.

The script above should call tox directly and that is the 3.7 version as far as I can tell.  I can run tox directly if that helps.

% tox --version
3.0.0 imported from /sw/lib/python3.7/site-packages/tox/__init__.py

I wasn’t sure I could attach a file to this list, but I will try with the test log.  There is some fink specific listing as it builds duplicity and then goes straight into the tests.  This file has both he standard and error output redirected to it.  I had some tests fail under python2.7 previously and we never got to the bottom of that.

The OS is Mac OS 10.11.6 and maybe had a flawed installation of unicode?  Or an old unicode missing some newer definitions?  The characters seem to show up in the error output correctly but maybe that is an issue.  However the unicode comparison should be comparing the 4 hex digits of a unicode character and that should work.

Thanks for the help.
-Scott



On Sep 2, 2019, at 5:16 PM, Kenneth Loafman <[hidden email]> wrote:

Hi Scott,

How are you running the tests?  We recommend just using "tox" as the test driver.

Are you using python37 from homebrew or macports.  I've tested using the homebrew install under 10.14.

You should not be getting any errors on py27 or py37.  There are 4 known errors on py36.

A bit more detail, e.g. full test logs, would help when reporting.

...Thanks,
...Ken


On Mon, Sep 2, 2019 at 3:58 PM Scott Hannahs via Duplicity-talk <[hidden email]> wrote:
I am going to reply to my own message.. :-)  Most of the errors were due to my not reading my own build notes. :-(  I have run the tests again and seem to have several errors due to unicode issues which was the big advantage of moving to duplicity 8.  I am down to 17 errors and 3 warnings:
======== 17 failed, 409 passed, 2 skipped, 3 warnings in 894.00 seconds ========


These two errors have me stumped not being a python or unicode expert.  os.py looks reasonable to me and should return a string or path.
b'  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/duplicity/path.py", line 476, in <lambda>'
b'    util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))'
b"PermissionError: [Errno 1] Operation not permitted: b'testfiles/cache/3f6f90580e457c90e7202bc33cd9e363/duplicity-full-signatures.19700102T034640Z.sigtar.gz'"
b'During handling of the above exception, another exception occurred:'
b'Traceback (most recent call last):'
b'  File "../bin/duplicity", line 157, in <module>'
b'    if u"Forced assertion for testing" in util.uexc(e):'
b'  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/duplicity/util.py", line 120, in uexc'
b'    return fsdecode(m)'
b'  File "/sw/lib/python3.7/os.py", line 821, in fsdecode'
b'    filename = fspath(filename)  # Does type-checking of `filename`.'
b'TypeError: expected str, bytes or os.PathLike object, not int'
...return_val: 1

So there is a permission error which may be normal since the testing is done in a non-admin account and cannot chown files.  But then there is a secondary error where fsdecode fails.  It looks like os.py does return the path and not an int but someone thinks it is getting a wrong type.

The other errors are also unicode related.
________________ TestUnicode.test_unicode_paths_square_brackets ________________

self = <testing.functional.test_selection.TestUnicode testMethod=test_unicode_paths_square_brackets>

    def test_unicode_paths_square_brackets(self):
        u""" Test --include and --exclude work with unicode paths with character options in []s and [!]s"""
        p = u"testfiles/select-unicode/"
        self.backup(u"full", u"testfiles/select-unicode",
                    options=[u"--exclude", p + u"прыклад/пример/例/Παράδειγμα/उदाहरण.txt",
                             u"--exclude", p + u"пры[к,и,р]лад/пример/例/Παράδειγμα/דוגמא.txt",
                             u"--exclude", p + u"прыклад/пр[!a,b,c]мер/例/მაგალითი/",
                             u"--include", p + u"прыклад/при[g,м,д]ер/例/",
                             u"--exclude", p + u"прыклад/пример/",
                             u"--include", p + u"прыклад/",
                             u"--include", p + u"օրինակ.txt",
                             u"--exclude", p + u"**"])
        self.restore()
        restore_dir = u"testfiles/restore_out"
        restored = self.directory_tree_to_list_of_lists(restore_dir)
        self.assertEqual(restored, [[u"прыклад", u"օրինակ.txt"],
>                                   [u"пример", u"উদাহরণ"], [u"例"], [u"Παράδειγμα"], [u"ઉદાહરણ.log"]])
E       AssertionError: Lists differ: [['пр[48 chars]'], ['Παράδειγμα'], ['דוגמא.txt', 'उदाहरण.txt', 'ઉદાહરણ.log']] != [['пр[48 chars]'], ['Παράδειγμα'], ['ઉદાહરણ.log']]
E       
E       First differing element 3:
E       ['Παράδειγμα']
E       ['Παράδειγμα']
E       
E         [['прыклад', 'օրինակ.txt'],
E          ['пример', 'উদাহরণ'],
E          ['例'],
E       -  ['Παράδειγμα'],
E       ?       ^^
E       
E       +  ['Παράδειγμα'],
E       ?       ^
E       
E       -  ['דוגמא.txt', 'उदाहरण.txt', 'ઉદાહરણ.log']]
E       +  ['ઉદાહરણ.log']]
The strings that are listed as different (element 3) but I am not quite sure what the issue is.  Unicode should be completely cross platform…  That is the point! :-)

-Scott


On Sep 1, 2019, at 5:49 PM, Scott Hannahs via Duplicity-talk <[hidden email]> wrote:

Duplicity wizards,

I think I have successfully build duplicity version 0.8.04 on Mac OS X (python 3.7 and Mac OS X 10.11).  This was upgrading from my previous installation using python 2.7.

However the testing phase seems to fail  a  lot of tests.  The first group I understand since there is no tahoe backend installed.  I can live with that and will get around to adding backend support later.


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

fink-duplicity.txt (216K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Scott,

When I run "tox -e py37" on my Mac I have no errors.

Your errors in test_selection are the same as on py36, very weird.

The permission errors causing test_replicate and test_restart are also odd.  Do you have the system locked down more than normal?  I test using just my own creds, not root..

The unicode errors when reporting the permission errors are something that needs to be fixed, but are a distraction at this point.

Try running under tox instead of fink just to eliminate one variable.

I've pasted my run.  Perhaps there is a version issue between our two systems?

...Thanks,
...Ken

ken@dione:~/workspace/duplicity-src8$ tox -e py37
GLOB sdist-make: /Users/ken/workspace/duplicity-src8/setup.py
py37 create: /Users/ken/workspace/duplicity-src8/.tox/py37
py37 installdeps: -rrequirements.txt
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 inst: /Users/ken/workspace/duplicity-src8/.tox/.tmp/package/1/duplicity-0.0dev.tar.gz
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 installed: alabaster==0.7.12,astroid==2.2.5,atomicwrites==1.3.0,attrs==19.1.0,Babel==2.7.0,certifi==2019.6.16,chardet==3.0.4,coverage==4.5.4,docutils==0.15.2,duplicity==0.0.dev0,fasteners==0.15,fastimport==0.9.2,filelock==3.0.12,future==0.17.1,idna==2.8,imagesize==1.1.0,importlib-metadata==0.20,isort==4.3.21,Jinja2==2.10.1,lazy-object-proxy==1.4.2,MarkupSafe==1.1.1,mccabe==0.6.1,mock==3.0.5,monotonic==1.5,more-itertools==7.2.0,packaging==19.1,pexpect==4.7.0,pluggy==0.12.0,ptyprocess==0.6.0,py==1.8.0,pycodestyle==2.5.0,Pygments==2.4.2,pylint==2.3.1,pyparsing==2.4.2,pytest==5.1.2,pytest-cov==2.7.1,pytest-runner==5.1,pytz==2019.2,requests==2.22.0,six==1.12.0,snowballstemmer==1.9.0,Sphinx==2.2.0,sphinxcontrib-applehelp==1.0.1,sphinxcontrib-devhelp==1.0.1,sphinxcontrib-htmlhelp==1.0.2,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.2,sphinxcontrib-serializinghtml==1.1.3,toml==0.10.0,tox==3.14.0,typed-ast==1.4.0,urllib3==1.25.3,virtualenv==16.7.5,wcwidth==0.1.7,wrapt==1.11.2,zipp==0.6.0
py37 run-test-pre: PYTHONHASHSEED='3996759775'
py37 run-test: commands[0] | python compilec.py
running build
running build_ext
building '_librsync' extension
creating build
creating build/temp.macosx-10.14-x86_64-3.7
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c _librsyncmodule.c -o build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o
creating build/lib.macosx-10.14-x86_64-3.7
clang -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o -L/usr/local/lib -lrsync -o build/lib.macosx-10.14-x86_64-3.7/_librsync.cpython-37m-darwin.so
py37 run-test: commands[1] | pytest --cov=duplicity --cov-append --cov-report=
================================================= test session starts ==================================================
platform darwin -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.12.0
cachedir: .tox/py37/.pytest_cache
rootdir: /Users/ken/workspace/duplicity-src8, inifile: tox.ini, testpaths: testing/unit, testing/functional
plugins: cov-2.7.1
collected 428 items                                                                                                    

testing/unit/test_backend.py ............                                                                        [  2%]
testing/unit/test_backend_instance.py ............................................................               [ 16%]
testing/unit/test_collections.py ........                                                                        [ 18%]
testing/unit/test_diffdir.py .......                                                                             [ 20%]
testing/unit/test_dup_temp.py ..                                                                                 [ 20%]
testing/unit/test_dup_time.py ......................                                                             [ 25%]
testing/unit/test_file_naming.py .............                                                                   [ 28%]
testing/unit/test_globmatch.py ...........................                                                       [ 35%]
testing/unit/test_gpg.py ..........                                                                              [ 37%]
testing/unit/test_gpginterface.py ......                                                                         [ 39%]
testing/unit/test_lazy.py ............................                                                           [ 45%]
testing/unit/test_manifest.py .....                                                                              [ 46%]
testing/unit/test_patchdir.py ........                                                                           [ 48%]
testing/unit/test_path.py .....                                                                                  [ 49%]
testing/unit/test_selection.py ............................................................................      [ 67%]
testing/unit/test_statistics.py .......                                                                          [ 69%]
testing/unit/test_tarfile.py .                                                                                   [ 69%]
testing/unit/test_tempdir.py ..                                                                                  [ 69%]
testing/unit/test_util.py .                                                                                      [ 70%]
testing/functional/test_badupload.py .                                                                           [ 70%]
testing/functional/test_cleanup.py ...                                                                           [ 71%]
testing/functional/test_final.py ...........................                                                     [ 77%]
testing/functional/test_log.py .                                                                                 [ 77%]
testing/functional/test_rdiffdir.py .                                                                            [ 77%]
testing/functional/test_replicate.py ...                                                                         [ 78%]
testing/functional/test_restart.py ................................                                              [ 85%]
testing/functional/test_selection.py .......................................ss...........                        [ 98%]
testing/functional/test_verify.py ........                                                                       [100%]

=================================================== warnings summary ===================================================
.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Iterable

.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

testing/unit/test_globmatch.py:100
  /Users/ken/workspace/duplicity-src8/testing/unit/test_globmatch.py:100: DeprecationWarning: invalid escape sequence \/
    u"\/usr\/[^/]*\/bin\/")

-- Docs: https://docs.pytest.org/en/latest/warnings.html


================================ 426 passed, 2 skipped, 3 warnings in 560.97s (0:09:20) ================================
_______________________________________________________ summary ________________________________________________________
  py37: commands succeeded
  congratulations :)

On Mon, Sep 2, 2019 at 4:54 PM Scott Hannahs <[hidden email]> wrote:
Ken,

I am running them under the test phase of “fink”.  I use fink rather than homebrew.

so the test script is the following.  I am not sure if the chmod is still necessary but it used to be.  The build is done while running as an unprivileged user called “fink-bld”. This script should call tox directly and not be an issue.  I am fairly happy that it passed 409 different tests!

TestScript: <<
        #!/bin/sh -ev
        ulimit -n 5120
        chmod 0700 `pwd`/testing/gnupg
        %p/bin/python3.7 setup.py test
<<

I am using the install of python from the 3.7 sources.  I patched duplicity so that every line “#!/bin/env python” was replaced with ‘#!/bin/env python3.7” as I have done in the past for python 2.7.

The python version is 3.7.4.
% python3.7 --version
Python 3.7.4

I heard about the errors in 3.6 and so chose to build under 3.7.  I am running a fairly old OS but python 2.7 is also getting a bit old.

The script above should call tox directly and that is the 3.7 version as far as I can tell.  I can run tox directly if that helps.

% tox --version
3.0.0 imported from /sw/lib/python3.7/site-packages/tox/__init__.py

I wasn’t sure I could attach a file to this list, but I will try with the test log.  There is some fink specific listing as it builds duplicity and then goes straight into the tests.  This file has both he standard and error output redirected to it.  I had some tests fail under python2.7 previously and we never got to the bottom of that.

The OS is Mac OS 10.11.6 and maybe had a flawed installation of unicode?  Or an old unicode missing some newer definitions?  The characters seem to show up in the error output correctly but maybe that is an issue.  However the unicode comparison should be comparing the 4 hex digits of a unicode character and that should work.

Thanks for the help.
-Scott


On Sep 2, 2019, at 5:16 PM, Kenneth Loafman <[hidden email]> wrote:

Hi Scott,

How are you running the tests?  We recommend just using "tox" as the test driver.

Are you using python37 from homebrew or macports.  I've tested using the homebrew install under 10.14.

You should not be getting any errors on py27 or py37.  There are 4 known errors on py36.

A bit more detail, e.g. full test logs, would help when reporting.

...Thanks,
...Ken


On Mon, Sep 2, 2019 at 3:58 PM Scott Hannahs via Duplicity-talk <[hidden email]> wrote:
I am going to reply to my own message.. :-)  Most of the errors were due to my not reading my own build notes. :-(  I have run the tests again and seem to have several errors due to unicode issues which was the big advantage of moving to duplicity 8.  I am down to 17 errors and 3 warnings:
======== 17 failed, 409 passed, 2 skipped, 3 warnings in 894.00 seconds ========


These two errors have me stumped not being a python or unicode expert.  os.py looks reasonable to me and should return a string or path.
b'  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/duplicity/path.py", line 476, in <lambda>'
b'    util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))'
b"PermissionError: [Errno 1] Operation not permitted: b'testfiles/cache/3f6f90580e457c90e7202bc33cd9e363/duplicity-full-signatures.19700102T034640Z.sigtar.gz'"
b'During handling of the above exception, another exception occurred:'
b'Traceback (most recent call last):'
b'  File "../bin/duplicity", line 157, in <module>'
b'    if u"Forced assertion for testing" in util.uexc(e):'
b'  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/duplicity/util.py", line 120, in uexc'
b'    return fsdecode(m)'
b'  File "/sw/lib/python3.7/os.py", line 821, in fsdecode'
b'    filename = fspath(filename)  # Does type-checking of `filename`.'
b'TypeError: expected str, bytes or os.PathLike object, not int'
...return_val: 1

So there is a permission error which may be normal since the testing is done in a non-admin account and cannot chown files.  But then there is a secondary error where fsdecode fails.  It looks like os.py does return the path and not an int but someone thinks it is getting a wrong type.

The other errors are also unicode related.
________________ TestUnicode.test_unicode_paths_square_brackets ________________

self = <testing.functional.test_selection.TestUnicode testMethod=test_unicode_paths_square_brackets>

    def test_unicode_paths_square_brackets(self):
        u""" Test --include and --exclude work with unicode paths with character options in []s and [!]s"""
        p = u"testfiles/select-unicode/"
        self.backup(u"full", u"testfiles/select-unicode",
                    options=[u"--exclude", p + u"прыклад/пример/例/Παράδειγμα/उदाहरण.txt",
                             u"--exclude", p + u"пры[к,и,р]лад/пример/例/Παράδειγμα/דוגמא.txt",
                             u"--exclude", p + u"прыклад/пр[!a,b,c]мер/例/მაგალითი/",
                             u"--include", p + u"прыклад/при[g,м,д]ер/例/",
                             u"--exclude", p + u"прыклад/пример/",
                             u"--include", p + u"прыклад/",
                             u"--include", p + u"օրինակ.txt",
                             u"--exclude", p + u"**"])
        self.restore()
        restore_dir = u"testfiles/restore_out"
        restored = self.directory_tree_to_list_of_lists(restore_dir)
        self.assertEqual(restored, [[u"прыклад", u"օրինակ.txt"],
>                                   [u"пример", u"উদাহরণ"], [u"例"], [u"Παράδειγμα"], [u"ઉદાહરણ.log"]])
E       AssertionError: Lists differ: [['пр[48 chars]'], ['Παράδειγμα'], ['דוגמא.txt', 'उदाहरण.txt', 'ઉદાહરણ.log']] != [['пр[48 chars]'], ['Παράδειγμα'], ['ઉદાહરણ.log']]
E       
E       First differing element 3:
E       ['Παράδειγμα']
E       ['Παράδειγμα']
E       
E         [['прыклад', 'օրինակ.txt'],
E          ['пример', 'উদাহরণ'],
E          ['例'],
E       -  ['Παράδειγμα'],
E       ?       ^^
E       
E       +  ['Παράδειγμα'],
E       ?       ^
E       
E       -  ['דוגמא.txt', 'उदाहरण.txt', 'ઉદાહરણ.log']]
E       +  ['ઉદાહરણ.log']]
The strings that are listed as different (element 3) but I am not quite sure what the issue is.  Unicode should be completely cross platform…  That is the point! :-)

-Scott


On Sep 1, 2019, at 5:49 PM, Scott Hannahs via Duplicity-talk <[hidden email]> wrote:

Duplicity wizards,

I think I have successfully build duplicity version 0.8.04 on Mac OS X (python 3.7 and Mac OS X 10.11).  This was upgrading from my previous installation using python 2.7.

However the testing phase seems to fail  a  lot of tests.  The first group I understand since there is no tahoe backend installed.  I can live with that and will get around to adding backend support later.


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Ken,

I am obviously doing something wrong or at least different than is done by “python3.7 setup.py test”.

I change to the directory where  the source is unpacked.  I have already built duplicity in this directory.  This is the level of the directory that has the file tox.ini in it.  The entire folder structure from here is owned by the fink-bld user.  That account has read/write access to all files and folders in the directory tree.

I then issue the command
sudo -u fink-bld tox-py37 -e py37

This immediately crashes out! 
GLOB sdist-make: /sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/setup.py
Traceback (most recent call last):
  File "/sw/bin/tox-py37", line 11, in <module>
    load_entry_point('tox==3.0.0', 'console_scripts', 'tox')()
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 40, in run_main
    main(args)
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 46, in main
    retcode = Session(config).runcommand()
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 415, in runcommand
    return self.subcommand_test()
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 593, in subcommand_test
    path = self.get_installpkg_path()
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 569, in get_installpkg_path
    path = self._makesdist()
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 449, in _makesdist
    return self.config.distdir.listdir()[0]
IndexError: list index out of range

And I get the same just running “run-tests” in the testing directory.

I am now thoroughly confused as I thought setup-py test.  If I issue the command "sudo -u fink-bld python3.7 setup.py test” then I get the output I sent previously of passing most tests.

-Scott


On Sep 3, 2019, at 1:52 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

When I run "tox -e py37" on my Mac I have no errors.

Your errors in test_selection are the same as on py36, very weird.

The permission errors causing test_replicate and test_restart are also odd.  Do you have the system locked down more than normal?  I test using just my own creds, not root..

The unicode errors when reporting the permission errors are something that needs to be fixed, but are a distraction at this point.

Try running under tox instead of fink just to eliminate one variable.

I've pasted my run.  Perhaps there is a version issue between our two systems?

...Thanks,
...Ken

ken@dione:~/workspace/duplicity-src8$ tox -e py37
GLOB sdist-make: /Users/ken/workspace/duplicity-src8/setup.py
py37 create: /Users/ken/workspace/duplicity-src8/.tox/py37
py37 installdeps: -rrequirements.txt
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 inst: /Users/ken/workspace/duplicity-src8/.tox/.tmp/package/1/duplicity-0.0dev.tar.gz
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 installed: alabaster==0.7.12,astroid==2.2.5,atomicwrites==1.3.0,attrs==19.1.0,Babel==2.7.0,certifi==2019.6.16,chardet==3.0.4,coverage==4.5.4,docutils==0.15.2,duplicity==0.0.dev0,fasteners==0.15,fastimport==0.9.2,filelock==3.0.12,future==0.17.1,idna==2.8,imagesize==1.1.0,importlib-metadata==0.20,isort==4.3.21,Jinja2==2.10.1,lazy-object-proxy==1.4.2,MarkupSafe==1.1.1,mccabe==0.6.1,mock==3.0.5,monotonic==1.5,more-itertools==7.2.0,packaging==19.1,pexpect==4.7.0,pluggy==0.12.0,ptyprocess==0.6.0,py==1.8.0,pycodestyle==2.5.0,Pygments==2.4.2,pylint==2.3.1,pyparsing==2.4.2,pytest==5.1.2,pytest-cov==2.7.1,pytest-runner==5.1,pytz==2019.2,requests==2.22.0,six==1.12.0,snowballstemmer==1.9.0,Sphinx==2.2.0,sphinxcontrib-applehelp==1.0.1,sphinxcontrib-devhelp==1.0.1,sphinxcontrib-htmlhelp==1.0.2,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.2,sphinxcontrib-serializinghtml==1.1.3,toml==0.10.0,tox==3.14.0,typed-ast==1.4.0,urllib3==1.25.3,virtualenv==16.7.5,wcwidth==0.1.7,wrapt==1.11.2,zipp==0.6.0
py37 run-test-pre: PYTHONHASHSEED='3996759775'
py37 run-test: commands[0] | python compilec.py
running build
running build_ext
building '_librsync' extension
creating build
creating build/temp.macosx-10.14-x86_64-3.7
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c _librsyncmodule.c -o build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o
creating build/lib.macosx-10.14-x86_64-3.7
clang -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o -L/usr/local/lib -lrsync -o build/lib.macosx-10.14-x86_64-3.7/_librsync.cpython-37m-darwin.so
py37 run-test: commands[1] | pytest --cov=duplicity --cov-append --cov-report=
================================================= test session starts ==================================================
platform darwin -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.12.0
cachedir: .tox/py37/.pytest_cache
rootdir: /Users/ken/workspace/duplicity-src8, inifile: tox.ini, testpaths: testing/unit, testing/functional
plugins: cov-2.7.1
collected 428 items                                                                                                    

testing/unit/test_backend.py ............                                                                        [  2%]
testing/unit/test_backend_instance.py ............................................................               [ 16%]
testing/unit/test_collections.py ........                                                                        [ 18%]
testing/unit/test_diffdir.py .......                                                                             [ 20%]
testing/unit/test_dup_temp.py ..                                                                                 [ 20%]
testing/unit/test_dup_time.py ......................                                                             [ 25%]
testing/unit/test_file_naming.py .............                                                                   [ 28%]
testing/unit/test_globmatch.py ...........................                                                       [ 35%]
testing/unit/test_gpg.py ..........                                                                              [ 37%]
testing/unit/test_gpginterface.py ......                                                                         [ 39%]
testing/unit/test_lazy.py ............................                                                           [ 45%]
testing/unit/test_manifest.py .....                                                                              [ 46%]
testing/unit/test_patchdir.py ........                                                                           [ 48%]
testing/unit/test_path.py .....                                                                                  [ 49%]
testing/unit/test_selection.py ............................................................................      [ 67%]
testing/unit/test_statistics.py .......                                                                          [ 69%]
testing/unit/test_tarfile.py .                                                                                   [ 69%]
testing/unit/test_tempdir.py ..                                                                                  [ 69%]
testing/unit/test_util.py .                                                                                      [ 70%]
testing/functional/test_badupload.py .                                                                           [ 70%]
testing/functional/test_cleanup.py ...                                                                           [ 71%]
testing/functional/test_final.py ...........................                                                     [ 77%]
testing/functional/test_log.py .                                                                                 [ 77%]
testing/functional/test_rdiffdir.py .                                                                            [ 77%]
testing/functional/test_replicate.py ...                                                                         [ 78%]
testing/functional/test_restart.py ................................                                              [ 85%]
testing/functional/test_selection.py .......................................ss...........                        [ 98%]
testing/functional/test_verify.py ........                                                                       [100%]

=================================================== warnings summary ===================================================
.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Iterable

.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

testing/unit/test_globmatch.py:100
  /Users/ken/workspace/duplicity-src8/testing/unit/test_globmatch.py:100: DeprecationWarning: invalid escape sequence \/
    u"\/usr\/[^/]*\/bin\/")

-- Docs: https://docs.pytest.org/en/latest/warnings.html


================================ 426 passed, 2 skipped, 3 warnings in 560.97s (0:09:20) ================================
_______________________________________________________ summary ________________________________________________________
  py37: commands succeeded
  congratulations :)

On Mon, Sep 2, 2019 at 4:54 PM Scott Hannahs <[hidden email]> wrote:
Ken,

I am running them under the test phase of “fink”.  I use fink rather than homebrew.

so the test script is the following.  I am not sure if the chmod is still necessary but it used to be.  The build is done while running as an unprivileged user called “fink-bld”. This script should call tox directly and not be an issue.  I am fairly happy that it passed 409 different tests!

TestScript: <<
        #!/bin/sh -ev
        ulimit -n 5120
        chmod 0700 `pwd`/testing/gnupg
        %p/bin/python3.7 setup.py test
<<

I am using the install of python from the 3.7 sources.  I patched duplicity so that every line “#!/bin/env python” was replaced with ‘#!/bin/env python3.7” as I have done in the past for python 2.7.

The python version is 3.7.4.
% python3.7 --version
Python 3.7.4

I heard about the errors in 3.6 and so chose to build under 3.7.  I am running a fairly old OS but python 2.7 is also getting a bit old.

The script above should call tox directly and that is the 3.7 version as far as I can tell.  I can run tox directly if that helps.

% tox --version
3.0.0 imported from /sw/lib/python3.7/site-packages/tox/__init__.py

I wasn’t sure I could attach a file to this list, but I will try with the test log.  There is some fink specific listing as it builds duplicity and then goes straight into the tests.  This file has both he standard and error output redirected to it.  I had some tests fail under python2.7 previously and we never got to the bottom of that.

The OS is Mac OS 10.11.6 and maybe had a flawed installation of unicode?  Or an old unicode missing some newer definitions?  The characters seem to show up in the error output correctly but maybe that is an issue.  However the unicode comparison should be comparing the 4 hex digits of a unicode character and that should work.

Thanks for the help.
-Scott


On Sep 2, 2019, at 5:16 PM, Kenneth Loafman <[hidden email]> wrote:

Hi Scott,

How are you running the tests?  We recommend just using "tox" as the test driver.

Are you using python37 from homebrew or macports.  I've tested using the homebrew install under 10.14.

You should not be getting any errors on py27 or py37.  There are 4 known errors on py36.

A bit more detail, e.g. full test logs, would help when reporting.

...Thanks,
...Ken


On Mon, Sep 2, 2019 at 3:58 PM Scott Hannahs via Duplicity-talk <[hidden email]> wrote:
I am going to reply to my own message.. :-)  Most of the errors were due to my not reading my own build notes. :-(  I have run the tests again and seem to have several errors due to unicode issues which was the big advantage of moving to duplicity 8.  I am down to 17 errors and 3 warnings:
======== 17 failed, 409 passed, 2 skipped, 3 warnings in 894.00 seconds ========


These two errors have me stumped not being a python or unicode expert.  os.py looks reasonable to me and should return a string or path.
b'  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/duplicity/path.py", line 476, in <lambda>'
b'    util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))'
b"PermissionError: [Errno 1] Operation not permitted: b'testfiles/cache/3f6f90580e457c90e7202bc33cd9e363/duplicity-full-signatures.19700102T034640Z.sigtar.gz'"
b'During handling of the above exception, another exception occurred:'
b'Traceback (most recent call last):'
b'  File "../bin/duplicity", line 157, in <module>'
b'    if u"Forced assertion for testing" in util.uexc(e):'
b'  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/duplicity/util.py", line 120, in uexc'
b'    return fsdecode(m)'
b'  File "/sw/lib/python3.7/os.py", line 821, in fsdecode'
b'    filename = fspath(filename)  # Does type-checking of `filename`.'
b'TypeError: expected str, bytes or os.PathLike object, not int'
...return_val: 1

So there is a permission error which may be normal since the testing is done in a non-admin account and cannot chown files.  But then there is a secondary error where fsdecode fails.  It looks like os.py does return the path and not an int but someone thinks it is getting a wrong type.

The other errors are also unicode related.
________________ TestUnicode.test_unicode_paths_square_brackets ________________

self = <testing.functional.test_selection.TestUnicode testMethod=test_unicode_paths_square_brackets>

    def test_unicode_paths_square_brackets(self):
        u""" Test --include and --exclude work with unicode paths with character options in []s and [!]s"""
        p = u"testfiles/select-unicode/"
        self.backup(u"full", u"testfiles/select-unicode",
                    options=[u"--exclude", p + u"прыклад/пример/例/Παράδειγμα/उदाहरण.txt",
                             u"--exclude", p + u"пры[к,и,р]лад/пример/例/Παράδειγμα/דוגמא.txt",
                             u"--exclude", p + u"прыклад/пр[!a,b,c]мер/例/მაგალითი/",
                             u"--include", p + u"прыклад/при[g,м,д]ер/例/",
                             u"--exclude", p + u"прыклад/пример/",
                             u"--include", p + u"прыклад/",
                             u"--include", p + u"օրինակ.txt",
                             u"--exclude", p + u"**"])
        self.restore()
        restore_dir = u"testfiles/restore_out"
        restored = self.directory_tree_to_list_of_lists(restore_dir)
        self.assertEqual(restored, [[u"прыклад", u"օրինակ.txt"],
>                                   [u"пример", u"উদাহরণ"], [u"例"], [u"Παράδειγμα"], [u"ઉદાહરણ.log"]])
E       AssertionError: Lists differ: [['пр[48 chars]'], ['Παράδειγμα'], ['דוגמא.txt', 'उदाहरण.txt', 'ઉદાહરણ.log']] != [['пр[48 chars]'], ['Παράδειγμα'], ['ઉદાહરણ.log']]
E       
E       First differing element 3:
E       ['Παράδειγμα']
E       ['Παράδειγμα']
E       
E         [['прыклад', 'օրինակ.txt'],
E          ['пример', 'উদাহরণ'],
E          ['例'],
E       -  ['Παράδειγμα'],
E       ?       ^^
E       
E       +  ['Παράδειγμα'],
E       ?       ^
E       
E       -  ['דוגמא.txt', 'उदाहरण.txt', 'ઉદાહરણ.log']]
E       +  ['ઉદાહરણ.log']]
The strings that are listed as different (element 3) but I am not quite sure what the issue is.  Unicode should be completely cross platform…  That is the point! :-)

-Scott


On Sep 1, 2019, at 5:49 PM, Scott Hannahs via Duplicity-talk <[hidden email]> wrote:

Duplicity wizards,

I think I have successfully build duplicity version 0.8.04 on Mac OS X (python 3.7 and Mac OS X 10.11).  This was upgrading from my previous installation using python 2.7.

However the testing phase seems to fail  a  lot of tests.  The first group I understand since there is no tahoe backend installed.  I can live with that and will get around to adding backend support later.


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk



_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Ken,

If I look at .tox/log the log file created tonight is:

actionid: tox
msg: packaging
cmdargs: ['/sw/bin/python3.7', local('/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/.tox/dist')]

running sdist
failed to open trace file: [Errno 13] Permission denied: '/Users/sth/.bzr.log'
bzr: ERROR: Not a branch: "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/".
fatal: Not a git repository (or any of the parent directories): .git
Processing version 0.8.04
Traceback (most recent call last):
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 134, in <module>
    Main()
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 125, in Main
    tarfile = MakeTar()
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 90, in MakeTar
    assert not os.mkdir(os.path.join(tardir, u"po", lang)), lang
FileNotFoundError: [Errno 2] No such file or directory: 'duplicity-0.8.04/po/bg'
mv: rename duplicity-0.8.04.tar.gz to .tox/dist/duplicity-0.8.04.tar.gz: No such file or directory
/sw/lib/python3.7/site-packages/setuptools/dist.py:398: UserWarning: Normalizing '0.8.04' to '0.8.4'
  normalized_version,

It seems that the testing needs access to a .bzr.log file in the home directory.  This could be an issue since I am trying to build this in an anonymous account as security testing for side effects.

I have tried running the test ("tox -e py37”) in my own directory as myself and still get the error the key lines are the bzr and git errors.  Why is the testing system trying to sync using bzr or git?
running sdist
bzr: ERROR: Not a branch: "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/".
fatal: Not a git repository (or any of the parent directories): .git
Processing version 0.8.04


-Scott


On Sep 3, 2019, at 1:52 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

When I run "tox -e py37" on my Mac I have no errors.

Your errors in test_selection are the same as on py36, very weird.

The permission errors causing test_replicate and test_restart are also odd.  Do you have the system locked down more than normal?  I test using just my own creds, not root..

The unicode errors when reporting the permission errors are something that needs to be fixed, but are a distraction at this point.

Try running under tox instead of fink just to eliminate one variable.

I've pasted my run.  Perhaps there is a version issue between our two systems?

...Thanks,
...Ken

ken@dione:~/workspace/duplicity-src8$ tox -e py37
GLOB sdist-make: /Users/ken/workspace/duplicity-src8/setup.py
py37 create: /Users/ken/workspace/duplicity-src8/.tox/py37
py37 installdeps: -rrequirements.txt
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 inst: /Users/ken/workspace/duplicity-src8/.tox/.tmp/package/1/duplicity-0.0dev.tar.gz
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 installed: alabaster==0.7.12,astroid==2.2.5,atomicwrites==1.3.0,attrs==19.1.0,Babel==2.7.0,certifi==2019.6.16,chardet==3.0.4,coverage==4.5.4,docutils==0.15.2,duplicity==0.0.dev0,fasteners==0.15,fastimport==0.9.2,filelock==3.0.12,future==0.17.1,idna==2.8,imagesize==1.1.0,importlib-metadata==0.20,isort==4.3.21,Jinja2==2.10.1,lazy-object-proxy==1.4.2,MarkupSafe==1.1.1,mccabe==0.6.1,mock==3.0.5,monotonic==1.5,more-itertools==7.2.0,packaging==19.1,pexpect==4.7.0,pluggy==0.12.0,ptyprocess==0.6.0,py==1.8.0,pycodestyle==2.5.0,Pygments==2.4.2,pylint==2.3.1,pyparsing==2.4.2,pytest==5.1.2,pytest-cov==2.7.1,pytest-runner==5.1,pytz==2019.2,requests==2.22.0,six==1.12.0,snowballstemmer==1.9.0,Sphinx==2.2.0,sphinxcontrib-applehelp==1.0.1,sphinxcontrib-devhelp==1.0.1,sphinxcontrib-htmlhelp==1.0.2,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.2,sphinxcontrib-serializinghtml==1.1.3,toml==0.10.0,tox==3.14.0,typed-ast==1.4.0,urllib3==1.25.3,virtualenv==16.7.5,wcwidth==0.1.7,wrapt==1.11.2,zipp==0.6.0
py37 run-test-pre: PYTHONHASHSEED='3996759775'
py37 run-test: commands[0] | python compilec.py
running build
running build_ext
building '_librsync' extension
creating build
creating build/temp.macosx-10.14-x86_64-3.7
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c _librsyncmodule.c -o build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o
creating build/lib.macosx-10.14-x86_64-3.7
clang -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o -L/usr/local/lib -lrsync -o build/lib.macosx-10.14-x86_64-3.7/_librsync.cpython-37m-darwin.so
py37 run-test: commands[1] | pytest --cov=duplicity --cov-append --cov-report=
================================================= test session starts ==================================================
platform darwin -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.12.0
cachedir: .tox/py37/.pytest_cache
rootdir: /Users/ken/workspace/duplicity-src8, inifile: tox.ini, testpaths: testing/unit, testing/functional
plugins: cov-2.7.1
collected 428 items                                                                                                    

testing/unit/test_backend.py ............                                                                        [  2%]
testing/unit/test_backend_instance.py ............................................................               [ 16%]
testing/unit/test_collections.py ........                                                                        [ 18%]
testing/unit/test_diffdir.py .......                                                                             [ 20%]
testing/unit/test_dup_temp.py ..                                                                                 [ 20%]
testing/unit/test_dup_time.py ......................                                                             [ 25%]
testing/unit/test_file_naming.py .............                                                                   [ 28%]
testing/unit/test_globmatch.py ...........................                                                       [ 35%]
testing/unit/test_gpg.py ..........                                                                              [ 37%]
testing/unit/test_gpginterface.py ......                                                                         [ 39%]
testing/unit/test_lazy.py ............................                                                           [ 45%]
testing/unit/test_manifest.py .....                                                                              [ 46%]
testing/unit/test_patchdir.py ........                                                                           [ 48%]
testing/unit/test_path.py .....                                                                                  [ 49%]
testing/unit/test_selection.py ............................................................................      [ 67%]
testing/unit/test_statistics.py .......                                                                          [ 69%]
testing/unit/test_tarfile.py .                                                                                   [ 69%]
testing/unit/test_tempdir.py ..                                                                                  [ 69%]
testing/unit/test_util.py .                                                                                      [ 70%]
testing/functional/test_badupload.py .                                                                           [ 70%]
testing/functional/test_cleanup.py ...                                                                           [ 71%]
testing/functional/test_final.py ...........................                                                     [ 77%]
testing/functional/test_log.py .                                                                                 [ 77%]
testing/functional/test_rdiffdir.py .                                                                            [ 77%]
testing/functional/test_replicate.py ...                                                                         [ 78%]
testing/functional/test_restart.py ................................                                              [ 85%]
testing/functional/test_selection.py .......................................ss...........                        [ 98%]
testing/functional/test_verify.py ........                                                                       [100%]

=================================================== warnings summary ===================================================
.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Iterable

.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

testing/unit/test_globmatch.py:100
  /Users/ken/workspace/duplicity-src8/testing/unit/test_globmatch.py:100: DeprecationWarning: invalid escape sequence \/
    u"\/usr\/[^/]*\/bin\/")

-- Docs: https://docs.pytest.org/en/latest/warnings.html


================================ 426 passed, 2 skipped, 3 warnings in 560.97s (0:09:20) ================================
_______________________________________________________ summary ________________________________________________________
  py37: commands succeeded
  congratulations :)



_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Scott,   

Have you tried updating your Mac?  It's several years behind current software.

duplicity uses a lot of python modules.  The top of the tox listing from earlier shows the version numbers of the ones used in testing.  Again, maybe an update?

Why bzr or git: dist/makedist makes an install tar from the current directory.  It uses bzr or git to get a list of the released filenames, builds a tarball using just those files, then tox uses the tarball to build the virtual environment under .tox/py37.  Keeps the distribution clean that way, no stray files from the developer.

That said, tox will not work for testing a tarball distribution.  You'll have to use "setup.py test" and I'll have to rethink dist/makedist.  My bad!

...Ken






On Tue, Sep 3, 2019 at 8:46 PM Scott Hannahs via Duplicity-talk <[hidden email]> wrote:
Ken,

If I look at .tox/log the log file created tonight is:

actionid: tox
msg: packaging
cmdargs: ['/sw/bin/python3.7', local('/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/.tox/dist')]

running sdist
failed to open trace file: [Errno 13] Permission denied: '/Users/sth/.bzr.log'
bzr: ERROR: Not a branch: "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/".
fatal: Not a git repository (or any of the parent directories): .git
Processing version 0.8.04
Traceback (most recent call last):
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 134, in <module>
    Main()
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 125, in Main
    tarfile = MakeTar()
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 90, in MakeTar
    assert not os.mkdir(os.path.join(tardir, u"po", lang)), lang
FileNotFoundError: [Errno 2] No such file or directory: 'duplicity-0.8.04/po/bg'
mv: rename duplicity-0.8.04.tar.gz to .tox/dist/duplicity-0.8.04.tar.gz: No such file or directory
/sw/lib/python3.7/site-packages/setuptools/dist.py:398: UserWarning: Normalizing '0.8.04' to '0.8.4'
  normalized_version,

It seems that the testing needs access to a .bzr.log file in the home directory.  This could be an issue since I am trying to build this in an anonymous account as security testing for side effects.

I have tried running the test ("tox -e py37”) in my own directory as myself and still get the error the key lines are the bzr and git errors.  Why is the testing system trying to sync using bzr or git?
running sdist
bzr: ERROR: Not a branch: "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/".
fatal: Not a git repository (or any of the parent directories): .git
Processing version 0.8.04


-Scott


On Sep 3, 2019, at 1:52 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

When I run "tox -e py37" on my Mac I have no errors.

Your errors in test_selection are the same as on py36, very weird.

The permission errors causing test_replicate and test_restart are also odd.  Do you have the system locked down more than normal?  I test using just my own creds, not root..

The unicode errors when reporting the permission errors are something that needs to be fixed, but are a distraction at this point.

Try running under tox instead of fink just to eliminate one variable.

I've pasted my run.  Perhaps there is a version issue between our two systems?

...Thanks,
...Ken

ken@dione:~/workspace/duplicity-src8$ tox -e py37
GLOB sdist-make: /Users/ken/workspace/duplicity-src8/setup.py
py37 create: /Users/ken/workspace/duplicity-src8/.tox/py37
py37 installdeps: -rrequirements.txt
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 inst: /Users/ken/workspace/duplicity-src8/.tox/.tmp/package/1/duplicity-0.0dev.tar.gz
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 installed: alabaster==0.7.12,astroid==2.2.5,atomicwrites==1.3.0,attrs==19.1.0,Babel==2.7.0,certifi==2019.6.16,chardet==3.0.4,coverage==4.5.4,docutils==0.15.2,duplicity==0.0.dev0,fasteners==0.15,fastimport==0.9.2,filelock==3.0.12,future==0.17.1,idna==2.8,imagesize==1.1.0,importlib-metadata==0.20,isort==4.3.21,Jinja2==2.10.1,lazy-object-proxy==1.4.2,MarkupSafe==1.1.1,mccabe==0.6.1,mock==3.0.5,monotonic==1.5,more-itertools==7.2.0,packaging==19.1,pexpect==4.7.0,pluggy==0.12.0,ptyprocess==0.6.0,py==1.8.0,pycodestyle==2.5.0,Pygments==2.4.2,pylint==2.3.1,pyparsing==2.4.2,pytest==5.1.2,pytest-cov==2.7.1,pytest-runner==5.1,pytz==2019.2,requests==2.22.0,six==1.12.0,snowballstemmer==1.9.0,Sphinx==2.2.0,sphinxcontrib-applehelp==1.0.1,sphinxcontrib-devhelp==1.0.1,sphinxcontrib-htmlhelp==1.0.2,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.2,sphinxcontrib-serializinghtml==1.1.3,toml==0.10.0,tox==3.14.0,typed-ast==1.4.0,urllib3==1.25.3,virtualenv==16.7.5,wcwidth==0.1.7,wrapt==1.11.2,zipp==0.6.0
py37 run-test-pre: PYTHONHASHSEED='3996759775'
py37 run-test: commands[0] | python compilec.py
running build
running build_ext
building '_librsync' extension
creating build
creating build/temp.macosx-10.14-x86_64-3.7
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c _librsyncmodule.c -o build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o
creating build/lib.macosx-10.14-x86_64-3.7
clang -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o -L/usr/local/lib -lrsync -o build/lib.macosx-10.14-x86_64-3.7/_librsync.cpython-37m-darwin.so
py37 run-test: commands[1] | pytest --cov=duplicity --cov-append --cov-report=
================================================= test session starts ==================================================
platform darwin -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.12.0
cachedir: .tox/py37/.pytest_cache
rootdir: /Users/ken/workspace/duplicity-src8, inifile: tox.ini, testpaths: testing/unit, testing/functional
plugins: cov-2.7.1
collected 428 items                                                                                                    

testing/unit/test_backend.py ............                                                                        [  2%]
testing/unit/test_backend_instance.py ............................................................               [ 16%]
testing/unit/test_collections.py ........                                                                        [ 18%]
testing/unit/test_diffdir.py .......                                                                             [ 20%]
testing/unit/test_dup_temp.py ..                                                                                 [ 20%]
testing/unit/test_dup_time.py ......................                                                             [ 25%]
testing/unit/test_file_naming.py .............                                                                   [ 28%]
testing/unit/test_globmatch.py ...........................                                                       [ 35%]
testing/unit/test_gpg.py ..........                                                                              [ 37%]
testing/unit/test_gpginterface.py ......                                                                         [ 39%]
testing/unit/test_lazy.py ............................                                                           [ 45%]
testing/unit/test_manifest.py .....                                                                              [ 46%]
testing/unit/test_patchdir.py ........                                                                           [ 48%]
testing/unit/test_path.py .....                                                                                  [ 49%]
testing/unit/test_selection.py ............................................................................      [ 67%]
testing/unit/test_statistics.py .......                                                                          [ 69%]
testing/unit/test_tarfile.py .                                                                                   [ 69%]
testing/unit/test_tempdir.py ..                                                                                  [ 69%]
testing/unit/test_util.py .                                                                                      [ 70%]
testing/functional/test_badupload.py .                                                                           [ 70%]
testing/functional/test_cleanup.py ...                                                                           [ 71%]
testing/functional/test_final.py ...........................                                                     [ 77%]
testing/functional/test_log.py .                                                                                 [ 77%]
testing/functional/test_rdiffdir.py .                                                                            [ 77%]
testing/functional/test_replicate.py ...                                                                         [ 78%]
testing/functional/test_restart.py ................................                                              [ 85%]
testing/functional/test_selection.py .......................................ss...........                        [ 98%]
testing/functional/test_verify.py ........                                                                       [100%]

=================================================== warnings summary ===================================================
.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Iterable

.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

testing/unit/test_globmatch.py:100
  /Users/ken/workspace/duplicity-src8/testing/unit/test_globmatch.py:100: DeprecationWarning: invalid escape sequence \/
    u"\/usr\/[^/]*\/bin\/")

-- Docs: https://docs.pytest.org/en/latest/warnings.html


================================ 426 passed, 2 skipped, 3 warnings in 560.97s (0:09:20) ================================
_______________________________________________________ summary ________________________________________________________
  py37: commands succeeded
  congratulations :)


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Scott,

I've attached a .tgz with fixes to allow you to run tox from the tarball.  They are in the trunk if you want.

Extract the two files into the <duplicity-root>/dist subdirectory and tox will run.

...Ken


On Wed, Sep 4, 2019 at 10:53 AM Kenneth Loafman <[hidden email]> wrote:
Scott,   

Have you tried updating your Mac?  It's several years behind current software.

duplicity uses a lot of python modules.  The top of the tox listing from earlier shows the version numbers of the ones used in testing.  Again, maybe an update?

Why bzr or git: dist/makedist makes an install tar from the current directory.  It uses bzr or git to get a list of the released filenames, builds a tarball using just those files, then tox uses the tarball to build the virtual environment under .tox/py37.  Keeps the distribution clean that way, no stray files from the developer.

That said, tox will not work for testing a tarball distribution.  You'll have to use "setup.py test" and I'll have to rethink dist/makedist.  My bad!

...Ken






On Tue, Sep 3, 2019 at 8:46 PM Scott Hannahs via Duplicity-talk <[hidden email]> wrote:
Ken,

If I look at .tox/log the log file created tonight is:

actionid: tox
msg: packaging
cmdargs: ['/sw/bin/python3.7', local('/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/.tox/dist')]

running sdist
failed to open trace file: [Errno 13] Permission denied: '/Users/sth/.bzr.log'
bzr: ERROR: Not a branch: "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/".
fatal: Not a git repository (or any of the parent directories): .git
Processing version 0.8.04
Traceback (most recent call last):
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 134, in <module>
    Main()
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 125, in Main
    tarfile = MakeTar()
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 90, in MakeTar
    assert not os.mkdir(os.path.join(tardir, u"po", lang)), lang
FileNotFoundError: [Errno 2] No such file or directory: 'duplicity-0.8.04/po/bg'
mv: rename duplicity-0.8.04.tar.gz to .tox/dist/duplicity-0.8.04.tar.gz: No such file or directory
/sw/lib/python3.7/site-packages/setuptools/dist.py:398: UserWarning: Normalizing '0.8.04' to '0.8.4'
  normalized_version,

It seems that the testing needs access to a .bzr.log file in the home directory.  This could be an issue since I am trying to build this in an anonymous account as security testing for side effects.

I have tried running the test ("tox -e py37”) in my own directory as myself and still get the error the key lines are the bzr and git errors.  Why is the testing system trying to sync using bzr or git?
running sdist
bzr: ERROR: Not a branch: "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/".
fatal: Not a git repository (or any of the parent directories): .git
Processing version 0.8.04


-Scott


On Sep 3, 2019, at 1:52 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

When I run "tox -e py37" on my Mac I have no errors.

Your errors in test_selection are the same as on py36, very weird.

The permission errors causing test_replicate and test_restart are also odd.  Do you have the system locked down more than normal?  I test using just my own creds, not root..

The unicode errors when reporting the permission errors are something that needs to be fixed, but are a distraction at this point.

Try running under tox instead of fink just to eliminate one variable.

I've pasted my run.  Perhaps there is a version issue between our two systems?

...Thanks,
...Ken

ken@dione:~/workspace/duplicity-src8$ tox -e py37
GLOB sdist-make: /Users/ken/workspace/duplicity-src8/setup.py
py37 create: /Users/ken/workspace/duplicity-src8/.tox/py37
py37 installdeps: -rrequirements.txt
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 inst: /Users/ken/workspace/duplicity-src8/.tox/.tmp/package/1/duplicity-0.0dev.tar.gz
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 installed: alabaster==0.7.12,astroid==2.2.5,atomicwrites==1.3.0,attrs==19.1.0,Babel==2.7.0,certifi==2019.6.16,chardet==3.0.4,coverage==4.5.4,docutils==0.15.2,duplicity==0.0.dev0,fasteners==0.15,fastimport==0.9.2,filelock==3.0.12,future==0.17.1,idna==2.8,imagesize==1.1.0,importlib-metadata==0.20,isort==4.3.21,Jinja2==2.10.1,lazy-object-proxy==1.4.2,MarkupSafe==1.1.1,mccabe==0.6.1,mock==3.0.5,monotonic==1.5,more-itertools==7.2.0,packaging==19.1,pexpect==4.7.0,pluggy==0.12.0,ptyprocess==0.6.0,py==1.8.0,pycodestyle==2.5.0,Pygments==2.4.2,pylint==2.3.1,pyparsing==2.4.2,pytest==5.1.2,pytest-cov==2.7.1,pytest-runner==5.1,pytz==2019.2,requests==2.22.0,six==1.12.0,snowballstemmer==1.9.0,Sphinx==2.2.0,sphinxcontrib-applehelp==1.0.1,sphinxcontrib-devhelp==1.0.1,sphinxcontrib-htmlhelp==1.0.2,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.2,sphinxcontrib-serializinghtml==1.1.3,toml==0.10.0,tox==3.14.0,typed-ast==1.4.0,urllib3==1.25.3,virtualenv==16.7.5,wcwidth==0.1.7,wrapt==1.11.2,zipp==0.6.0
py37 run-test-pre: PYTHONHASHSEED='3996759775'
py37 run-test: commands[0] | python compilec.py
running build
running build_ext
building '_librsync' extension
creating build
creating build/temp.macosx-10.14-x86_64-3.7
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c _librsyncmodule.c -o build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o
creating build/lib.macosx-10.14-x86_64-3.7
clang -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o -L/usr/local/lib -lrsync -o build/lib.macosx-10.14-x86_64-3.7/_librsync.cpython-37m-darwin.so
py37 run-test: commands[1] | pytest --cov=duplicity --cov-append --cov-report=
================================================= test session starts ==================================================
platform darwin -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.12.0
cachedir: .tox/py37/.pytest_cache
rootdir: /Users/ken/workspace/duplicity-src8, inifile: tox.ini, testpaths: testing/unit, testing/functional
plugins: cov-2.7.1
collected 428 items                                                                                                    

testing/unit/test_backend.py ............                                                                        [  2%]
testing/unit/test_backend_instance.py ............................................................               [ 16%]
testing/unit/test_collections.py ........                                                                        [ 18%]
testing/unit/test_diffdir.py .......                                                                             [ 20%]
testing/unit/test_dup_temp.py ..                                                                                 [ 20%]
testing/unit/test_dup_time.py ......................                                                             [ 25%]
testing/unit/test_file_naming.py .............                                                                   [ 28%]
testing/unit/test_globmatch.py ...........................                                                       [ 35%]
testing/unit/test_gpg.py ..........                                                                              [ 37%]
testing/unit/test_gpginterface.py ......                                                                         [ 39%]
testing/unit/test_lazy.py ............................                                                           [ 45%]
testing/unit/test_manifest.py .....                                                                              [ 46%]
testing/unit/test_patchdir.py ........                                                                           [ 48%]
testing/unit/test_path.py .....                                                                                  [ 49%]
testing/unit/test_selection.py ............................................................................      [ 67%]
testing/unit/test_statistics.py .......                                                                          [ 69%]
testing/unit/test_tarfile.py .                                                                                   [ 69%]
testing/unit/test_tempdir.py ..                                                                                  [ 69%]
testing/unit/test_util.py .                                                                                      [ 70%]
testing/functional/test_badupload.py .                                                                           [ 70%]
testing/functional/test_cleanup.py ...                                                                           [ 71%]
testing/functional/test_final.py ...........................                                                     [ 77%]
testing/functional/test_log.py .                                                                                 [ 77%]
testing/functional/test_rdiffdir.py .                                                                            [ 77%]
testing/functional/test_replicate.py ...                                                                         [ 78%]
testing/functional/test_restart.py ................................                                              [ 85%]
testing/functional/test_selection.py .......................................ss...........                        [ 98%]
testing/functional/test_verify.py ........                                                                       [100%]

=================================================== warnings summary ===================================================
.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Iterable

.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

testing/unit/test_globmatch.py:100
  /Users/ken/workspace/duplicity-src8/testing/unit/test_globmatch.py:100: DeprecationWarning: invalid escape sequence \/
    u"\/usr\/[^/]*\/bin\/")

-- Docs: https://docs.pytest.org/en/latest/warnings.html


================================ 426 passed, 2 skipped, 3 warnings in 560.97s (0:09:20) ================================
_______________________________________________________ summary ________________________________________________________
  py37: commands succeeded
  congratulations :)


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

fixes0804.tgz (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Scott,

Any luck?

...Ken


On Thu, Sep 5, 2019 at 1:03 PM Kenneth Loafman <[hidden email]> wrote:
Scott,

I've attached a .tgz with fixes to allow you to run tox from the tarball.  They are in the trunk if you want.

Extract the two files into the <duplicity-root>/dist subdirectory and tox will run.

...Ken


On Wed, Sep 4, 2019 at 10:53 AM Kenneth Loafman <[hidden email]> wrote:
Scott,   

Have you tried updating your Mac?  It's several years behind current software.

duplicity uses a lot of python modules.  The top of the tox listing from earlier shows the version numbers of the ones used in testing.  Again, maybe an update?

Why bzr or git: dist/makedist makes an install tar from the current directory.  It uses bzr or git to get a list of the released filenames, builds a tarball using just those files, then tox uses the tarball to build the virtual environment under .tox/py37.  Keeps the distribution clean that way, no stray files from the developer.

That said, tox will not work for testing a tarball distribution.  You'll have to use "setup.py test" and I'll have to rethink dist/makedist.  My bad!

...Ken






On Tue, Sep 3, 2019 at 8:46 PM Scott Hannahs via Duplicity-talk <[hidden email]> wrote:
Ken,

If I look at .tox/log the log file created tonight is:

actionid: tox
msg: packaging
cmdargs: ['/sw/bin/python3.7', local('/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/.tox/dist')]

running sdist
failed to open trace file: [Errno 13] Permission denied: '/Users/sth/.bzr.log'
bzr: ERROR: Not a branch: "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/".
fatal: Not a git repository (or any of the parent directories): .git
Processing version 0.8.04
Traceback (most recent call last):
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 134, in <module>
    Main()
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 125, in Main
    tarfile = MakeTar()
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 90, in MakeTar
    assert not os.mkdir(os.path.join(tardir, u"po", lang)), lang
FileNotFoundError: [Errno 2] No such file or directory: 'duplicity-0.8.04/po/bg'
mv: rename duplicity-0.8.04.tar.gz to .tox/dist/duplicity-0.8.04.tar.gz: No such file or directory
/sw/lib/python3.7/site-packages/setuptools/dist.py:398: UserWarning: Normalizing '0.8.04' to '0.8.4'
  normalized_version,

It seems that the testing needs access to a .bzr.log file in the home directory.  This could be an issue since I am trying to build this in an anonymous account as security testing for side effects.

I have tried running the test ("tox -e py37”) in my own directory as myself and still get the error the key lines are the bzr and git errors.  Why is the testing system trying to sync using bzr or git?
running sdist
bzr: ERROR: Not a branch: "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/".
fatal: Not a git repository (or any of the parent directories): .git
Processing version 0.8.04


-Scott


On Sep 3, 2019, at 1:52 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

When I run "tox -e py37" on my Mac I have no errors.

Your errors in test_selection are the same as on py36, very weird.

The permission errors causing test_replicate and test_restart are also odd.  Do you have the system locked down more than normal?  I test using just my own creds, not root..

The unicode errors when reporting the permission errors are something that needs to be fixed, but are a distraction at this point.

Try running under tox instead of fink just to eliminate one variable.

I've pasted my run.  Perhaps there is a version issue between our two systems?

...Thanks,
...Ken

ken@dione:~/workspace/duplicity-src8$ tox -e py37
GLOB sdist-make: /Users/ken/workspace/duplicity-src8/setup.py
py37 create: /Users/ken/workspace/duplicity-src8/.tox/py37
py37 installdeps: -rrequirements.txt
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 inst: /Users/ken/workspace/duplicity-src8/.tox/.tmp/package/1/duplicity-0.0dev.tar.gz
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 installed: alabaster==0.7.12,astroid==2.2.5,atomicwrites==1.3.0,attrs==19.1.0,Babel==2.7.0,certifi==2019.6.16,chardet==3.0.4,coverage==4.5.4,docutils==0.15.2,duplicity==0.0.dev0,fasteners==0.15,fastimport==0.9.2,filelock==3.0.12,future==0.17.1,idna==2.8,imagesize==1.1.0,importlib-metadata==0.20,isort==4.3.21,Jinja2==2.10.1,lazy-object-proxy==1.4.2,MarkupSafe==1.1.1,mccabe==0.6.1,mock==3.0.5,monotonic==1.5,more-itertools==7.2.0,packaging==19.1,pexpect==4.7.0,pluggy==0.12.0,ptyprocess==0.6.0,py==1.8.0,pycodestyle==2.5.0,Pygments==2.4.2,pylint==2.3.1,pyparsing==2.4.2,pytest==5.1.2,pytest-cov==2.7.1,pytest-runner==5.1,pytz==2019.2,requests==2.22.0,six==1.12.0,snowballstemmer==1.9.0,Sphinx==2.2.0,sphinxcontrib-applehelp==1.0.1,sphinxcontrib-devhelp==1.0.1,sphinxcontrib-htmlhelp==1.0.2,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.2,sphinxcontrib-serializinghtml==1.1.3,toml==0.10.0,tox==3.14.0,typed-ast==1.4.0,urllib3==1.25.3,virtualenv==16.7.5,wcwidth==0.1.7,wrapt==1.11.2,zipp==0.6.0
py37 run-test-pre: PYTHONHASHSEED='3996759775'
py37 run-test: commands[0] | python compilec.py
running build
running build_ext
building '_librsync' extension
creating build
creating build/temp.macosx-10.14-x86_64-3.7
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c _librsyncmodule.c -o build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o
creating build/lib.macosx-10.14-x86_64-3.7
clang -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o -L/usr/local/lib -lrsync -o build/lib.macosx-10.14-x86_64-3.7/_librsync.cpython-37m-darwin.so
py37 run-test: commands[1] | pytest --cov=duplicity --cov-append --cov-report=
================================================= test session starts ==================================================
platform darwin -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.12.0
cachedir: .tox/py37/.pytest_cache
rootdir: /Users/ken/workspace/duplicity-src8, inifile: tox.ini, testpaths: testing/unit, testing/functional
plugins: cov-2.7.1
collected 428 items                                                                                                    

testing/unit/test_backend.py ............                                                                        [  2%]
testing/unit/test_backend_instance.py ............................................................               [ 16%]
testing/unit/test_collections.py ........                                                                        [ 18%]
testing/unit/test_diffdir.py .......                                                                             [ 20%]
testing/unit/test_dup_temp.py ..                                                                                 [ 20%]
testing/unit/test_dup_time.py ......................                                                             [ 25%]
testing/unit/test_file_naming.py .............                                                                   [ 28%]
testing/unit/test_globmatch.py ...........................                                                       [ 35%]
testing/unit/test_gpg.py ..........                                                                              [ 37%]
testing/unit/test_gpginterface.py ......                                                                         [ 39%]
testing/unit/test_lazy.py ............................                                                           [ 45%]
testing/unit/test_manifest.py .....                                                                              [ 46%]
testing/unit/test_patchdir.py ........                                                                           [ 48%]
testing/unit/test_path.py .....                                                                                  [ 49%]
testing/unit/test_selection.py ............................................................................      [ 67%]
testing/unit/test_statistics.py .......                                                                          [ 69%]
testing/unit/test_tarfile.py .                                                                                   [ 69%]
testing/unit/test_tempdir.py ..                                                                                  [ 69%]
testing/unit/test_util.py .                                                                                      [ 70%]
testing/functional/test_badupload.py .                                                                           [ 70%]
testing/functional/test_cleanup.py ...                                                                           [ 71%]
testing/functional/test_final.py ...........................                                                     [ 77%]
testing/functional/test_log.py .                                                                                 [ 77%]
testing/functional/test_rdiffdir.py .                                                                            [ 77%]
testing/functional/test_replicate.py ...                                                                         [ 78%]
testing/functional/test_restart.py ................................                                              [ 85%]
testing/functional/test_selection.py .......................................ss...........                        [ 98%]
testing/functional/test_verify.py ........                                                                       [100%]

=================================================== warnings summary ===================================================
.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Iterable

.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

testing/unit/test_globmatch.py:100
  /Users/ken/workspace/duplicity-src8/testing/unit/test_globmatch.py:100: DeprecationWarning: invalid escape sequence \/
    u"\/usr\/[^/]*\/bin\/")

-- Docs: https://docs.pytest.org/en/latest/warnings.html


================================ 426 passed, 2 skipped, 3 warnings in 560.97s (0:09:20) ================================
_______________________________________________________ summary ________________________________________________________
  py37: commands succeeded
  congratulations :)


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
Ken,

I unpacked the 0.8.04 tarball.  I put the two files in duplicity-0.8.04/dist subdirectory and overwrote the makedist file.  But I get the same error:
% tox -e py37 -v
using tox.ini: /Users/sth/Downloads/duplicity-0.8.04/tox.ini
using tox-3.0.0 from /sw/lib/python3.7/site-packages/tox/__init__.py
GLOB sdist-make: /Users/sth/Downloads/duplicity-0.8.04/setup.py
  /Users/sth/Downloads/duplicity-0.8.04$ /sw/bin/python3.7 /Users/sth/Downloads/duplicity-0.8.04/setup.py sdist --formats=zip --dist-dir /Users/sth/Downloads/duplicity-0.8.04/.tox/dist >/Users/sth/Downloads/duplicity-0.8.04/.tox/log/tox-0.log
Traceback (most recent call last):
  File "/sw/bin/tox", line 11, in <module>
    load_entry_point('tox==3.0.0', 'console_scripts', 'tox')()
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 40, in run_main
    main(args)
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 46, in main
    retcode = Session(config).runcommand()
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 415, in runcommand
    return self.subcommand_test()
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 593, in subcommand_test
    path = self.get_installpkg_path()
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 569, in get_installpkg_path
    path = self._makesdist()
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 449, in _makesdist
    return self.config.distdir.listdir()[0]
IndexError: list index out of range
The log file has:
% cat .tox/log/tox-0.log 
actionid: tox
msg: packaging
cmdargs: ['/sw/bin/python3.7', local('/Users/sth/Downloads/duplicity-0.8.04/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/Users/sth/Downloads/duplicity-0.8.04/.tox/dist')]

running sdist
Traceback (most recent call last):
  File "/Users/sth/Downloads/duplicity-0.8.04/dist/makedist", line 24, in <module>
    from future import standard_library
ImportError: No module named future
mv: rename duplicity-0.8.04.tar.gz to .tox/dist/duplicity-0.8.04.tar.gz: No such file or directory
/sw/lib/python3.7/site-packages/setuptools/dist.py:398: UserWarning: Normalizing '0.8.04' to '0.8.4'
  normalized_version,
% 

It seems that it is trying to copy a copy of the tar ball from somewhere?  The tarball is actually one directory higher than the working directory.  In Downloads, and the working directory is "Downloads/duplicity-0.8.04” as normal?

I am fine running the tests as "python3.7 setup.py test”.  However I looked at the modules you showed versions for and I have some older versions installed.  Here is a list of the versions I have.  Does anything standout as a possible cause to the few failures I have?  IIRC they were mostly restart issues.

I underlined some of the larger version differences.

-Scott

alabaster 0.7.12 0.7.11
astroid 2.2.5 1.6.3
atomicwrites 1.3.0 1.1.5
attrs 19.1.0 18.1.0
Babel 2.7.0 2.6.0
certifi 2019.6.16 2018.8.13
chardet 3.0.4 3.0.4
coverage 4.5.4 4.5.1
docutils 0.15.2 ??
duplicity 0.0.dev0 0.8.04
fasteners 0.15 0.15
fastimport 0.9.2 0.9.8
filelock 3.0.12 ??
future 0.17.1 0.17.1
idna 2.8 2.7
imagesize 1.1.0 1.1.0
importlib-metadata 0.20 ??
isort 4.3.21 4.3.4
Jinja2 2.10.1 2.10
lazy-object-proxy 1.4.2 1.3.1
MarkupSafe 1.1.1 1.0
mccabe 0.6.1 0.61
mock 3.0.5 2.0.0
monotonic 1.5 1.5
more-itertools 7.2.0 4.3.0
packaging 19.1 17.1
pexpect 4.7.0 4.6.0
pluggy 0.12.0 0.7.1
ptyprocess 0.6.0 0.6.0
py 1.8.0 1.7.0
pycodestyle 2.5.0 2.5.0 (not installed???)
Pygments 2.4.2 2.2.0
pylint 2.3.1 1.8.4
pyparsing 2.4.2 2.2.0
pytest 5.1.2 3.10.1
pytest-cov 2.7.1 2.6.0 (not installed???)
pytest-runner 5.1 4.2
pytz 2019.2 2018.5
requests 2.22.0 2.19.1
six 1.12.0 1.12.0
snowballstemmer 1.9.0 1.2.1
Sphinx 2.2.0 1.8.4
sphinxcontrib-applehelp 1.0.1 ??? (not installed???)
sphinxcontrib-devhelp 1.0.1 ??? (not installed???)
sphinxcontrib-htmlhelp 1.0.2 ??? (not installed???)
sphinxcontrib-jsmath 1.0.1 ??? (not installed???)
sphinxcontrib-qthelp 1.0.2 ??? (not installed???)
sphinxcontrib-serializinghtml 1.1.3 ??? (not installed???)
toml 0.10.0 0.1.11 (not installed???)
tox 3.14.0 3.0.0
typed-ast 1.4.0 ??? (not installed???)
urllib3 1.25.3 1.23
virtualenv 16.7.5 16.0.0
wcwidth 0.1.7 0.1.7 (not installed???)
wrapt 1.11.2 1.10.11
zipp 0.6.0 ??? (not installed???)

On Sep 5, 2019, at 2:03 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

I've attached a .tgz with fixes to allow you to run tox from the tarball.  They are in the trunk if you want.

Extract the two files into the <duplicity-root>/dist subdirectory and tox will run.

...Ken


On Wed, Sep 4, 2019 at 10:53 AM Kenneth Loafman <[hidden email]> wrote:
Scott,   

Have you tried updating your Mac?  It's several years behind current software.

duplicity uses a lot of python modules.  The top of the tox listing from earlier shows the version numbers of the ones used in testing.  Again, maybe an update?

Why bzr or git: dist/makedist makes an install tar from the current directory.  It uses bzr or git to get a list of the released filenames, builds a tarball using just those files, then tox uses the tarball to build the virtual environment under .tox/py37.  Keeps the distribution clean that way, no stray files from the developer.

That said, tox will not work for testing a tarball distribution.  You'll have to use "setup.py test" and I'll have to rethink dist/makedist.  My bad!

...Ken






On Tue, Sep 3, 2019 at 8:46 PM Scott Hannahs via Duplicity-talk <[hidden email]> wrote:
Ken,

If I look at .tox/log the log file created tonight is:

actionid: tox
msg: packaging
cmdargs: ['/sw/bin/python3.7', local('/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/.tox/dist')]

running sdist
failed to open trace file: [Errno 13] Permission denied: '/Users/sth/.bzr.log'
bzr: ERROR: Not a branch: "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/".
fatal: Not a git repository (or any of the parent directories): .git
Processing version 0.8.04
Traceback (most recent call last):
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 134, in <module>
    Main()
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 125, in Main
    tarfile = MakeTar()
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 90, in MakeTar
    assert not os.mkdir(os.path.join(tardir, u"po", lang)), lang
FileNotFoundError: [Errno 2] No such file or directory: 'duplicity-0.8.04/po/bg'
mv: rename duplicity-0.8.04.tar.gz to .tox/dist/duplicity-0.8.04.tar.gz: No such file or directory
/sw/lib/python3.7/site-packages/setuptools/dist.py:398: UserWarning: Normalizing '0.8.04' to '0.8.4'
  normalized_version,

It seems that the testing needs access to a .bzr.log file in the home directory.  This could be an issue since I am trying to build this in an anonymous account as security testing for side effects.

I have tried running the test ("tox -e py37”) in my own directory as myself and still get the error the key lines are the bzr and git errors.  Why is the testing system trying to sync using bzr or git?
running sdist
bzr: ERROR: Not a branch: "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/".
fatal: Not a git repository (or any of the parent directories): .git
Processing version 0.8.04


-Scott


On Sep 3, 2019, at 1:52 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

When I run "tox -e py37" on my Mac I have no errors.

Your errors in test_selection are the same as on py36, very weird.

The permission errors causing test_replicate and test_restart are also odd.  Do you have the system locked down more than normal?  I test using just my own creds, not root..

The unicode errors when reporting the permission errors are something that needs to be fixed, but are a distraction at this point.

Try running under tox instead of fink just to eliminate one variable.

I've pasted my run.  Perhaps there is a version issue between our two systems?

...Thanks,
...Ken

ken@dione:~/workspace/duplicity-src8$ tox -e py37
GLOB sdist-make: /Users/ken/workspace/duplicity-src8/setup.py
py37 create: /Users/ken/workspace/duplicity-src8/.tox/py37
py37 installdeps: -rrequirements.txt
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 inst: /Users/ken/workspace/duplicity-src8/.tox/.tmp/package/1/duplicity-0.0dev.tar.gz
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 installed: alabaster==0.7.12,astroid==2.2.5,atomicwrites==1.3.0,attrs==19.1.0,Babel==2.7.0,certifi==2019.6.16,chardet==3.0.4,coverage==4.5.4,docutils==0.15.2,duplicity==0.0.dev0,fasteners==0.15,fastimport==0.9.2,filelock==3.0.12,future==0.17.1,idna==2.8,imagesize==1.1.0,importlib-metadata==0.20,isort==4.3.21,Jinja2==2.10.1,lazy-object-proxy==1.4.2,MarkupSafe==1.1.1,mccabe==0.6.1,mock==3.0.5,monotonic==1.5,more-itertools==7.2.0,packaging==19.1,pexpect==4.7.0,pluggy==0.12.0,ptyprocess==0.6.0,py==1.8.0,pycodestyle==2.5.0,Pygments==2.4.2,pylint==2.3.1,pyparsing==2.4.2,pytest==5.1.2,pytest-cov==2.7.1,pytest-runner==5.1,pytz==2019.2,requests==2.22.0,six==1.12.0,snowballstemmer==1.9.0,Sphinx==2.2.0,sphinxcontrib-applehelp==1.0.1,sphinxcontrib-devhelp==1.0.1,sphinxcontrib-htmlhelp==1.0.2,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.2,sphinxcontrib-serializinghtml==1.1.3,toml==0.10.0,tox==3.14.0,typed-ast==1.4.0,urllib3==1.25.3,virtualenv==16.7.5,wcwidth==0.1.7,wrapt==1.11.2,zipp==0.6.0
py37 run-test-pre: PYTHONHASHSEED='3996759775'
py37 run-test: commands[0] | python compilec.py
running build
running build_ext
building '_librsync' extension
creating build
creating build/temp.macosx-10.14-x86_64-3.7
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c _librsyncmodule.c -o build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o
creating build/lib.macosx-10.14-x86_64-3.7
clang -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o -L/usr/local/lib -lrsync -o build/lib.macosx-10.14-x86_64-3.7/_librsync.cpython-37m-darwin.so
py37 run-test: commands[1] | pytest --cov=duplicity --cov-append --cov-report=
================================================= test session starts ==================================================
platform darwin -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.12.0
cachedir: .tox/py37/.pytest_cache
rootdir: /Users/ken/workspace/duplicity-src8, inifile: tox.ini, testpaths: testing/unit, testing/functional
plugins: cov-2.7.1
collected 428 items                                                                                                    

testing/unit/test_backend.py ............                                                                        [  2%]
testing/unit/test_backend_instance.py ............................................................               [ 16%]
testing/unit/test_collections.py ........                                                                        [ 18%]
testing/unit/test_diffdir.py .......                                                                             [ 20%]
testing/unit/test_dup_temp.py ..                                                                                 [ 20%]
testing/unit/test_dup_time.py ......................                                                             [ 25%]
testing/unit/test_file_naming.py .............                                                                   [ 28%]
testing/unit/test_globmatch.py ...........................                                                       [ 35%]
testing/unit/test_gpg.py ..........                                                                              [ 37%]
testing/unit/test_gpginterface.py ......                                                                         [ 39%]
testing/unit/test_lazy.py ............................                                                           [ 45%]
testing/unit/test_manifest.py .....                                                                              [ 46%]
testing/unit/test_patchdir.py ........                                                                           [ 48%]
testing/unit/test_path.py .....                                                                                  [ 49%]
testing/unit/test_selection.py ............................................................................      [ 67%]
testing/unit/test_statistics.py .......                                                                          [ 69%]
testing/unit/test_tarfile.py .                                                                                   [ 69%]
testing/unit/test_tempdir.py ..                                                                                  [ 69%]
testing/unit/test_util.py .                                                                                      [ 70%]
testing/functional/test_badupload.py .                                                                           [ 70%]
testing/functional/test_cleanup.py ...                                                                           [ 71%]
testing/functional/test_final.py ...........................                                                     [ 77%]
testing/functional/test_log.py .                                                                                 [ 77%]
testing/functional/test_rdiffdir.py .                                                                            [ 77%]
testing/functional/test_replicate.py ...                                                                         [ 78%]
testing/functional/test_restart.py ................................                                              [ 85%]
testing/functional/test_selection.py .......................................ss...........                        [ 98%]
testing/functional/test_verify.py ........                                                                       [100%]

=================================================== warnings summary ===================================================
.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Iterable

.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

testing/unit/test_globmatch.py:100
  /Users/ken/workspace/duplicity-src8/testing/unit/test_globmatch.py:100: DeprecationWarning: invalid escape sequence \/
    u"\/usr\/[^/]*\/bin\/")

-- Docs: https://docs.pytest.org/en/latest/warnings.html


================================ 426 passed, 2 skipped, 3 warnings in 560.97s (0:09:20) ================================
_______________________________________________________ summary ________________________________________________________
  py37: commands succeeded
  congratulations :)


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
<fixes0804.tgz>


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Scott,

setup.py (uses dist/makedist) is failing because you do not have the future module installed.  There may be more:

    install_requires=[u'fasteners', u'future'],
    tests_require=[u'pytest', u'pytest-runner', u'fasteners', u'future', u'mock', u'pexpect'],

dist/makedist will make a tarball from the files in dist/relfiles, then tox will use that tarball to install in a virtualenv for testing.

The only modules I see that are way out of date is tox and virtualenv.  An update would help.

...Ken


On Sun, Sep 8, 2019 at 12:18 PM Scott Hannahs <[hidden email]> wrote:
Ken,

I unpacked the 0.8.04 tarball.  I put the two files in duplicity-0.8.04/dist subdirectory and overwrote the makedist file.  But I get the same error:
% tox -e py37 -v
using tox.ini: /Users/sth/Downloads/duplicity-0.8.04/tox.ini
using tox-3.0.0 from /sw/lib/python3.7/site-packages/tox/__init__.py
GLOB sdist-make: /Users/sth/Downloads/duplicity-0.8.04/setup.py
  /Users/sth/Downloads/duplicity-0.8.04$ /sw/bin/python3.7 /Users/sth/Downloads/duplicity-0.8.04/setup.py sdist --formats=zip --dist-dir /Users/sth/Downloads/duplicity-0.8.04/.tox/dist >/Users/sth/Downloads/duplicity-0.8.04/.tox/log/tox-0.log
Traceback (most recent call last):
  File "/sw/bin/tox", line 11, in <module>
    load_entry_point('tox==3.0.0', 'console_scripts', 'tox')()
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 40, in run_main
    main(args)
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 46, in main
    retcode = Session(config).runcommand()
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 415, in runcommand
    return self.subcommand_test()
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 593, in subcommand_test
    path = self.get_installpkg_path()
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 569, in get_installpkg_path
    path = self._makesdist()
  File "/sw/lib/python3.7/site-packages/tox/session.py", line 449, in _makesdist
    return self.config.distdir.listdir()[0]
IndexError: list index out of range
The log file has:
% cat .tox/log/tox-0.log 
actionid: tox
msg: packaging
cmdargs: ['/sw/bin/python3.7', local('/Users/sth/Downloads/duplicity-0.8.04/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/Users/sth/Downloads/duplicity-0.8.04/.tox/dist')]

running sdist
Traceback (most recent call last):
  File "/Users/sth/Downloads/duplicity-0.8.04/dist/makedist", line 24, in <module>
    from future import standard_library
ImportError: No module named future
mv: rename duplicity-0.8.04.tar.gz to .tox/dist/duplicity-0.8.04.tar.gz: No such file or directory
/sw/lib/python3.7/site-packages/setuptools/dist.py:398: UserWarning: Normalizing '0.8.04' to '0.8.4'
  normalized_version,
% 

It seems that it is trying to copy a copy of the tar ball from somewhere?  The tarball is actually one directory higher than the working directory.  In Downloads, and the working directory is "Downloads/duplicity-0.8.04” as normal?

I am fine running the tests as "python3.7 setup.py test”.  However I looked at the modules you showed versions for and I have some older versions installed.  Here is a list of the versions I have.  Does anything standout as a possible cause to the few failures I have?  IIRC they were mostly restart issues.

I underlined some of the larger version differences.

-Scott

alabaster 0.7.12 0.7.11
astroid 2.2.5 1.6.3
atomicwrites 1.3.0 1.1.5
attrs 19.1.0 18.1.0
Babel 2.7.0 2.6.0
certifi 2019.6.16 2018.8.13
chardet 3.0.4 3.0.4
coverage 4.5.4 4.5.1
docutils 0.15.2 ??
duplicity 0.0.dev0 0.8.04
fasteners 0.15 0.15
fastimport 0.9.2 0.9.8
filelock 3.0.12 ??
future 0.17.1 0.17.1
idna 2.8 2.7
imagesize 1.1.0 1.1.0
importlib-metadata 0.20 ??
isort 4.3.21 4.3.4
Jinja2 2.10.1 2.10
lazy-object-proxy 1.4.2 1.3.1
MarkupSafe 1.1.1 1.0
mccabe 0.6.1 0.61
mock 3.0.5 2.0.0
monotonic 1.5 1.5
more-itertools 7.2.0 4.3.0
packaging 19.1 17.1
pexpect 4.7.0 4.6.0
pluggy 0.12.0 0.7.1
ptyprocess 0.6.0 0.6.0
py 1.8.0 1.7.0
pycodestyle 2.5.0 2.5.0 (not installed???)
Pygments 2.4.2 2.2.0
pylint 2.3.1 1.8.4
pyparsing 2.4.2 2.2.0
pytest 5.1.2 3.10.1
pytest-cov 2.7.1 2.6.0 (not installed???)
pytest-runner 5.1 4.2
pytz 2019.2 2018.5
requests 2.22.0 2.19.1
six 1.12.0 1.12.0
snowballstemmer 1.9.0 1.2.1
Sphinx 2.2.0 1.8.4
sphinxcontrib-applehelp 1.0.1 ??? (not installed???)
sphinxcontrib-devhelp 1.0.1 ??? (not installed???)
sphinxcontrib-htmlhelp 1.0.2 ??? (not installed???)
sphinxcontrib-jsmath 1.0.1 ??? (not installed???)
sphinxcontrib-qthelp 1.0.2 ??? (not installed???)
sphinxcontrib-serializinghtml 1.1.3 ??? (not installed???)
toml 0.10.0 0.1.11 (not installed???)
tox 3.14.0 3.0.0
typed-ast 1.4.0 ??? (not installed???)
urllib3 1.25.3 1.23
virtualenv 16.7.5 16.0.0
wcwidth 0.1.7 0.1.7 (not installed???)
wrapt 1.11.2 1.10.11
zipp 0.6.0 ??? (not installed???)

On Sep 5, 2019, at 2:03 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

I've attached a .tgz with fixes to allow you to run tox from the tarball.  They are in the trunk if you want.

Extract the two files into the <duplicity-root>/dist subdirectory and tox will run.

...Ken


On Wed, Sep 4, 2019 at 10:53 AM Kenneth Loafman <[hidden email]> wrote:
Scott,   

Have you tried updating your Mac?  It's several years behind current software.

duplicity uses a lot of python modules.  The top of the tox listing from earlier shows the version numbers of the ones used in testing.  Again, maybe an update?

Why bzr or git: dist/makedist makes an install tar from the current directory.  It uses bzr or git to get a list of the released filenames, builds a tarball using just those files, then tox uses the tarball to build the virtual environment under .tox/py37.  Keeps the distribution clean that way, no stray files from the developer.

That said, tox will not work for testing a tarball distribution.  You'll have to use "setup.py test" and I'll have to rethink dist/makedist.  My bad!

...Ken






On Tue, Sep 3, 2019 at 8:46 PM Scott Hannahs via Duplicity-talk <[hidden email]> wrote:
Ken,

If I look at .tox/log the log file created tonight is:

actionid: tox
msg: packaging
cmdargs: ['/sw/bin/python3.7', local('/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/.tox/dist')]

running sdist
failed to open trace file: [Errno 13] Permission denied: '/Users/sth/.bzr.log'
bzr: ERROR: Not a branch: "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/".
fatal: Not a git repository (or any of the parent directories): .git
Processing version 0.8.04
Traceback (most recent call last):
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 134, in <module>
    Main()
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 125, in Main
    tarfile = MakeTar()
  File "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/dist/makedist", line 90, in MakeTar
    assert not os.mkdir(os.path.join(tardir, u"po", lang)), lang
FileNotFoundError: [Errno 2] No such file or directory: 'duplicity-0.8.04/po/bg'
mv: rename duplicity-0.8.04.tar.gz to .tox/dist/duplicity-0.8.04.tar.gz: No such file or directory
/sw/lib/python3.7/site-packages/setuptools/dist.py:398: UserWarning: Normalizing '0.8.04' to '0.8.4'
  normalized_version,

It seems that the testing needs access to a .bzr.log file in the home directory.  This could be an issue since I am trying to build this in an anonymous account as security testing for side effects.

I have tried running the test ("tox -e py37”) in my own directory as myself and still get the error the key lines are the bzr and git errors.  Why is the testing system trying to sync using bzr or git?
running sdist
bzr: ERROR: Not a branch: "/sw/src/fink.build/duplicity-0.8.04-1/duplicity-0.8.04/".
fatal: Not a git repository (or any of the parent directories): .git
Processing version 0.8.04


-Scott


On Sep 3, 2019, at 1:52 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

When I run "tox -e py37" on my Mac I have no errors.

Your errors in test_selection are the same as on py36, very weird.

The permission errors causing test_replicate and test_restart are also odd.  Do you have the system locked down more than normal?  I test using just my own creds, not root..

The unicode errors when reporting the permission errors are something that needs to be fixed, but are a distraction at this point.

Try running under tox instead of fink just to eliminate one variable.

I've pasted my run.  Perhaps there is a version issue between our two systems?

...Thanks,
...Ken

ken@dione:~/workspace/duplicity-src8$ tox -e py37
GLOB sdist-make: /Users/ken/workspace/duplicity-src8/setup.py
py37 create: /Users/ken/workspace/duplicity-src8/.tox/py37
py37 installdeps: -rrequirements.txt
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 inst: /Users/ken/workspace/duplicity-src8/.tox/.tmp/package/1/duplicity-0.0dev.tar.gz
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py37 installed: alabaster==0.7.12,astroid==2.2.5,atomicwrites==1.3.0,attrs==19.1.0,Babel==2.7.0,certifi==2019.6.16,chardet==3.0.4,coverage==4.5.4,docutils==0.15.2,duplicity==0.0.dev0,fasteners==0.15,fastimport==0.9.2,filelock==3.0.12,future==0.17.1,idna==2.8,imagesize==1.1.0,importlib-metadata==0.20,isort==4.3.21,Jinja2==2.10.1,lazy-object-proxy==1.4.2,MarkupSafe==1.1.1,mccabe==0.6.1,mock==3.0.5,monotonic==1.5,more-itertools==7.2.0,packaging==19.1,pexpect==4.7.0,pluggy==0.12.0,ptyprocess==0.6.0,py==1.8.0,pycodestyle==2.5.0,Pygments==2.4.2,pylint==2.3.1,pyparsing==2.4.2,pytest==5.1.2,pytest-cov==2.7.1,pytest-runner==5.1,pytz==2019.2,requests==2.22.0,six==1.12.0,snowballstemmer==1.9.0,Sphinx==2.2.0,sphinxcontrib-applehelp==1.0.1,sphinxcontrib-devhelp==1.0.1,sphinxcontrib-htmlhelp==1.0.2,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.2,sphinxcontrib-serializinghtml==1.1.3,toml==0.10.0,tox==3.14.0,typed-ast==1.4.0,urllib3==1.25.3,virtualenv==16.7.5,wcwidth==0.1.7,wrapt==1.11.2,zipp==0.6.0
py37 run-test-pre: PYTHONHASHSEED='3996759775'
py37 run-test: commands[0] | python compilec.py
running build
running build_ext
building '_librsync' extension
creating build
creating build/temp.macosx-10.14-x86_64-3.7
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c _librsyncmodule.c -o build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o
creating build/lib.macosx-10.14-x86_64-3.7
clang -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk build/temp.macosx-10.14-x86_64-3.7/_librsyncmodule.o -L/usr/local/lib -lrsync -o build/lib.macosx-10.14-x86_64-3.7/_librsync.cpython-37m-darwin.so
py37 run-test: commands[1] | pytest --cov=duplicity --cov-append --cov-report=
================================================= test session starts ==================================================
platform darwin -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.12.0
cachedir: .tox/py37/.pytest_cache
rootdir: /Users/ken/workspace/duplicity-src8, inifile: tox.ini, testpaths: testing/unit, testing/functional
plugins: cov-2.7.1
collected 428 items                                                                                                    

testing/unit/test_backend.py ............                                                                        [  2%]
testing/unit/test_backend_instance.py ............................................................               [ 16%]
testing/unit/test_collections.py ........                                                                        [ 18%]
testing/unit/test_diffdir.py .......                                                                             [ 20%]
testing/unit/test_dup_temp.py ..                                                                                 [ 20%]
testing/unit/test_dup_time.py ......................                                                             [ 25%]
testing/unit/test_file_naming.py .............                                                                   [ 28%]
testing/unit/test_globmatch.py ...........................                                                       [ 35%]
testing/unit/test_gpg.py ..........                                                                              [ 37%]
testing/unit/test_gpginterface.py ......                                                                         [ 39%]
testing/unit/test_lazy.py ............................                                                           [ 45%]
testing/unit/test_manifest.py .....                                                                              [ 46%]
testing/unit/test_patchdir.py ........                                                                           [ 48%]
testing/unit/test_path.py .....                                                                                  [ 49%]
testing/unit/test_selection.py ............................................................................      [ 67%]
testing/unit/test_statistics.py .......                                                                          [ 69%]
testing/unit/test_tarfile.py .                                                                                   [ 69%]
testing/unit/test_tempdir.py ..                                                                                  [ 69%]
testing/unit/test_util.py .                                                                                      [ 70%]
testing/functional/test_badupload.py .                                                                           [ 70%]
testing/functional/test_cleanup.py ...                                                                           [ 71%]
testing/functional/test_final.py ...........................                                                     [ 77%]
testing/functional/test_log.py .                                                                                 [ 77%]
testing/functional/test_rdiffdir.py .                                                                            [ 77%]
testing/functional/test_replicate.py ...                                                                         [ 78%]
testing/functional/test_restart.py ................................                                              [ 85%]
testing/functional/test_selection.py .......................................ss...........                        [ 98%]
testing/functional/test_verify.py ........                                                                       [100%]

=================================================== warnings summary ===================================================
.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/types/oldstr.py:5: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Iterable

.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4
  /Users/ken/workspace/duplicity-src8/.tox/py37/lib/python3.7/site-packages/past/builtins/misc.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

testing/unit/test_globmatch.py:100
  /Users/ken/workspace/duplicity-src8/testing/unit/test_globmatch.py:100: DeprecationWarning: invalid escape sequence \/
    u"\/usr\/[^/]*\/bin\/")

-- Docs: https://docs.pytest.org/en/latest/warnings.html


================================ 426 passed, 2 skipped, 3 warnings in 560.97s (0:09:20) ================================
_______________________________________________________ summary ________________________________________________________
  py37: commands succeeded
  congratulations :)


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
<fixes0804.tgz>


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Ken,

I just managed to try this on a newer OS (took awhile since there seems to be no access to the ulimit command).  I need to set the max files to some fairly large number.  This does not seem to be necessary for running duplicity but for the testing.  Duplicity seems to run correctly on my system and back things up to the archive.  However, I don’t have funny unicode file names nor am I trying to restart a backup which are the failing tests.

I have updated tox (to 3.14.0).  I installed future version 0.17 and I had found that dependency earlier.  That dependency is just if I need to run the testing phase, correct?  I updated virtualenv to version 16.7.5.

There are two types of errors.  This one has 12 instances and happens on both OS versions.

b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/path.py", line 476, in <lambda>'
b'    util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))'
b"PermissionError: [Errno 1] Operation not permitted: b'testfiles/cache/3f6f90580e457c90e7202bc33cd9e363/duplicity-full-signatures.19700102T034640Z.sigtar.gz'"
b'During handling of the above exception, another exception occurred:'
b'Traceback (most recent call last):'
b'  File "../bin/duplicity", line 157, in <module>'
b'    if u"Forced assertion for testing" in util.uexc(e):'
b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/util.py", line 120, in uexc'
b'    return fsdecode(m)'
b'  File "/sw/lib/python3.7/os.py", line 821, in fsdecode'
b'    filename = fspath(filename)  # Does type-checking of `filename`.'
b'TypeError: expected str, bytes or os.PathLike object, not int'
...return_val: 1
This seems to be a permission error.  But I can’t even find the cache file that shows the error.  That may be part of the test, but then it has another exception while handling the error in decoding the path.  It seems like there is a some sort of an issue with filename being an “int” instead of str, bytes or os.PathLike object.  I am lost at this point….  :-)

The other error occurs under the older OS (10.11.6) El Capitan and not under the newer (10.14.6) Mojave.  This seems to be in the unicode handling and the arrays are of different number of bytes for the same visual glyphs.  They appear visually to be the same strings, but one lists the last items a s 60 chars, and the second string is 175 chars.  Is El Capitan using some older unicode encoding?  Python 3.7 should be consistent on both systems, or so I thought.  This happens in 5 instances in the older OS.

unit/test_selection.py:240: in ParseTest
    self.assertEqual(indicies, results_as_list)
E   AssertionError: Lists differ: [(), [91 chars] 'Παράδειγμα'), ('прыклад', 'пример', '例', 'Πα[60 chars]t',)] != [(), [91 chars] 'Παράδειγμα'), ('прыклад', 'пример', '例', 'Π[175 chars]t',)]
E   
E   First differing element 4:
E   ('прыклад', 'пример', '例', 'Παράδειγμα')
E   ('прыклад', 'пример', '例', 'Παράδειγμα')
E   
E   Second list contains 2 additional elements.
E   First extra element 8:
E   ('прыклад', 'উদাহরণ')
E   
E     [(),
E      ('прыклад',),
E      ('прыклад', 'пример'),
E      ('прыклад', 'пример', '例'),
E   -  ('прыклад', 'пример', '例', 'Παράδειγμα'),
E   ?                                 ^
E   
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα'),
E   ?                                 ^^
E   
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα', 'דוגמא.txt'),
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα', 'उदाहरण.txt'),
E   -  ('прыклад', 'пример', '例', 'Παράδειγμα', 'ઉદાહરણ.log'),
E   ?                                 ^
E   
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα', 'ઉદાહરણ.log'),
E   ?                                 ^^
E   
E      ('прыклад', 'উদাহরণ'),
E      ('օրինակ.txt',)]




On Sep 9, 2019, at 10:59 AM, Kenneth Loafman <[hidden email]> wrote:

Scott,

setup.py (uses dist/makedist) is failing because you do not have the future module installed.  There may be more:

    install_requires=[u'fasteners', u'future'],
    tests_require=[u'pytest', u'pytest-runner', u'fasteners', u'future', u'mock', u'pexpect'],

dist/makedist will make a tarball from the files in dist/relfiles, then tox will use that tarball to install in a virtualenv for testing.

The only modules I see that are way out of date is tox and virtualenv.  An update would help.

...Ken


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Scott,

I've had access to ulimit from the first.  I'm not sure what you mean.

Who owns the directory where duplicity is testing?  If the testing user can't write to that directory most of the functional tests will fail.

I'm not sure about Python's unicode environment.  How did you install Python?  Was it from binary or source?  If source, how?

...Ken


On Wed, Sep 11, 2019 at 4:39 PM Scott Hannahs <[hidden email]> wrote:
Ken,

I just managed to try this on a newer OS (took awhile since there seems to be no access to the ulimit command).  I need to set the max files to some fairly large number.  This does not seem to be necessary for running duplicity but for the testing.  Duplicity seems to run correctly on my system and back things up to the archive.  However, I don’t have funny unicode file names nor am I trying to restart a backup which are the failing tests.

I have updated tox (to 3.14.0).  I installed future version 0.17 and I had found that dependency earlier.  That dependency is just if I need to run the testing phase, correct?  I updated virtualenv to version 16.7.5.

There are two types of errors.  This one has 12 instances and happens on both OS versions.

b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/path.py", line 476, in <lambda>'
b'    util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))'
b"PermissionError: [Errno 1] Operation not permitted: b'testfiles/cache/3f6f90580e457c90e7202bc33cd9e363/duplicity-full-signatures.19700102T034640Z.sigtar.gz'"
b'During handling of the above exception, another exception occurred:'
b'Traceback (most recent call last):'
b'  File "../bin/duplicity", line 157, in <module>'
b'    if u"Forced assertion for testing" in util.uexc(e):'
b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/util.py", line 120, in uexc'
b'    return fsdecode(m)'
b'  File "/sw/lib/python3.7/os.py", line 821, in fsdecode'
b'    filename = fspath(filename)  # Does type-checking of `filename`.'
b'TypeError: expected str, bytes or os.PathLike object, not int'
...return_val: 1
This seems to be a permission error.  But I can’t even find the cache file that shows the error.  That may be part of the test, but then it has another exception while handling the error in decoding the path.  It seems like there is a some sort of an issue with filename being an “int” instead of str, bytes or os.PathLike object.  I am lost at this point….  :-)

The other error occurs under the older OS (10.11.6) El Capitan and not under the newer (10.14.6) Mojave.  This seems to be in the unicode handling and the arrays are of different number of bytes for the same visual glyphs.  They appear visually to be the same strings, but one lists the last items a s 60 chars, and the second string is 175 chars.  Is El Capitan using some older unicode encoding?  Python 3.7 should be consistent on both systems, or so I thought.  This happens in 5 instances in the older OS.

unit/test_selection.py:240: in ParseTest
    self.assertEqual(indicies, results_as_list)
E   AssertionError: Lists differ: [(), [91 chars] 'Παράδειγμα'), ('прыклад', 'пример', '例', 'Πα[60 chars]t',)] != [(), [91 chars] 'Παράδειγμα'), ('прыклад', 'пример', '例', 'Π[175 chars]t',)]
E   
E   First differing element 4:
E   ('прыклад', 'пример', '例', 'Παράδειγμα')
E   ('прыклад', 'пример', '例', 'Παράδειγμα')
E   
E   Second list contains 2 additional elements.
E   First extra element 8:
E   ('прыклад', 'উদাহরণ')
E   
E     [(),
E      ('прыклад',),
E      ('прыклад', 'пример'),
E      ('прыклад', 'пример', '例'),
E   -  ('прыклад', 'пример', '例', 'Παράδειγμα'),
E   ?                                 ^
E   
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα'),
E   ?                                 ^^
E   
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα', 'דוגמא.txt'),
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα', 'उदाहरण.txt'),
E   -  ('прыклад', 'пример', '例', 'Παράδειγμα', 'ઉદાહરણ.log'),
E   ?                                 ^
E   
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα', 'ઉદાહરણ.log'),
E   ?                                 ^^
E   
E      ('прыклад', 'উদাহরণ'),
E      ('օրինակ.txt',)]




On Sep 9, 2019, at 10:59 AM, Kenneth Loafman <[hidden email]> wrote:

Scott,

setup.py (uses dist/makedist) is failing because you do not have the future module installed.  There may be more:

    install_requires=[u'fasteners', u'future'],
    tests_require=[u'pytest', u'pytest-runner', u'fasteners', u'future', u'mock', u'pexpect'],

dist/makedist will make a tarball from the files in dist/relfiles, then tox will use that tarball to install in a virtualenv for testing.

The only modules I see that are way out of date is tox and virtualenv.  An update would help.

...Ken


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Ken,

Under Mac OS X 10.11.X you can run “ulimit -n 5120” which seems to be sufficient to do the testing phase.  It might be possible to use a smaller number but I haven’t checked.

Under Mac OS 10.14.X running "ulimit -n 5120” will give the error "-bash: ulimit: open files: cannot modify limit: Operation not permitted” if 5120 is more than the current setting.  This was a surprise to me.  So my shell script to run the "setup pytest.py” would fail immediately, since the default is 4096 and I can’t increase to 5120 under the newer Mojave system.  Once I worked around that, I got the errors I sent.

I run the build as a user called fink-bld, this user is not a privileged user.  But this is the user who unpacks the files and owns the entire folder structure where the build and testing are taking place.  This should work and makes sure the build is for a plain vanilla user and not using any of my personal preferences settings and can be built/tested by a non-administrator account.

I have asked that the build directory be kept at the end of testing and there is no folder “testifies” only the zipped version.  However I think the testing suite creates and removes that directory repeatedly.   So having it not exist at the end of testing seems normal.  The pytest goest through all the tests and just records the errors and does not stop.  Was that test supposed to be a case where the cache file does not exist?  There is no cache folder when I extract the testifiles.tar.gz archive manually so not finding a cache is normal and it should be regenerated during testing?

I used “fink” (of course) to install python 3.7, this is installed from source.  And is actually 3.7.4.  These should have the same modules installed on both OS systems.  There may be something in more recently defined unicode glyphs that is incompatible.  At some time, some glyphs (or whatever unicode calls them) have been redefined.  The A with a circle (or angstrom) is an example.  I think there were two different unicode identifiers for these.

-Scott


On Sep 12, 2019, at 12:01 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

I've had access to ulimit from the first.  I'm not sure what you mean.

Who owns the directory where duplicity is testing?  If the testing user can't write to that directory most of the functional tests will fail.

I'm not sure about Python's unicode environment.  How did you install Python?  Was it from binary or source?  If source, how?

...Ken


On Wed, Sep 11, 2019 at 4:39 PM Scott Hannahs <[hidden email]> wrote:
Ken,

I just managed to try this on a newer OS (took awhile since there seems to be no access to the ulimit command).  I need to set the max files to some fairly large number.  This does not seem to be necessary for running duplicity but for the testing.  Duplicity seems to run correctly on my system and back things up to the archive.  However, I don’t have funny unicode file names nor am I trying to restart a backup which are the failing tests.

I have updated tox (to 3.14.0).  I installed future version 0.17 and I had found that dependency earlier.  That dependency is just if I need to run the testing phase, correct?  I updated virtualenv to version 16.7.5.

There are two types of errors.  This one has 12 instances and happens on both OS versions.

b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/path.py", line 476, in <lambda>'
b'    util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))'
b"PermissionError: [Errno 1] Operation not permitted: b'testfiles/cache/3f6f90580e457c90e7202bc33cd9e363/duplicity-full-signatures.19700102T034640Z.sigtar.gz'"
b'During handling of the above exception, another exception occurred:'
b'Traceback (most recent call last):'
b'  File "../bin/duplicity", line 157, in <module>'
b'    if u"Forced assertion for testing" in util.uexc(e):'
b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/util.py", line 120, in uexc'
b'    return fsdecode(m)'
b'  File "/sw/lib/python3.7/os.py", line 821, in fsdecode'
b'    filename = fspath(filename)  # Does type-checking of `filename`.'
b'TypeError: expected str, bytes or os.PathLike object, not int'
...return_val: 1
This seems to be a permission error.  But I can’t even find the cache file that shows the error.  That may be part of the test, but then it has another exception while handling the error in decoding the path.  It seems like there is a some sort of an issue with filename being an “int” instead of str, bytes or os.PathLike object.  I am lost at this point….  :-)

The other error occurs under the older OS (10.11.6) El Capitan and not under the newer (10.14.6) Mojave.  This seems to be in the unicode handling and the arrays are of different number of bytes for the same visual glyphs.  They appear visually to be the same strings, but one lists the last items a s 60 chars, and the second string is 175 chars.  Is El Capitan using some older unicode encoding?  Python 3.7 should be consistent on both systems, or so I thought.  This happens in 5 instances in the older OS.

unit/test_selection.py:240: in ParseTest
    self.assertEqual(indicies, results_as_list)
E   AssertionError: Lists differ: [(), [91 chars] 'Παράδειγμα'), ('прыклад', 'пример', '例', 'Πα[60 chars]t',)] != [(), [91 chars] 'Παράδειγμα'), ('прыклад', 'пример', '例', 'Π[175 chars]t',)]
E   
E   First differing element 4:
E   ('прыклад', 'пример', '例', 'Παράδειγμα')
E   ('прыклад', 'пример', '例', 'Παράδειγμα')
E   
E   Second list contains 2 additional elements.
E   First extra element 8:
E   ('прыклад', 'উদাহরণ')
E   
E     [(),
E      ('прыклад',),
E      ('прыклад', 'пример'),
E      ('прыклад', 'пример', '例'),
E   -  ('прыклад', 'пример', '例', 'Παράδειγμα'),
E   ?                                 ^
E   
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα'),
E   ?                                 ^^
E   
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα', 'דוגמא.txt'),
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα', 'उदाहरण.txt'),
E   -  ('прыклад', 'пример', '例', 'Παράδειγμα', 'ઉદાહરણ.log'),
E   ?                                 ^
E   
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα', 'ઉદાહરણ.log'),
E   ?                                 ^^
E   
E      ('прыклад', 'উদাহরণ'),
E      ('օրինակ.txt',)]




On Sep 9, 2019, at 10:59 AM, Kenneth Loafman <[hidden email]> wrote:

Scott,

setup.py (uses dist/makedist) is failing because you do not have the future module installed.  There may be more:

    install_requires=[u'fasteners', u'future'],
    tests_require=[u'pytest', u'pytest-runner', u'fasteners', u'future', u'mock', u'pexpect'],

dist/makedist will make a tarball from the files in dist/relfiles, then tox will use that tarball to install in a virtualenv for testing.

The only modules I see that are way out of date is tox and virtualenv.  An update would help.

...Ken



_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Scott,

What C/C++ compiler and libraries?  I'm assuming Xcode.  The core of Python is C/C++, so the libraries could be a problem.

Honestly, unless Homrbrew is installing patches I don't know about, I'm lost as to what is different.

...Ken


On Thu, Sep 12, 2019 at 11:29 AM Scott Hannahs <[hidden email]> wrote:
Ken,

Under Mac OS X 10.11.X you can run “ulimit -n 5120” which seems to be sufficient to do the testing phase.  It might be possible to use a smaller number but I haven’t checked.

Under Mac OS 10.14.X running "ulimit -n 5120” will give the error "-bash: ulimit: open files: cannot modify limit: Operation not permitted” if 5120 is more than the current setting.  This was a surprise to me.  So my shell script to run the "setup pytest.py” would fail immediately, since the default is 4096 and I can’t increase to 5120 under the newer Mojave system.  Once I worked around that, I got the errors I sent.

I run the build as a user called fink-bld, this user is not a privileged user.  But this is the user who unpacks the files and owns the entire folder structure where the build and testing are taking place.  This should work and makes sure the build is for a plain vanilla user and not using any of my personal preferences settings and can be built/tested by a non-administrator account.

I have asked that the build directory be kept at the end of testing and there is no folder “testifies” only the zipped version.  However I think the testing suite creates and removes that directory repeatedly.   So having it not exist at the end of testing seems normal.  The pytest goest through all the tests and just records the errors and does not stop.  Was that test supposed to be a case where the cache file does not exist?  There is no cache folder when I extract the testifiles.tar.gz archive manually so not finding a cache is normal and it should be regenerated during testing?

I used “fink” (of course) to install python 3.7, this is installed from source.  And is actually 3.7.4.  These should have the same modules installed on both OS systems.  There may be something in more recently defined unicode glyphs that is incompatible.  At some time, some glyphs (or whatever unicode calls them) have been redefined.  The A with a circle (or angstrom) is an example.  I think there were two different unicode identifiers for these.

-Scott


On Sep 12, 2019, at 12:01 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

I've had access to ulimit from the first.  I'm not sure what you mean.

Who owns the directory where duplicity is testing?  If the testing user can't write to that directory most of the functional tests will fail.

I'm not sure about Python's unicode environment.  How did you install Python?  Was it from binary or source?  If source, how?

...Ken


On Wed, Sep 11, 2019 at 4:39 PM Scott Hannahs <[hidden email]> wrote:
Ken,

I just managed to try this on a newer OS (took awhile since there seems to be no access to the ulimit command).  I need to set the max files to some fairly large number.  This does not seem to be necessary for running duplicity but for the testing.  Duplicity seems to run correctly on my system and back things up to the archive.  However, I don’t have funny unicode file names nor am I trying to restart a backup which are the failing tests.

I have updated tox (to 3.14.0).  I installed future version 0.17 and I had found that dependency earlier.  That dependency is just if I need to run the testing phase, correct?  I updated virtualenv to version 16.7.5.

There are two types of errors.  This one has 12 instances and happens on both OS versions.

b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/path.py", line 476, in <lambda>'
b'    util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))'
b"PermissionError: [Errno 1] Operation not permitted: b'testfiles/cache/3f6f90580e457c90e7202bc33cd9e363/duplicity-full-signatures.19700102T034640Z.sigtar.gz'"
b'During handling of the above exception, another exception occurred:'
b'Traceback (most recent call last):'
b'  File "../bin/duplicity", line 157, in <module>'
b'    if u"Forced assertion for testing" in util.uexc(e):'
b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/util.py", line 120, in uexc'
b'    return fsdecode(m)'
b'  File "/sw/lib/python3.7/os.py", line 821, in fsdecode'
b'    filename = fspath(filename)  # Does type-checking of `filename`.'
b'TypeError: expected str, bytes or os.PathLike object, not int'
...return_val: 1
This seems to be a permission error.  But I can’t even find the cache file that shows the error.  That may be part of the test, but then it has another exception while handling the error in decoding the path.  It seems like there is a some sort of an issue with filename being an “int” instead of str, bytes or os.PathLike object.  I am lost at this point….  :-)

The other error occurs under the older OS (10.11.6) El Capitan and not under the newer (10.14.6) Mojave.  This seems to be in the unicode handling and the arrays are of different number of bytes for the same visual glyphs.  They appear visually to be the same strings, but one lists the last items a s 60 chars, and the second string is 175 chars.  Is El Capitan using some older unicode encoding?  Python 3.7 should be consistent on both systems, or so I thought.  This happens in 5 instances in the older OS.

unit/test_selection.py:240: in ParseTest
    self.assertEqual(indicies, results_as_list)
E   AssertionError: Lists differ: [(), [91 chars] 'Παράδειγμα'), ('прыклад', 'пример', '例', 'Πα[60 chars]t',)] != [(), [91 chars] 'Παράδειγμα'), ('прыклад', 'пример', '例', 'Π[175 chars]t',)]
E   
E   First differing element 4:
E   ('прыклад', 'пример', '例', 'Παράδειγμα')
E   ('прыклад', 'пример', '例', 'Παράδειγμα')
E   
E   Second list contains 2 additional elements.
E   First extra element 8:
E   ('прыклад', 'উদাহরণ')
E   
E     [(),
E      ('прыклад',),
E      ('прыклад', 'пример'),
E      ('прыклад', 'пример', '例'),
E   -  ('прыклад', 'пример', '例', 'Παράδειγμα'),
E   ?                                 ^
E   
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα'),
E   ?                                 ^^
E   
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα', 'דוגמא.txt'),
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα', 'उदाहरण.txt'),
E   -  ('прыклад', 'пример', '例', 'Παράδειγμα', 'ઉદાહરણ.log'),
E   ?                                 ^
E   
E   +  ('прыклад', 'пример', '例', 'Παράδειγμα', 'ઉદાહરણ.log'),
E   ?                                 ^^
E   
E      ('прыклад', 'উদাহরণ'),
E      ('օրինակ.txt',)]




On Sep 9, 2019, at 10:59 AM, Kenneth Loafman <[hidden email]> wrote:

Scott,

setup.py (uses dist/makedist) is failing because you do not have the future module installed.  There may be more:

    install_requires=[u'fasteners', u'future'],
    tests_require=[u'pytest', u'pytest-runner', u'fasteners', u'future', u'mock', u'pexpect'],

dist/makedist will make a tarball from the files in dist/relfiles, then tox will use that tarball to install in a virtualenv for testing.

The only modules I see that are way out of date is tox and virtualenv.  An update would help.

...Ken



_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Ken,

I am willing to leave the unicode errors as just an old OS weirdness.  The others I am a bit concerned that I am not testing correctly or it isn’t passing the test.  This happens both on the old OS and the new OS with Python 3.7.

To take the second failed test (the first failure is a unicode problem in test_selection).

self = <testing.functional.test_replicate.ReplicateTest testMethod=test_replicate>
        basepython = os.environ.get(u'TOXPYTHON', None)
        if basepython is not None:
            cmd_list.extend([basepython])
            cmd_list.extend([u"-m", u"coverage", u"run", u"--source=duplicity", u"-p"])
        cmd_list.extend([u"../bin/duplicity"])
        cmd_list.extend(options)
        cmd_list.extend([u"-v0"])
        cmd_list.extend([u"--no-print-statistics"])
        cmd_list.extend([u"--allow-source-mismatch"])
        cmd_list.extend([u"--archive-dir=testfiles/cache"])
        if current_time:
            cmd_list.extend([u"--current-time", current_time])
        cmd_list.extend(self.class_args)
        if fail:
            cmd_list.extend([u"--fail", u"".__class__(fail)])
        cmdline = u" ".join([u'"%s"' % x for x in cmd_list])
which if I read this right gives me a command line:
...command: "../bin/duplicity" "file://testfiles/replicate_out" "testfiles/restore_out" "--restore-time" "100000" "-v0" "--no-print-statistics" "--allow-source-mismatch" "--archive-dir=testfiles/cache" < /dev/null

And then the error which seems way beyond my python knowledge.  I can find the fsdecode function in os.py but why it is getting a bad argument is beyond me.  I do not know why it should think that it will find the cache directory since that is not in the gzip tar file.  But maybe it was generated in a previous test?  In reading the test language I don’t see that but that may be my limitation.

b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/path.py", line 476, in <lambda>'
b'    util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))'
b"PermissionError: [Errno 1] Operation not permitted: b'testfiles/cache/3f6f90580e457c90e7202bc33cd9e363/duplicity-full-signatures.19700102T034640Z.sigtar.gz'"
b'During handling of the above exception, another exception occurred:'
b'Traceback (most recent call last):'
b'  File "../bin/duplicity", line 157, in <module>'
b'    if u"Forced assertion for testing" in util.uexc(e):'
b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/util.py", line 120, in uexc'
b'    return fsdecode(m)'
b'  File "/sw/lib/python3.7/os.py", line 821, in fsdecode'
b'    filename = fspath(filename)  # Does type-checking of `filename`.'
b'TypeError: expected str, bytes or os.PathLike object, not int'
...return_val: 1


Including the whole output including the build and testing phase.

-Scott



On Sep 12, 2019, at 1:55 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

What C/C++ compiler and libraries?  I'm assuming Xcode.  The core of Python is C/C++, so the libraries could be a problem.

Honestly, unless Homrbrew is installing patches I don't know about, I'm lost as to what is different.

...Ken


On Thu, Sep 12, 2019 at 11:29 AM Scott Hannahs <[hidden email]> wrote:
Ken,

Under Mac OS X 10.11.X you can run “ulimit -n 5120” which seems to be sufficient to do the testing phase.  It might be possible to use a smaller number but I haven’t checked.

Under Mac OS 10.14.X running "ulimit -n 5120” will give the error "-bash: ulimit: open files: cannot modify limit: Operation not permitted” if 5120 is more than the current setting.  This was a surprise to me.  So my shell script to run the "setup pytest.py” would fail immediately, since the default is 4096 and I can’t increase to 5120 under the newer Mojave system.  Once I worked around that, I got the errors I sent.

I run the build as a user called fink-bld, this user is not a privileged user.  But this is the user who unpacks the files and owns the entire folder structure where the build and testing are taking place.  This should work and makes sure the build is for a plain vanilla user and not using any of my personal preferences settings and can be built/tested by a non-administrator account.

I have asked that the build directory be kept at the end of testing and there is no folder “testifies” only the zipped version.  However I think the testing suite creates and removes that directory repeatedly.   So having it not exist at the end of testing seems normal.  The pytest goest through all the tests and just records the errors and does not stop.  Was that test supposed to be a case where the cache file does not exist?  There is no cache folder when I extract the testifiles.tar.gz archive manually so not finding a cache is normal and it should be regenerated during testing?

I used “fink” (of course) to install python 3.7, this is installed from source.  And is actually 3.7.4.  These should have the same modules installed on both OS systems.  There may be something in more recently defined unicode glyphs that is incompatible.  At some time, some glyphs (or whatever unicode calls them) have been redefined.  The A with a circle (or angstrom) is an example.  I think there were two different unicode identifiers for these.

-Scott


On Sep 12, 2019, at 12:01 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

I've had access to ulimit from the first.  I'm not sure what you mean.

Who owns the directory where duplicity is testing?  If the testing user can't write to that directory most of the functional tests will fail.

I'm not sure about Python's unicode environment.  How did you install Python?  Was it from binary or source?  If source, how?

...Ken




_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

fink-duplicity.txt (214K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Scctt,

An answer on my last email, please.

I'm looking at the log you sent.  Still trying to figure out what's going on.

...Ken


On Thu, Sep 12, 2019 at 4:54 PM Scott Hannahs <[hidden email]> wrote:
Ken,

I am willing to leave the unicode errors as just an old OS weirdness.  The others I am a bit concerned that I am not testing correctly or it isn’t passing the test.  This happens both on the old OS and the new OS with Python 3.7.

To take the second failed test (the first failure is a unicode problem in test_selection).

self = <testing.functional.test_replicate.ReplicateTest testMethod=test_replicate>
        basepython = os.environ.get(u'TOXPYTHON', None)
        if basepython is not None:
            cmd_list.extend([basepython])
            cmd_list.extend([u"-m", u"coverage", u"run", u"--source=duplicity", u"-p"])
        cmd_list.extend([u"../bin/duplicity"])
        cmd_list.extend(options)
        cmd_list.extend([u"-v0"])
        cmd_list.extend([u"--no-print-statistics"])
        cmd_list.extend([u"--allow-source-mismatch"])
        cmd_list.extend([u"--archive-dir=testfiles/cache"])
        if current_time:
            cmd_list.extend([u"--current-time", current_time])
        cmd_list.extend(self.class_args)
        if fail:
            cmd_list.extend([u"--fail", u"".__class__(fail)])
        cmdline = u" ".join([u'"%s"' % x for x in cmd_list])
which if I read this right gives me a command line:
...command: "../bin/duplicity" "file://testfiles/replicate_out" "testfiles/restore_out" "--restore-time" "100000" "-v0" "--no-print-statistics" "--allow-source-mismatch" "--archive-dir=testfiles/cache" < /dev/null

And then the error which seems way beyond my python knowledge.  I can find the fsdecode function in os.py but why it is getting a bad argument is beyond me.  I do not know why it should think that it will find the cache directory since that is not in the gzip tar file.  But maybe it was generated in a previous test?  In reading the test language I don’t see that but that may be my limitation.

b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/path.py", line 476, in <lambda>'
b'    util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))'
b"PermissionError: [Errno 1] Operation not permitted: b'testfiles/cache/3f6f90580e457c90e7202bc33cd9e363/duplicity-full-signatures.19700102T034640Z.sigtar.gz'"
b'During handling of the above exception, another exception occurred:'
b'Traceback (most recent call last):'
b'  File "../bin/duplicity", line 157, in <module>'
b'    if u"Forced assertion for testing" in util.uexc(e):'
b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/util.py", line 120, in uexc'
b'    return fsdecode(m)'
b'  File "/sw/lib/python3.7/os.py", line 821, in fsdecode'
b'    filename = fspath(filename)  # Does type-checking of `filename`.'
b'TypeError: expected str, bytes or os.PathLike object, not int'
...return_val: 1


Including the whole output including the build and testing phase.

-Scott


On Sep 12, 2019, at 1:55 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

What C/C++ compiler and libraries?  I'm assuming Xcode.  The core of Python is C/C++, so the libraries could be a problem.

Honestly, unless Homrbrew is installing patches I don't know about, I'm lost as to what is different.

...Ken


On Thu, Sep 12, 2019 at 11:29 AM Scott Hannahs <[hidden email]> wrote:
Ken,

Under Mac OS X 10.11.X you can run “ulimit -n 5120” which seems to be sufficient to do the testing phase.  It might be possible to use a smaller number but I haven’t checked.

Under Mac OS 10.14.X running "ulimit -n 5120” will give the error "-bash: ulimit: open files: cannot modify limit: Operation not permitted” if 5120 is more than the current setting.  This was a surprise to me.  So my shell script to run the "setup pytest.py” would fail immediately, since the default is 4096 and I can’t increase to 5120 under the newer Mojave system.  Once I worked around that, I got the errors I sent.

I run the build as a user called fink-bld, this user is not a privileged user.  But this is the user who unpacks the files and owns the entire folder structure where the build and testing are taking place.  This should work and makes sure the build is for a plain vanilla user and not using any of my personal preferences settings and can be built/tested by a non-administrator account.

I have asked that the build directory be kept at the end of testing and there is no folder “testifies” only the zipped version.  However I think the testing suite creates and removes that directory repeatedly.   So having it not exist at the end of testing seems normal.  The pytest goest through all the tests and just records the errors and does not stop.  Was that test supposed to be a case where the cache file does not exist?  There is no cache folder when I extract the testifiles.tar.gz archive manually so not finding a cache is normal and it should be regenerated during testing?

I used “fink” (of course) to install python 3.7, this is installed from source.  And is actually 3.7.4.  These should have the same modules installed on both OS systems.  There may be something in more recently defined unicode glyphs that is incompatible.  At some time, some glyphs (or whatever unicode calls them) have been redefined.  The A with a circle (or angstrom) is an example.  I think there were two different unicode identifiers for these.

-Scott


On Sep 12, 2019, at 12:01 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

I've had access to ulimit from the first.  I'm not sure what you mean.

Who owns the directory where duplicity is testing?  If the testing user can't write to that directory most of the functional tests will fail.

I'm not sure about Python's unicode environment.  How did you install Python?  Was it from binary or source?  If source, how?

...Ken




_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list


On Sep 12, 2019, at 6:56 PM, Kenneth Loafman <[hidden email]> wrote:

Scctt,

An answer on my last email, please.

Ken,

Sorry, doing the development on two different OS and systems seems to be confusing things.  On the Mac OS 10.14.6 test system: (Xcode 10.3)

% gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

And the libraries would be those associated with Xcode 10.3 developer tools.  Since the missing file of fsdecode error also shows on the much older Mac OS with the much older Xcode tools it is probably not an Xcode or version.  Since it works on your system, I may be missing installing an extra tool.

On the older Mac OS 10.11.6 system: (Xcode 8.2.1)
% gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

The test in “test_replicate.py” is the first one test_replicate.  This runs the test by backing up dir1 and  dir2 then restoring each one into “restore_out” and then checking for differences.  (right?)

In looking carefully, does the cache file get generated in the build users home directory or in the testing directory structure?  This would be the problem since I am testing this under a sandboxed user I assume all files generated are in the testing directories and not in other places in the file system.  Should the testing specify and archive-dir option that is within the testing set of folders?  I don’t believe it does.  This would explain why it can not find the cache file at testfiles/cache/….

-Scott



I'm looking at the log you sent.  Still trying to figure out what's going on.

...Ken


On Thu, Sep 12, 2019 at 4:54 PM Scott Hannahs <[hidden email]> wrote:
Ken,

I am willing to leave the unicode errors as just an old OS weirdness.  The others I am a bit concerned that I am not testing correctly or it isn’t passing the test.  This happens both on the old OS and the new OS with Python 3.7.

To take the second failed test (the first failure is a unicode problem in test_selection).

self = <testing.functional.test_replicate.ReplicateTest testMethod=test_replicate>
        basepython = os.environ.get(u'TOXPYTHON', None)
        if basepython is not None:
            cmd_list.extend([basepython])
            cmd_list.extend([u"-m", u"coverage", u"run", u"--source=duplicity", u"-p"])
        cmd_list.extend([u"../bin/duplicity"])
        cmd_list.extend(options)
        cmd_list.extend([u"-v0"])
        cmd_list.extend([u"--no-print-statistics"])
        cmd_list.extend([u"--allow-source-mismatch"])
        cmd_list.extend([u"--archive-dir=testfiles/cache"])
        if current_time:
            cmd_list.extend([u"--current-time", current_time])
        cmd_list.extend(self.class_args)
        if fail:
            cmd_list.extend([u"--fail", u"".__class__(fail)])
        cmdline = u" ".join([u'"%s"' % x for x in cmd_list])
which if I read this right gives me a command line:
...command: "../bin/duplicity" "file://testfiles/replicate_out" "testfiles/restore_out" "--restore-time" "100000" "-v0" "--no-print-statistics" "--allow-source-mismatch" "--archive-dir=testfiles/cache" < /dev/null

And then the error which seems way beyond my python knowledge.  I can find the fsdecode function in os.py but why it is getting a bad argument is beyond me.  I do not know why it should think that it will find the cache directory since that is not in the gzip tar file.  But maybe it was generated in a previous test?  In reading the test language I don’t see that but that may be my limitation.

b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/path.py", line 476, in <lambda>'
b'    util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))'
b"PermissionError: [Errno 1] Operation not permitted: b'testfiles/cache/3f6f90580e457c90e7202bc33cd9e363/duplicity-full-signatures.19700102T034640Z.sigtar.gz'"
b'During handling of the above exception, another exception occurred:'
b'Traceback (most recent call last):'
b'  File "../bin/duplicity", line 157, in <module>'
b'    if u"Forced assertion for testing" in util.uexc(e):'
b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/util.py", line 120, in uexc'
b'    return fsdecode(m)'
b'  File "/sw/lib/python3.7/os.py", line 821, in fsdecode'
b'    filename = fspath(filename)  # Does type-checking of `filename`.'
b'TypeError: expected str, bytes or os.PathLike object, not int'
...return_val: 1


Including the whole output including the build and testing phase.

-Scott


On Sep 12, 2019, at 1:55 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

What C/C++ compiler and libraries?  I'm assuming Xcode.  The core of Python is C/C++, so the libraries could be a problem.

Honestly, unless Homrbrew is installing patches I don't know about, I'm lost as to what is different.

...Ken


On Thu, Sep 12, 2019 at 11:29 AM Scott Hannahs <[hidden email]> wrote:
Ken,

Under Mac OS X 10.11.X you can run “ulimit -n 5120” which seems to be sufficient to do the testing phase.  It might be possible to use a smaller number but I haven’t checked.

Under Mac OS 10.14.X running "ulimit -n 5120” will give the error "-bash: ulimit: open files: cannot modify limit: Operation not permitted” if 5120 is more than the current setting.  This was a surprise to me.  So my shell script to run the "setup pytest.py” would fail immediately, since the default is 4096 and I can’t increase to 5120 under the newer Mojave system.  Once I worked around that, I got the errors I sent.

I run the build as a user called fink-bld, this user is not a privileged user.  But this is the user who unpacks the files and owns the entire folder structure where the build and testing are taking place.  This should work and makes sure the build is for a plain vanilla user and not using any of my personal preferences settings and can be built/tested by a non-administrator account.

I have asked that the build directory be kept at the end of testing and there is no folder “testifies” only the zipped version.  However I think the testing suite creates and removes that directory repeatedly.   So having it not exist at the end of testing seems normal.  The pytest goest through all the tests and just records the errors and does not stop.  Was that test supposed to be a case where the cache file does not exist?  There is no cache folder when I extract the testifiles.tar.gz archive manually so not finding a cache is normal and it should be regenerated during testing?

I used “fink” (of course) to install python 3.7, this is installed from source.  And is actually 3.7.4.  These should have the same modules installed on both OS systems.  There may be something in more recently defined unicode glyphs that is incompatible.  At some time, some glyphs (or whatever unicode calls them) have been redefined.  The A with a circle (or angstrom) is an example.  I think there were two different unicode identifiers for these.

-Scott


On Sep 12, 2019, at 12:01 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

I've had access to ulimit from the first.  I'm not sure what you mean.

Who owns the directory where duplicity is testing?  If the testing user can't write to that directory most of the functional tests will fail.

I'm not sure about Python's unicode environment.  How did you install Python?  Was it from binary or source?  If source, how?

...Ken





_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Failure running tests.

duplicity-talk mailing list
Scott.

Yes, testing only writes to the testing/* directories and testing/cache takes the place of ~/.cache/duplicity.

...Ken


On Fri, Sep 13, 2019 at 8:42 AM Scott Hannahs <[hidden email]> wrote:


On Sep 12, 2019, at 6:56 PM, Kenneth Loafman <[hidden email]> wrote:

Scctt,

An answer on my last email, please.

Ken,

Sorry, doing the development on two different OS and systems seems to be confusing things.  On the Mac OS 10.14.6 test system: (Xcode 10.3)

% gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

And the libraries would be those associated with Xcode 10.3 developer tools.  Since the missing file of fsdecode error also shows on the much older Mac OS with the much older Xcode tools it is probably not an Xcode or version.  Since it works on your system, I may be missing installing an extra tool.

On the older Mac OS 10.11.6 system: (Xcode 8.2.1)
% gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

The test in “test_replicate.py” is the first one test_replicate.  This runs the test by backing up dir1 and  dir2 then restoring each one into “restore_out” and then checking for differences.  (right?)

In looking carefully, does the cache file get generated in the build users home directory or in the testing directory structure?  This would be the problem since I am testing this under a sandboxed user I assume all files generated are in the testing directories and not in other places in the file system.  Should the testing specify and archive-dir option that is within the testing set of folders?  I don’t believe it does.  This would explain why it can not find the cache file at testfiles/cache/….

-Scott



I'm looking at the log you sent.  Still trying to figure out what's going on.

...Ken


On Thu, Sep 12, 2019 at 4:54 PM Scott Hannahs <[hidden email]> wrote:
Ken,

I am willing to leave the unicode errors as just an old OS weirdness.  The others I am a bit concerned that I am not testing correctly or it isn’t passing the test.  This happens both on the old OS and the new OS with Python 3.7.

To take the second failed test (the first failure is a unicode problem in test_selection).

self = <testing.functional.test_replicate.ReplicateTest testMethod=test_replicate>
        basepython = os.environ.get(u'TOXPYTHON', None)
        if basepython is not None:
            cmd_list.extend([basepython])
            cmd_list.extend([u"-m", u"coverage", u"run", u"--source=duplicity", u"-p"])
        cmd_list.extend([u"../bin/duplicity"])
        cmd_list.extend(options)
        cmd_list.extend([u"-v0"])
        cmd_list.extend([u"--no-print-statistics"])
        cmd_list.extend([u"--allow-source-mismatch"])
        cmd_list.extend([u"--archive-dir=testfiles/cache"])
        if current_time:
            cmd_list.extend([u"--current-time", current_time])
        cmd_list.extend(self.class_args)
        if fail:
            cmd_list.extend([u"--fail", u"".__class__(fail)])
        cmdline = u" ".join([u'"%s"' % x for x in cmd_list])
which if I read this right gives me a command line:
...command: "../bin/duplicity" "file://testfiles/replicate_out" "testfiles/restore_out" "--restore-time" "100000" "-v0" "--no-print-statistics" "--allow-source-mismatch" "--archive-dir=testfiles/cache" < /dev/null

And then the error which seems way beyond my python knowledge.  I can find the fsdecode function in os.py but why it is getting a bad argument is beyond me.  I do not know why it should think that it will find the cache directory since that is not in the gzip tar file.  But maybe it was generated in a previous test?  In reading the test language I don’t see that but that may be my limitation.

b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/path.py", line 476, in <lambda>'
b'    util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))'
b"PermissionError: [Errno 1] Operation not permitted: b'testfiles/cache/3f6f90580e457c90e7202bc33cd9e363/duplicity-full-signatures.19700102T034640Z.sigtar.gz'"
b'During handling of the above exception, another exception occurred:'
b'Traceback (most recent call last):'
b'  File "../bin/duplicity", line 157, in <module>'
b'    if u"Forced assertion for testing" in util.uexc(e):'
b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/util.py", line 120, in uexc'
b'    return fsdecode(m)'
b'  File "/sw/lib/python3.7/os.py", line 821, in fsdecode'
b'    filename = fspath(filename)  # Does type-checking of `filename`.'
b'TypeError: expected str, bytes or os.PathLike object, not int'
...return_val: 1


Including the whole output including the build and testing phase.

-Scott


On Sep 12, 2019, at 1:55 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

What C/C++ compiler and libraries?  I'm assuming Xcode.  The core of Python is C/C++, so the libraries could be a problem.

Honestly, unless Homrbrew is installing patches I don't know about, I'm lost as to what is different.

...Ken


On Thu, Sep 12, 2019 at 11:29 AM Scott Hannahs <[hidden email]> wrote:
Ken,

Under Mac OS X 10.11.X you can run “ulimit -n 5120” which seems to be sufficient to do the testing phase.  It might be possible to use a smaller number but I haven’t checked.

Under Mac OS 10.14.X running "ulimit -n 5120” will give the error "-bash: ulimit: open files: cannot modify limit: Operation not permitted” if 5120 is more than the current setting.  This was a surprise to me.  So my shell script to run the "setup pytest.py” would fail immediately, since the default is 4096 and I can’t increase to 5120 under the newer Mojave system.  Once I worked around that, I got the errors I sent.

I run the build as a user called fink-bld, this user is not a privileged user.  But this is the user who unpacks the files and owns the entire folder structure where the build and testing are taking place.  This should work and makes sure the build is for a plain vanilla user and not using any of my personal preferences settings and can be built/tested by a non-administrator account.

I have asked that the build directory be kept at the end of testing and there is no folder “testifies” only the zipped version.  However I think the testing suite creates and removes that directory repeatedly.   So having it not exist at the end of testing seems normal.  The pytest goest through all the tests and just records the errors and does not stop.  Was that test supposed to be a case where the cache file does not exist?  There is no cache folder when I extract the testifiles.tar.gz archive manually so not finding a cache is normal and it should be regenerated during testing?

I used “fink” (of course) to install python 3.7, this is installed from source.  And is actually 3.7.4.  These should have the same modules installed on both OS systems.  There may be something in more recently defined unicode glyphs that is incompatible.  At some time, some glyphs (or whatever unicode calls them) have been redefined.  The A with a circle (or angstrom) is an example.  I think there were two different unicode identifiers for these.

-Scott


On Sep 12, 2019, at 12:01 PM, Kenneth Loafman <[hidden email]> wrote:

Scott,

I've had access to ulimit from the first.  I'm not sure what you mean.

Who owns the directory where duplicity is testing?  If the testing user can't write to that directory most of the functional tests will fail.

I'm not sure about Python's unicode environment.  How did you install Python?  Was it from binary or source?  If source, how?

...Ken





_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
12