[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: Python3 work [was: The run up to Subversion 1.13.0]

From: Yasuhito FUTATSUKI <futatuki_at_poem.co.jp>
Date: Mon, 30 Sep 2019 03:38:15 +0900

On 2019/09/29 23:36, Johan Corveleyn wrote:
> On Sun, Sep 29, 2019 at 2:00 PM Yasuhito FUTATSUKI <futatuki_at_poem.co.jp> wrote:
>>
>> On 2019/09/28 17:17, Yasuhito FUTATSUKI wrote:
>>> On 2019/09/28 5:51, Johan Corveleyn wrote:
>>
>>>> [[[
>>>> Testing Release configuration on local repository.
>>>> -- Running Swig Python tests --
>>>> ..........................................................C:\Python37\lib\subprocess.py:858:
>>>> ResourceWarning: subprocess
>>>> 8548 is still running
>>>> ResourceWarning, source=self)
>>>> ResourceWarning: Enable tracemalloc to get the object allocation traceback
>>>> C:\Python37\lib\unittest\case.py:628: ResourceWarning: unclosed file
>>>> <_io.BufferedReader name=3>
>>>> testMethod()
>>>> ResourceWarning: Enable tracemalloc to get the object allocation traceback
>>>> .C:\Python37\lib\unittest\case.py:628: ResourceWarning: unclosed file
>>>> <_io.BufferedReader name='R:\\temp\\tmpryeb61g1'>
>>>> testMethod()
>>>> ResourceWarning: Enable tracemalloc to get the object allocation traceback
>>>> ..............................................................................................
>>>> ----------------------------------------------------------------------
>>>> Ran 153 tests in 66.182s
>>>>
>>>> OK
>>>> ]]]
>>>
>>> On FreeBSD those warning are also produced.
>>>
>>> [[[
>>> futatuki_at_retina-alpha[15] make check-swig-py
>>> if [ "LD_LIBRARY_PATH" = "DYLD_LIBRARY_PATH" ]; then for d in /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/libsvn_swig_py /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/../../../libsvn_*; do if [ -n "$DYLD_LIBRARY_PATH" ]; then LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$d/.libs"; else LD_LIBRARY_PATH="$d/.libs"; fi; done; export LD_LIBRARY_PATH; fi; cd /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python; /usr/local/bin/python3.7 /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/tests/run_all.py
>>> ........................................................../usr/local/lib/python3.7/subprocess.py:858: ResourceWarning: subprocess 81136 is still running
>>> ResourceWarning, source=self)
>>> ResourceWarning: Enable tracemalloc to get the object allocation traceback
>>> /usr/local/lib/python3.7/unittest/case.py:615: ResourceWarning: unclosed file <_io.BufferedReader name=4>
>>> testMethod()
>>> ResourceWarning: Enable tracemalloc to get the object allocation traceback
>>> ./usr/local/lib/python3.7/unittest/case.py:615: ResourceWarning: unclosed file <_io.BufferedReader name='/tmp/tmporh5038d'>
>>> testMethod()
>>> ResourceWarning: Enable tracemalloc to get the object allocation traceback
>>> ..............................................................................................
>>> ----------------------------------------------------------------------
>>> Ran 153 tests in 10.773s
>>>
>>> OK
>>> ]]]
>>>
>>> I think these warnings comes from difference of warning between Python 2
>>> and Python 3, and not affect quality of swig Python binding itself but
>>> affect quality of its test code.
>>
>> This was partly wrong. The warning caused by subprocess comes from
>> class svn.fs.FileDiff.get_pipe(), not in test code but in support class.
>>
>>>
>>> ResourceWarning itself appeared in Python 3.2
>>>
>>> https://docs.python.org/3/whatsnew/3.2.html
>>> (section "Other Language Changes")
>>>
>>> If we can allow this warning and want to hide it, we can use
>>> environment variable PYTHONWARNINGS='ignore::ResourceWarning::',
>>> as the document above says.
>>>
>>> [[[
>>> futatuki_at_retina-alpha[17] env PYTHONWARNINGS='ignore::ResourceWarning::' make check-swig-py
>>> if [ "LD_LIBRARY_PATH" = "DYLD_LIBRARY_PATH" ]; then for d in /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/libsvn_swig_py /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/../../../libsvn_*; do if [ -n "$DYLD_LIBRARY_PATH" ]; then LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$d/.libs"; else LD_LIBRARY_PATH="$d/.libs"; fi; done; export LD_LIBRARY_PATH; fi; cd /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python; /usr/local/bin/python3.7 /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/tests/run_all.py
>>> .........................................................................................................................................................
>>> ----------------------------------------------------------------------
>>> Ran 153 tests in 10.757s
>>>
>>> OK
>>> ]]]
>>>
>>> Of course, we can also resolve those warnings by fixing clean up code
>>> on tests.
>>
>> The patch attached is expected to fix these warnings. I already tested
>> on FreeBSD with Python 3.7 (treat ResourceWarning as an error).
>>
>> [[[
>> futatuki_at_retina-alpha[113] env PYTHONWARNINGS='error::ResourceWarning::' make check-swig-py
>> if [ "LD_LIBRARY_PATH" = "DYLD_LIBRARY_PATH" ]; then for d in /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/libsvn_swig_py /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/../../../libsvn_*; do if [ -n "$DYLD_LIBRARY_PATH" ]; then LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$d/.libs"; else LD_LIBRARY_PATH="$d/.libs"; fi; done; export LD_LIBRARY_PATH; fi; cd /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python; /usr/local/bin/python3.7 /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/tests/run_all.py
>> .........................................................................................................................................................
>> ----------------------------------------------------------------------
>> Ran 153 tests in 11.040s
>>
>> OK
>> ]]]
>
> On my Windows setup, there is still 1 ResourceWarning (before your
> patch there were 3, now only 1 remains).
>
> [[[
> C:\research\svn\dev\swig-py3>runtest.bat --swig=python
> 'ruby' is not recognized as an internal or external command,
> operable program or batch file.
> Testing Release configuration on local repository.
> -- Running Swig Python tests --
> ..........................................................C:\Python37\lib\subprocess.py:858:
> ResourceWarning: subprocess
> 1408 is still running
> ResourceWarning, source=self)
> ResourceWarning: Enable tracemalloc to get the object allocation traceback
> ...............................................................................................
> ----------------------------------------------------------------------
> Ran 153 tests in 63.597s
>
> OK
> ]]]

Hmm.. It seems the diffrence comes from difference of implementation
of class subprocess.Popen.

I tweaked again clean up code for class svn.fs.FileDiff. Only if
Python >= 3.3, it wait for maximum 10 sec to terminate subprocess.
'timeout' parameter for subprocess.Popen.wait() is added on Python 3.3.
and not back ported to Python 2, so if we want to use wait() on
earlier Python, it may wait forever.... (It can be possible to implement
timeout by using poll() and sleep loop, but is it needed here?)

> When I treat ResourceWarning as an error, I get a lot more output,
> especially before the test runs start. I suppose those are unrelated
> to that one RW above.

Yes, ... and ResourceWarnings are usually hided by default warning filter.

https://docs.python.org/3/library/warnings.html#default-warning-filter

Only while executing unittest.main(), warning filter is overrided by
its optional parameter `warning'. Its default is specified filter on
Python start up by environment variable PYTHONWARNINGS or -W option
if it is specified, otherwise "default" action for all warning
categories (i.e. show all warnings).

> [[[
> C:\research\svn\dev\swig-py3>set PYTHONWARNINGS=error::ResourceWarning::
>
> C:\research\svn\dev\swig-py3>runtest.bat --swig=python
> Exception ignored in: <_io.FileIO name='build.conf' mode='rb' closefd=True>
> ResourceWarning: unclosed file <_io.TextIOWrapper name='build.conf'
> mode='r' encoding='cp1252'>
> Exception ignored in: <_io.FileIO
> name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\srclib\\apr\\include\\apr_version.h'
> mode='rb' closefd=True>
> ResourceWarning: unclosed file <_io.TextIOWrapper
> name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\srclib\\apr\\include
> \\apr_version.h' mode='r' encoding='cp1252'>
> Exception ignored in: <_io.FileIO
> name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\srclib\\apr-util\\include\\apu_versi
> on.h' mode='rb' closefd=True>
> ResourceWarning: unclosed file <_io.TextIOWrapper
> name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\srclib\\apr-util\\in
> clude\\apu_version.h' mode='r' encoding='cp1252'>
> Exception ignored in: <_io.FileIO
> name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\srclib\\apr-util\\xml/expat/lib\\exp
> at.h' mode='rb' closefd=True>
> ResourceWarning: unclosed file <_io.TextIOWrapper
> name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\srclib\\apr-util\\xm
> l/expat/lib\\expat.h' mode='r' encoding='cp1252'>
> Exception ignored in: <_io.FileIO
> name='C:\\research\\svn\\dev\\deps\\zlib-1.2.11\\zlib.h' mode='rb'
> closefd=True>
> ResourceWarning: unclosed file <_io.TextIOWrapper
> name='C:\\research\\svn\\dev\\deps\\zlib-1.2.11\\zlib.h' mode='r' enco
> ding='cp1252'>
> Exception ignored in: <_io.FileIO
> name='C:\\research\\svn\\dev\\deps\\sqlite-amalgamation-3.22.0.0\\sqlite3.h'
> mode='rb'
> closefd=True>
> ResourceWarning: unclosed file <_io.TextIOWrapper
> name='C:\\research\\svn\\dev\\deps\\sqlite-amalgamation-3.22.0.0\\sqli
> te3.h' mode='r' encoding='cp1252'>
> Exception ignored in: <_io.FileIO
> name='subversion\\libsvn_subr\\lz4\\lz4internal.h' mode='rb'
> closefd=True>
> ResourceWarning: unclosed file <_io.TextIOWrapper
> name='subversion\\libsvn_subr\\lz4\\lz4internal.h' mode='r' encoding='
> cp1252'>
> Exception ignored in: <_io.FileIO
> name='subversion\\libsvn_subr\\utf8proc\\utf8proc_internal.h'
> mode='rb' closefd=True>
> ResourceWarning: unclosed file <_io.TextIOWrapper
> name='subversion\\libsvn_subr\\utf8proc\\utf8proc_internal.h' mode='r'
> encoding='cp1252'>
> Exception ignored in: <_io.FileIO
> name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\include\\ap_release.h'
> mode='rb' clo
> sefd=True>
> ResourceWarning: unclosed file <_io.TextIOWrapper
> name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\include\\ap_release.
> h' mode='r' encoding='cp1252'>
> Exception ignored in: <_io.FileIO
> name='C:\\research\\svn\\dev\\deps\\openssl-1.0.2n\\inc32/openssl/opensslv.h'
> mode='rb
> ' closefd=True>
> ResourceWarning: unclosed file <_io.TextIOWrapper
> name='C:\\research\\svn\\dev\\deps\\openssl-1.0.2n\\inc32/openssl/open
> sslv.h' mode='r' encoding='cp1252'>
> Exception ignored in: <_io.FileIO
> name='C:\\research\\svn\\dev\\deps\\serf-1.3.9\\serf.h' mode='rb'
> closefd=True>
> ResourceWarning: unclosed file <_io.TextIOWrapper
> name='C:\\research\\svn\\dev\\deps\\serf-1.3.9\\serf.h' mode='r' encod
> ing='cp1252'>
> Exception ignored in: <function Popen.__del__ at 0x01043420>
> Traceback (most recent call last):
> File "C:\Python37\lib\subprocess.py", line 858, in __del__
> ResourceWarning, source=self)
> ResourceWarning: subprocess 3716 is still running
> Exception ignored in: <function Popen.__del__ at 0x01043420>
> Traceback (most recent call last):
> File "C:\Python37\lib\subprocess.py", line 858, in __del__
> ResourceWarning, source=self)
> ResourceWarning: subprocess 1356 is still running
> 'ruby' is not recognized as an internal or external command,
> operable program or batch file.
> Testing Release configuration on local repository.
> -- Running Swig Python tests --
> ..........................................................Exception
> ignored in: <function Popen.__del__ at 0x02D4B030>
> Traceback (most recent call last):
> File "C:\Python37\lib\subprocess.py", line 858, in __del__
> ResourceWarning, source=self)
> ResourceWarning: subprocess 2972 is still running
> ...............................................................................................
> ----------------------------------------------------------------------
> Ran 153 tests in 93.884s
>
> OK
> ]]]
>

These ResourceWarning except last one are raised in test driver
win_test.py itself and in gen_win_dependencies module which is
used by win_test.py, I guess.

Cheers,

-- 
Yasuhito FUTATSUKI <futatuki_at_poem.co.jp>

Received on 2019-09-29 20:42:21 CEST

This is an archived mail posted to the Subversion Dev mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.