[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

[PATCH] Add mutex to parallel testsuite

From: Philip Martin <philip_at_codematters.co.uk>
Date: 2007-09-09 22:34:54 CEST

I'm a bit out of touch with Subversion development. I see that a
parallel mode has been added to the testsuite (great), but it doesn't
seem to work on my computer (boo). It usually hangs in getopt_test,
the first parallel set, or occasionally in basic_tests, the second
parallel set. When interrupted I typically see something like:

  File "threading.py", line 442, in __bootstrap
    self.run()
  File "/home/pm/sw/subversion/subversion/tests/cmdline/svntest/main.py", line 879, in run
    self.result, self.stdout_lines, self.stderr_lines =\
  File "/home/pm/sw/subversion/subversion/tests/cmdline/svntest/main.py", line 328, in spawn_process
    pid, wait_code = os.wait()
OSError: [Errno 10] No child processes

Looking at the python code there is a list that is modified in
multiple threads without locking. I don't know much about python
threading, but the following patch makes it "work for me":

The tests list needs to be protected by a mutex when running tests in
parallel.

* subversion/tests/cmdline/svntest/main.py
  (SpawnTest.__init__): Add lock parameter.
  (SpawnTest.run): Use lock when modifying list.
  (run_one_test): Add lock parameter.
  (_internal_run_tests): Create lock, use lock when acessing list.
  
Index: subversion/tests/cmdline/svntest/main.py
===================================================================
--- subversion/tests/cmdline/svntest/main.py (revision 26499)
+++ subversion/tests/cmdline/svntest/main.py (working copy)
@@ -850,10 +850,11 @@
   """Encapsulate a single test case, run it in a separate child process.
   Instead of waiting till the process is finished, add this class to a
   list of active tests for follow up in the parent process."""
- def __init__(self, index, tests = None):
+ def __init__(self, index, tests = None, tests_lock = None):
     threading.Thread.__init__(self)
     self.index = index
     self.tests = tests
+ self.tests_lock = tests_lock
     self.result = None
     self.stdout_lines = None
     self.stderr_lines = None
@@ -882,7 +883,9 @@
     if filter(lambda x: x.startswith('FAIL: ') or x.startswith('XPASS: '),
               self.stdout_lines):
       self.result = 1
+ self.tests_lock.acquire()
     self.tests.append(self)
+ self.tests_lock.release()
     sys.stdout.write('.')
 
 class TestRunner:
@@ -980,7 +983,8 @@
 # it can be displayed by the 'list' command.)
 
 # Func to run one test in the list.
-def run_one_test(n, test_list, parallel = 0, finished_tests = None):
+def run_one_test(n, test_list, parallel = 0,
+ finished_tests = None, finished_tests_lock = None):
   """Run the Nth client test in TEST_LIST, return the result.
 
   If we're running the tests in parallel spawn the test in a new process.
@@ -992,7 +996,7 @@
 
   # Run the test.
   if parallel:
- st = SpawnTest(n, finished_tests)
+ st = SpawnTest(n, finished_tests, finished_tests_lock)
     st.start()
     return 0
   else:
@@ -1009,6 +1013,7 @@
 
   exit_code = 0
   finished_tests = []
+ finished_tests_lock = threading.Lock()
   tests_started = 0
 
   if not parallel:
@@ -1018,14 +1023,23 @@
   else:
     for testnum in testnums:
       # wait till there's a free spot.
+ finished_tests_lock.acquire()
       while tests_started - len(finished_tests) > parallel:
+ finished_tests_lock.release()
         time.sleep(0.2)
- run_one_test(testnum, test_list, parallel, finished_tests)
+ finished_tests_lock.acquire()
+ run_one_test(testnum, test_list, parallel, finished_tests,
+ finished_tests_lock)
       tests_started += 1
+ finished_tests_lock.release()
 
     # wait for all tests to finish
+ finished_tests_lock.acquire()
     while len(finished_tests) < len(testnums):
+ finished_tests_lock.release()
       time.sleep(0.2)
+ finished_tests_lock.acquire()
+ finished_tests_lock.release()
 
     # Sort test results list by test nr.
     deco = [(test.index, test) for test in finished_tests]

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Sun Sep 9 22:31:36 2007

This is an archived mail posted to the Subversion Dev mailing list.