Mark Phippard wrote:
> On 4/30/07, Lieven Govaerts <svnlgo@mobsol.be> wrote:
>> We have XFAIL and XPASS so that we can explicitly mark tests as failing
>> and even learn when they are unknowingly solved, it's not like we accept
>> tests to fail for more than a few days.
> I was under the impression that XFAIL tests are occasionally added
> when a problem is recognied and will live in the code until a fix is
> made, which could be a long time. I am less concerned with listing
> these other than to keep an accurate accounting.
That's right. Note that I said "tests to fail", not "tests to xfail".We
do use XFAIL to indicate not completely implemented features too
(example in depth-tests.py), there it makes sense to keep track of the
parts that have to be implemented for the feature to be useful,
especially when planning to release.
>> AFAIC the tests markers should be matched exactly by the actual results
>> at all times, so for statistical reasons all there is to know can be
>> gathered by listing the tests without running them, which was basically
>> what I did with a semi-automated python script.
>
> How do you do this? I was not aware there were methods other than
> running the tests. Feel free to point me at some docs. I just know
> make check.
I actually forgot that I only added this yesterday. Each test already
takes a '--list' option, which just lists the number of tests in the
file, which ones are PASSing, XFAILing or SKIPped. In r24830 I added
this --list option to win-tests.py, so I can get one file which contains
the list of all tests. It's probably trivial to add this for linux too.
The attached script takes two of such files (ex. one of the 1.4.x branch
and one of trunk) and compares them, thereby creating the report I sent
to the list. The only issue is that win-tests.py can not list all tests
at once, it's either fsfs or bdb, so I had to cut and paste those
manually. Oh, if you want to know the status of a 1.4.x based branch you
need the attached patch to add --list to win-tests.py.
hth,
Lieven
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Mon Apr 30 19:24:21 2007