[PATCH] Test suite to summarize XFAILs as well as SKIPs

The test suite prints a summary skipped tests, and a summary of failed
tests, at the end. It doesn't say anything about the XFAIL tests. I
think it is a problem that we are not being reminded of our XFAILs, as
"out of sight is out of mind".

I am attaching a patch to rectify this.

The output of a current trunk test run will then look like this:

[[[
> $ (cd obj-dir/ && make check CLEANUP=1)
> Running all tests in cache-test [1/63]...success
> Running all tests in checksum-test [2/63]...success
> Running all tests in client-test [3/63]...success
[...]
> Running all tests in key-test [61/63]...success
> Running all tests in skel-test [62/63]...success
> Running all tests in strings-reps-test [63/63]...success
> At least one test was SKIPPED. Checking /home/julianfoad/build/subversion-trunk3/tests.log
> SKIP: lt-cache-test 2: basic memcache svn_cache test
> SKIP: lt-cache-test 3: memcache svn_cache with very long keys
> SKIP: update_tests.py 32: update wc on the root of a Windows (virtual) drive
> SKIP: merge_tests.py 100: commits to subtrees added by merge wreak havoc
> SKIP: merge_authz_tests.py 1: skipped paths get overriding mergeinfo
> SKIP: utf8_tests.py 1: conversion of paths and logs to/from utf8
> SKIP: svnsync_tests.py 14: verify that unreadable content is not synced
> SKIP: svnsync_tests.py 15: verify that copies from unreadable dirs work
> SKIP: svnsync_tests.py 16: verify copies with mods from unreadable dirs
> SKIP: svnsync_tests.py 17: verify copies with mods from unreadable dirs +copy
> SKIP: authz_tests.py 1: authz issue #2486 - open root
> SKIP: authz_tests.py 2: authz issue #2486 - open directory
> SKIP: authz_tests.py 3: broken authz files cause errors
> SKIP: authz_tests.py 4: test authz for read operations
> SKIP: authz_tests.py 5: test authz for write operations
> SKIP: authz_tests.py 6: test authz for checkout
> SKIP: authz_tests.py 7: test authz for log and tracing path changes
> SKIP: authz_tests.py 8: test authz for checkout and update
> SKIP: authz_tests.py 9: test authz for export with unreadable subfolder
> SKIP: authz_tests.py 10: test authz for aliases
> SKIP: authz_tests.py 11: test the authz validation rules
> SKIP: authz_tests.py 12: test authz for locking
> SKIP: authz_tests.py 13: authz issue #2712
> SKIP: authz_tests.py 14: switched to directory, no read access on parents
> At least one test XFAILED. Checking /home/julianfoad/build/subversion-trunk3/tests.log
> XFAIL: lt-fs-test 18: merging commit
> XFAIL: lt-locks-test 9: able to reserve a name (lock non-existent path)
> XFAIL: lt-locks-test 10: directory locks (kinda)
> XFAIL: basic_tests.py 38: remotely remove directories from two repositories
> XFAIL: switch_tests.py 10: switch a file to a dir and back to the file
> XFAIL: log_tests.py 21: test log -c on range of changes
> XFAIL: log_tests.py 22: test log -c on comma-separated list of changes
> XFAIL: diff_tests.py 28: diff a renamed directory
> XFAIL: merge_tests.py 34: replace both dir and one of its children
> XFAIL: merge_tests.py 55: avoid repeated merges for cyclic merging
> XFAIL: merge_tests.py 72: merge target with non inheritable mergeinfo
> XFAIL: merge_tests.py 77: subtrees added after start of merge range are ok
> XFAIL: merge_tests.py 79: merge --reintegrate with renamed file on branch
> XFAIL: merge_tests.py 80: merge --reintegrate on a never-updated branch
> XFAIL: merge_tests.py 85: merge --reintegrate should fail on stale source
> XFAIL: merge_tests.py 90: merge added subtree
> XFAIL: revert_tests.py 4: revert a moved file
> XFAIL: mergeinfo_tests.py 4: 'mergeinfo' with uninteresting source selection
> XFAIL: special_tests.py 10: diff a symlink to a directory
> XFAIL: depth_tests.py 26: gradually fold wc from depth=infinity to empty
> XFAIL: depth_tests.py 27: gradually fold wc, focusing on depth=immediates
> XFAIL: depth_tests.py 28: expand target while contracting subtree
> XFAIL: depth_tests.py 30: unversioned & modified items left untouched
]]]

Any FAIL results will go after the XFAILs.

Let me know if there's anything wrong with that.

XFAIL and SKIP tests are both in the same category in the sense of being
fairly static sets of tests that we want to examine occasionally but not
on a daily basis. Instead of printing both of them, one could argue for
printing neither by default, because we are normally going to ignore
them anyway. However, if we were to do that, I would want some other way
of being reminded of our XFAILs more often than we are today.

I'm not sure if anybody has scripts depending on the current output
format, that might need to be updated, such as build-bots.

- Julian

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe_at_subversion.tigris.org
For additional commands, e-mail: dev-help_at_subversion.tigris.org