Ben Collins-Sussman Wrote:
> Again, in-process testing isn't good when you want to test the svn
> commandline binary. In principle, you want your test to have the
> *exact* same input interface as a user, and you want to
> collect/examine the *exact* output that a user would get back.
Well, yeah. Did you take a look at the tests/client/tests.tcl
file in the tar ball? It does exactly what you describe here.
> I'm not understanding you. I think we have two completely
> distinct issues on the table here:
>
> 1. Is your system a good language-neutral replacement for our
> existing `make check` rule?
Yes, I claim that it is superior to the existing `make check`
approach because of the integrated logging features, it
will also do a nice jobs of naming core dumps. I also
want to have it kick of a subprocess in gdb using the
`svntest debug $name` subcommand (but that is not implemented yet).
> 2. Should all existing tests (C and Python) be rewritten in a single
> scripting language, thereby using only "in-process" testing?
...
> I understand your itch -- it would be nice to have *all* tests written
> in a single language. But one size doesn't fit all; the reason we
> have two systems is because of differing testing requirements, and
> your unifed-svig proposal only partially satisfies each scenario. I
> think you're going to have a very hard time persuading this list that
> the unified-language goal is worth the trade-off of decreasing the
> accuracy of our existing internal and external tests.
Well, it is not realistic to rewrite all of the C based tests
right away, and this really does not have anything to do with
the test framework itself. I guess raising this in-process issue
has just clouded the waters. Sorry about that.
The test framework I propose can handle both exec based
"black box" testing of the command line client and
exec based running of existing tests written in C.
It can also do in-process testing. There will be
cases where an in-process tests is far easier to
write than a C based tests, since you have access
to a high level scripting language and can do nice
things like regexps and so on.
Later, Greg Hudson Wrote:
> For that matter, is there anything wrong with our existing "make
> check" rule which suggests the introduction of a new language
> dependency in addition to C and Python?
The goals of the test framework are already covered by
www/testing-goals.html. In particular, I claim that
the "Unique Test Identifiers", "Selective Execution",
"No Monitoring", "Automatic Logging of Results",
"Report Results Only", and "Platform Specific Results"
test goals are better handled (if at all) in my proposed
test framework. I can't speak to the "language dependency"
issue, that has already been floating around for months.
(... Later ...)
> I may have mistaken the context of the argument, such that my last
> message looks like an argument against Python as well as tcl. I
> didn't realize that Mo's test framework was supposed to replace the
> recent Python work as well as the C test framework.
Well, it was really just meant to replace the C framework, it
just happens to overlap with the Python classes since they
seem to be solving the same problem.
> * I'd like to keep the number of languages in use in
> Subversion to a minimum. Even if only one language (C) is
> required to build and run the thing, developers should not
> have to learn more languages than necessary to hack on it.
> (Even if a language is simple, the tools to use and debug it
> are generally not.)
I agree 100%. Depending on just 1 scripting language for
the tests is bad enough, there is no reason to depend on 2.
> * I think people have made a convincing argument that we need
> some kind of high-level scripting language for black-box
> testing of programs, and that sh/awk is a poor choice
> because of portability to Windows and because sh/awk isn't
> good at poking around in XML format files.
I think that point has been made convincingly, hacking around
in bash scripts is a waste of time. A nice high level scripting
language really is the right way to do regression testing.
> and the test framework looks like it will be written
> in python or (god forbid) tcl.
Is "fear of Tcl" going to present such a problem that
folks are going to ignore the goals of the proposed
test framework and harp on the language it is
implemented in? I would hope not, it seems silly
to me but everyone is entitled to their opinions.
Folks have been using Tcl in test harnesses for
quite some time. I would assume that every engineer
here at Red Hat has some Tcl experience, since the
gcc and gdb test suites are written in Tcl.
As far as new users go, Tcl just ain't that hard
to learn. It is far less complex than Perl or Python.
Many see that as a weakness, in the case of a test
framework it is a strength. If folks are dead set
on using Python to write tests, I would hope that
the test framework I wrote could be ported over
to Python so that the same test case result
logging, comparison to previous test results,
and crash recovery features would still be available.
The features really are quite useful, I have been
using this framework in the Jacks Java compiler
testing project for about a year now, and it
really does scale well.
Mo DeJong
Red Hat Inc
Received on Sat Oct 21 14:36:28 2006