Mo, thanks for all the work on a test framework.
At the moment, we probably cannot integrate it. There's a lot of work
to do to meet M3 (heck, to finish up M2, though we're very close to
that now), and switching to a new test system would take a lot of
time. Everyone has higher priority things to work on, and given that
the existing test harnesses work & are familiar to everyone, a
switchover now would be counterproductive. It's too early to say what
will happen post-M3; there might or might not be significant changes
to the test framework, depending on whether a lot of developers feel
change is needed.
Ben's point about always testing C interfaces in C is also good, I
think (from your response to him, it wasn't clear to me whether you
agreed with him on that or not, so just ignore this if you did agree
:-) ).
In the meantime, are there language-independent ideas from your system
that could be applied to the current system(s)? You're probably in
the best position to judge, so if you can identify such improvements,
that would be very helpful.
Thanks,
-K
Mo DeJong <mdejong@cygnus.com> writes:
> Ben Collins-Sussman Wrote:
>
> > Again, in-process testing isn't good when you want to test the svn
> > commandline binary. In principle, you want your test to have the
> > *exact* same input interface as a user, and you want to
> > collect/examine the *exact* output that a user would get back.
>
> Well, yeah. Did you take a look at the tests/client/tests.tcl
> file in the tar ball? It does exactly what you describe here.
>
> > I'm not understanding you. I think we have two completely
> > distinct issues on the table here:
> >
> > 1. Is your system a good language-neutral replacement for our
> > existing `make check` rule?
>
> Yes, I claim that it is superior to the existing `make check`
> approach because of the integrated logging features, it
> will also do a nice jobs of naming core dumps. I also
> want to have it kick of a subprocess in gdb using the
> `svntest debug $name` subcommand (but that is not implemented yet).
>
> > 2. Should all existing tests (C and Python) be rewritten in a single
> > scripting language, thereby using only "in-process" testing?
>
> ...
>
> > I understand your itch -- it would be nice to have *all* tests written
> > in a single language. But one size doesn't fit all; the reason we
> > have two systems is because of differing testing requirements, and
> > your unifed-svig proposal only partially satisfies each scenario. I
> > think you're going to have a very hard time persuading this list that
> > the unified-language goal is worth the trade-off of decreasing the
> > accuracy of our existing internal and external tests.
>
> Well, it is not realistic to rewrite all of the C based tests
> right away, and this really does not have anything to do with
> the test framework itself. I guess raising this in-process issue
> has just clouded the waters. Sorry about that.
>
> The test framework I propose can handle both exec based
> "black box" testing of the command line client and
> exec based running of existing tests written in C.
> It can also do in-process testing. There will be
> cases where an in-process tests is far easier to
> write than a C based tests, since you have access
> to a high level scripting language and can do nice
> things like regexps and so on.
>
>
>
> Later, Greg Hudson Wrote:
>
> > For that matter, is there anything wrong with our existing "make
> > check" rule which suggests the introduction of a new language
> > dependency in addition to C and Python?
>
> The goals of the test framework are already covered by
> www/testing-goals.html. In particular, I claim that
> the "Unique Test Identifiers", "Selective Execution",
> "No Monitoring", "Automatic Logging of Results",
> "Report Results Only", and "Platform Specific Results"
> test goals are better handled (if at all) in my proposed
> test framework. I can't speak to the "language dependency"
> issue, that has already been floating around for months.
>
> (... Later ...)
>
> > I may have mistaken the context of the argument, such that my last
> > message looks like an argument against Python as well as tcl. I
> > didn't realize that Mo's test framework was supposed to replace the
> > recent Python work as well as the C test framework.
>
> Well, it was really just meant to replace the C framework, it
> just happens to overlap with the Python classes since they
> seem to be solving the same problem.
>
> > * I'd like to keep the number of languages in use in
> > Subversion to a minimum. Even if only one language (C) is
> > required to build and run the thing, developers should not
> > have to learn more languages than necessary to hack on it.
> > (Even if a language is simple, the tools to use and debug it
> > are generally not.)
>
> I agree 100%. Depending on just 1 scripting language for
> the tests is bad enough, there is no reason to depend on 2.
>
> > * I think people have made a convincing argument that we need
> > some kind of high-level scripting language for black-box
> > testing of programs, and that sh/awk is a poor choice
> > because of portability to Windows and because sh/awk isn't
> > good at poking around in XML format files.
>
> I think that point has been made convincingly, hacking around
> in bash scripts is a waste of time. A nice high level scripting
> language really is the right way to do regression testing.
>
> > and the test framework looks like it will be written
> > in python or (god forbid) tcl.
>
> Is "fear of Tcl" going to present such a problem that
> folks are going to ignore the goals of the proposed
> test framework and harp on the language it is
> implemented in? I would hope not, it seems silly
> to me but everyone is entitled to their opinions.
>
> Folks have been using Tcl in test harnesses for
> quite some time. I would assume that every engineer
> here at Red Hat has some Tcl experience, since the
> gcc and gdb test suites are written in Tcl.
> As far as new users go, Tcl just ain't that hard
> to learn. It is far less complex than Perl or Python.
> Many see that as a weakness, in the case of a test
> framework it is a strength. If folks are dead set
> on using Python to write tests, I would hope that
> the test framework I wrote could be ported over
> to Python so that the same test case result
> logging, comparison to previous test results,
> and crash recovery features would still be available.
> The features really are quite useful, I have been
> using this framework in the Jacks Java compiler
> testing project for about a year now, and it
> really does scale well.
>
> Mo DeJong
> Red Hat Inc
Received on Sat Oct 21 14:36:28 2006