Mo DeJong <mdejong@cygnus.com> writes:
> > My question is: if this is indeed what you've written, is the system
> > able to run all of our C-library API tests,
>
> The tests implemented in C that I looked at would execute some
> C code and then exit the process with a 0 exit status if
> the test passed. That sort of test would be easy to add,
> but my long term goal is to replace all these subprocess
> tests with in-process tests. One should be able to write
> the tests in a scripting language, not in C, and that
> nifty swig wrapper tool is the the way to do that.
Mo, do you realize that fs-test.c alone probably has more code than
all of libsvn_fs? That's quite an ambitious long-term goal of yours.
I think I'd have to respectfully disagree with it, too. If one wants
to test a C API, then the 'purest' tests should probably be written in
C, not a scripting language. Going through swig bindings adds an
extra layer of variables, and thus blurs test-results: if your test
failed, was it because of a deep problem in libsvn_foo, or just
because of a bug in the latest swig binding?
> Well, if you are talking about just execing a subprocess and
> checking the exit code, that is about as "language-unbiased"
> as it gets [...]
Agree. This is encouraging.
> If you ask me, in-process testing is the way to go. [...]
Again, in-process testing isn't good when you want to test the svn
commandline binary. In principle, you want your test to have the
*exact* same input interface as a user, and you want to
collect/examine the *exact* output that a user would get back. This
means running 'svn' from a scripting language, and then parsing the
output from specific subcommands; it also means grepping through the
working copy. Testing svn "from the inside" just isn't as accurate --
internal tests might miss things that the user experiences. As a QA
person would say, the svn binary needs true "black box" testing.
> I guess you could always re-implement the test framework I posted in
> Python. It seems like a waste of time to me, but everyone has
> different tastes.
I'm not understanding you. I think we have two completely
distinct issues on the table here:
1. Is your system a good language-neutral replacement for our
existing `make check` rule?
2. Should all existing tests (C and Python) be rewritten in a single
scripting language, thereby using only "in-process" testing?
You've already heard me argue against question #2. What I'm trying to
figure out is whether the answer to question #1 *depends* on question
#2. Does your testing system lose features if it *doesn't* call
in-process tests written in tcl?
> At the end of the day, the ease with which someone can
> look at the tests and figure out what they are doing
> is the thing that matters.
We have two testing systems right now: a C test framework for testing
library APIs, and a python test framework for "black box" testing the
commandline client. They're deliberately similar systems, so that
developers can easily jump from one to the other.
I understand your itch -- it would be nice to have *all* tests written
in a single language. But one size doesn't fit all; the reason we
have two systems is because of differing testing requirements, and
your unifed-svig proposal only partially satisfies each scenario. I
think you're going to have a very hard time persuading this list that
the unified-language goal is worth the trade-off of decreasing the
accuracy of our existing internal and external tests.
Received on Sat Oct 21 14:36:28 2006