Re: subversion client test suite

On Wed, Mar 07, 2001 at 12:57:34PM -0600, Lee Burgess wrote:
>
> So I spent some time Tuesday talking to Ben and Karl about the client
> test suite. Basically what is need is something that is fully
> automated rather than partially automated.

I would like to see the same test suite / mechanism applied to both the
client *and* the libraries. Using two mechanisms feels like a non-starter.
Certainly, I could see setting up a suite for one part, and migrating the
other over time. But long term? I'd think "one".

>...
> I think this could be done in Bash; but might be more trouble than
> it's worth (harder to maintain and extend?).

Please explain this one. I don't see an issue with a bash script that runs
the client in interesting ways, then compares the resulting output against a
snapshot/template of the "correct" output. What more is there?

diff, diff -r, and/or cmp can be used to compare output. sed can be used to
replace (changing) timestamps with a fixed value before comparison. etc

You obviously have some kind of function in mind that /bin/sh and some other
tools can't handle. What were you thinking of?

Note that I would also suggest "awk" for another card in your test suite
deck. Awk is probably more common than Perl/Python/Tcl. I also know there
are handy prebuilt awk.exe files floating around (we use it in Apache 2.0
httpd for some stuff on Windows installation).

>...
> The way I see it, I have two choices: Perl or Python. I am more
> fluent in Perl, but I like Python more. I would just as soon use
> Python, but I wanted to put it to the list before acting. So I am
> looking for *constructive* feedback regarding what other people think
> is the Right Tool For This Job.

Both can do the task; I'd say Python is more approachable for building and
maintaining test cases. Do I have quantitative/objective reasons for stating
that? Not really. A Perl expert is going to find heaps of Perl code quite
approachable. I'd just believe that you do need an (medium?) experienced
Perl person to do these right, whereas Python is probably not so demanding.
But I have nothing concrete to refer to or back that up other than long
experience.

In any case, I'd rather explore /bin/sh, sed, diff, and awk before pulling
out full-on scripting languages.

There is also something to be said for looking at other possible test
suites, rather than rolling our own. I have no references for this, though,
as I'm not hardcore about test suites (so I've never bothered to look).

>...
> A couple of issues to think about are portability and maintenance.
> Portability is probably paramount.

Shell tools, Perl, and Python are all quite portable. The question that came
up over the past day is "availability" :-) All are quite available for "all"
platforms, even though they may not be installed by default. I agree with
the general sentiment of "it is optional end-user functionality, so we don't
have to work overly hard to make it available for people." In that sense, a
scripting language is fine with me, but I think that may be overkill
relative to shell tools.

Note that if you're talking about testing a separate executable, it is
actually quite a bit easier / clearer to use /bin/sh rather than P*. For
example:

--- test-foo.sh
./client blah blah > output

--- test-foo.py
os.system("./client blah blah > output")

(dunno the Perl, but it won't be as clear as .sh)

Oh, I'm sure you can build up nifty utility functions and whatnot, but that
is just hiding the basic issue of inherent clarity.

> A close second, is the combination
> of clarity, maintainability and the ability to easily extend what will
> be written. Most of that is basically good programming practice and
> largely independent of language.

Agreed in general, although I'd tend to (subjectively) disagree with the
"largely" charactization. I definitely think that the .sh example above is
clearer than the .py example, and then my earlier comments about Perl vs
Python.

>...
> Do not think of this as some sort of competition between Perl or
> Python. What we want is what wins for Subverison. If you think that
> is Bash or C or something else, then say so and back it up.

Sure :-)

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/