On Mon, 26 Mar 2001, Greg Stein wrote:
> Did we decide on Perl or Python? :-)
Well, before folks go off deciding on a solution, could
we agree on the problem? I wrote up and attached an HTML
page that details the functional requirements I
see for a SVN test suite. I have tried to avoid
language wars here, this document simply outlines
the problems that need to be solved without specifying
what tool would be use to solve them.
Can folks take a look at these requirements and suggest
any revisions or clarifications you think might be needed?
It is a first draft so don't be shy about pointing
out bits that need to be clarified.
thanks
Mo DeJong
Red Hat Inc
<html>
<title>SVN Test</title>
<body bgcolor="white">
<h1>Design goals for the SVN test suite</h1>
<ul>
<li>
<A HREF="#WHY">Why Test?</A>
</li>
<li>
<A HREF="#AUDIENCE">Audience</A>
</li>
<li>
<A HREF="#REQUIREMENTS">Requirements</A>
</li>
<li>
<A HREF="#EASEOFUSE">Ease of Use</A>
</li>
<li>
<A HREF="#LOCATION">Location</A>
</li>
<li>
<A HREF="#EXTERNAL">External dependencies</A>
</li>
</ul>
<A NAME="WHY"><H3>Why Test?</H3></A>
Regression testing is an essential
element of high quality software.
Unfortunately, some developers
have not had first hand exposure
to a high quality testing framework.
Lack of familiarity with the positive
effects of testing can be blamed
for statements like:
<br>
<blockquote>
"I don't need to test my code,
I know it works."
</blockquote>
It is safe to say that the
idea that developers do not
introduce bugs
has been disproved.
</p>
<A NAME="AUDIENCE"><H3>Audience</H3></A>
The test suite will be used by
both developers and end users.
<p>
<b>Developers</b> need a test suite to help with:
</p>
<p>
<b><i>Fixing Bugs:</i></b><br>
Each time a bug is fixed, a test case should be
added to the test suite. Creating a test case
that reproduces a bug is a seemingly obvious
requirement. If a bug cannot be reproduced,
there is no way to be sure a given change
will actually fix the problem. Once a
test case has been created, it can be used
to validate the correctness of a given patch.
Adding a new test case for each bug also
ensures that the same bug will not be
introduced again in the future.
</p>
<p>
<b><i>Impact Analysis:</i></b><br>
A developer fixing a bug or adding
a new feature needs to know if
a given change breaks other parts
of the code. It may seem obvious,
but keeping a developer from
introducing new bugs is one
of the primary benefits of
a using a regression test
system.
</p>
<p>
<b><i>Regression Analysis:</i></b><br>
When a test regression occurs,
a developer will need to manually
determine what has caused the failure.
The test system is not able
to determine why a test case
failed. The test system should
simply report exactly which test
results changed and when the
last results were generated.
</p>
<b>Users</b> need a test suite to help with:
<p>
<b><i>Building:</i></b><br>
Building software can be a scary process.
Users that have never built software
may be unwilling to try. Others may
have tried to build a piece of software
in the past, only to be thwarted by
a difficult build process. Even if
the build completed without an error,
how can a user be confident that the
generated executable actually works?
The only workable solution to this
problem is to provide an easily
accessible set of tests that the
user can run after building.
</p>
<p>
<b><i>Porting:</i></b><br>
Often, users become porters when
the need to run on a previously
unsupported system arises. This
porting process typically require
some minor tweaking of include files.
It is absolutely critical that
testing be available when porting
since the primary developers
may not have any way to test
changes submitted by someone
doing a port.
</p>
<p>
<b><i>Testing:</i></b><br>
Different installations
of the exact same OS can
contain subtle differences
that cause software to
operate incorrectly.
Only testing on different
systems will expose problems
of this nature. A test suite
can help identify these sorts
of problems before a program
is actually put to use.
</p>
<A NAME="REQUIREMENTS"><H3>Requirements</H3></A>
Functional requirements of
an acceptable test suite include:
<p>
<b><i>Exact Results:</i></b><br>
A test case must have one expected
result. If the result of running the
tests does not exactly match the
expected result, the test must fail.
</p>
<p>
<b><i>Reproducible Results:</b></i><br>
Test results should be reproducible.
If a test result matches the expected
result, it should do so every time
the test is run. External
factors like time stamps must
not effect the results of a test.
</p>
<p>
<b><i>Self-Contained Tests:</b></i><br>
Each test should be self-contained.
Results for one test should not
depend on side effects of previous
tests. This is obviously a good
practice, since one is able to
understand everything a test is
doing without having to look
at other tests. The test system
should also support random access
so that a single test or set of
tests can be run. If a test is not
self-contained, it cannot be run
in isolation.
</p>
<p>
<b><i>Selective Execution:</i></b><br>
It may not be possible to run
a given set of tests on certain
systems. The suite must provide
a means of selectively running
tests cases based on the
environment. The test system
must also provide a way to
selectively run a given
test case or set of test
cases on a per invocation
basis. It would be incredibly
tedious to run the entire
suite to see the results
for a single test.
</p>
<p>
<b><i>No Monitoring:</i></b><br>
The tests must run from start to
end without operator intervention.
Test results must be generated
automatically. It is critical
that an operator not need to
manually compare test results
to figure out which test failed
and which ones passed.
</p>
<p>
<b><i>Automatic Recovery:</i></b><br>
The test system must be able
to recover from crashes and
unexpected delays. For
example, a child process might
go into a infinite loop and
would need to be killed. The
test shell itself might
also crash or go into
an infinite loop. In these
cases, the test run must
automatically recover and
continue running starting
with the tests directly
after the one that crashed.
If this was not supported,
then a test halfway through
could crash causing the
system to fail to report
results for the second half
of the tests.
The process must
be completely automated,
no operator intervention
should be required.
</p>
<p>
<b><i>Report Results Only:</i></b><br>
When a regression is found, a
developer will need to manually
determine the reason for the
regression.
The system should tell the
developer exactly what
tests have failed and
when the last set of
results were generated,
but that is all. Any additional
functionality is outside the
scope of the test system.
</p>
<p>
<b><i>Platform Specific Results:</i></b><br>
Each supported platform should
have an associated set of
test results. The alternative
would be to only maintain
results for a reference
platform. The downside
to the reference platform
approach is that is does
not keep track of the
case where a specific
set of test results
differs from the results
on the reference platform.
With a platform specific
set of test results, these
failures would have already
been recorded in a previous
result log and would therefore
not be flagged as regressions.
</p>
<p>
<b><i>Test Types:</i></b><br>
The test suite should support two
types of tests. The first makes
use of an external program
like the svn client.
These kinds of tests will need
to exec the second program and
check the output and exit status
of the child process. Note that
it will not be possible to run
this sort of test on Mac OS.
The second type of test will
load subversion shared libraries
and invoke methods directly.
This provides the ability to
do extensive testing of the
various subversion APIs without
using the svn client. This also
has the nice benefit that it
will work on Mac OS as well.
</p>
<A NAME="EASEOFUSE"><H3>Ease of Use</H3></A>
<p>
Developers will tend to avoid using
a test suite if it is not easy to
add new tests and maintain old ones.
If developers are uninterested in
using the test suite, it will
quickly fall into disrepair
and become a burden instead of
an aide.
</p>
<p>
Users will simply avoid running
the test suite if it is not
extremely simple to use. A
user should be able to build
the software and then run:
<blockquote>
<code>
% make check
</code>
</blockquote>
This should run the test suite
and provide a very high level
set of results that include
how many tests results have
changed since the last run.
</p>
<p>
While this high level report
is useful to developers, they
will often need to examine
results in more detail.
The system should provide a
means to manually examine
results and compare output
from a previous run.
</p>
<A NAME="LOCATION"><H3>Location</H3></A>
<p>
The test suite should be packaged
along with the source code instead
of being made available as a separate
download. This significantly
simplifies the process of running
tests since since they are
already incorporated into
the build tree.
</p>
<p>
The test suite must support
building and running inside
and outside of the source
directory. For example,
a developer might want to
run tests on both Solaris
and Linux. The developer
should be able to run the
tests concurrently in two
different build directories
without having the tests
interfere with each other.
</p>
<A NAME="EXTERNAL"><H3>External program dependencies</H3></A>
<p>
As much as possible, the test suite should avoid
depending on external programs or libraries.
Of course, there is a nasty bootstrap problem
with a test suite implemented in a
scripting language. A wide variety
of systems provide no support for modern
scripting languages. We will avoid
this issue for now and assume that
the scripting language of choice is
supported by the system.
</p>
<p>For example, the test suite should not depend
on CVS to generate test results. Many users
will not have access to CVS on the system
they want to test subversion on.</p>
<hr>
</body>
</html>
Received on Sat Oct 21 14:36:26 2006