Re: HEADS UP: cmdline test factory

Branko Cibej wrote:
> Neels Janosch Hofmeyr wrote:
>> This is how I implemented the same test cmpilato wrote in about two minutes
>> sharp:
>>
>> [[[
>> def mine(sbox):
>> "lock a file using a comment with xml special chars"
>> svntest.factory.make(sbox, """
>> echo "This represents a binary file\\n" >> iota
>> svn ci
>> svn lock -m 'lock & load' iota
>> """)
>> ]]]
>> (Note that I had to double-escape the '\n' to '\\n' because of python
>> parsing it before factory can.)
>>
>
> Good start. Now add some syntactic sugar so you can write the tests like
> this:
>
> [[[
> @svntest.factory("lock a file using a comment with xml special chars")
> def mine(sbox):
> """
> echo "This represents a binary file\\n" >> iota
> svn ci
> svn lock -m 'lock & load' iota
> """
> ]]]

Heh, interesting idea.
Just making sure: you're not proposing that this is the final test, but that
this is parsed by factory to produce the actual test py, right?

Because... (read on two paragraphs below)

>
> I leave the implementation as an exercise for the reader. :)
>
> However this approach has some serious flaws:
>
> * it introduces another scripting language for defining test cases;
> we already have Python, so this appears to be serious overkill.
> * Testcase syntax is only verified at runtime.
> * New features for testcases must be implemented twice -- once in
> the factory parser, and once in the svntest infrastructure

Yes, but we don't *have* to update the factory. Nothing depends on the
factory to work properly. If it breaks, well, then it's tough for whoever
wants to write a test.

If someone wants a new feature in factory, they can just go and add it,
it'll probably be shorter than writing a whole test by hand, as I previously
did all the time, every time thinking "WTF".

You can use one or two factory calls in the midst of lots of normal svntest
code. So you could always write that new feature in tests lingo, using
factory for the remaining trivial stuff.

>
> This testcase could written like this:
>
> [[[
> from svntest import testcase
>
> @testcase
> def mine(sbox):
> "lock a file using a comment with xml special chars"
> iota = sbox.wc_file('iota')
> iota.append("This represents a binary file\n")
> svn.commit(iota)
> svn.lock(iota, m='lock & load', EXPECT=(".*locked by user", []))
> ]]]

We do need to check all those little bits of changes going on: construct an
expected tree for all of output, disk and status. Without those, most tests
are pretty much useless.

This particular test simply checks for successful completion, making sure a
certain line of output appears. That's simple.

But most tests need to see that all data is in fact in the file system and
in the WC metadata exactly as expected.

I think our current py test language is quite ok. I just think that it is a
real pain to write such a thing, let alone to get used to writing cmdline
tests when you know nothing about it.

So my approach is a mix of three things:

a) I don't want to probe and go back and forth, just to *find out* exactly
what svn currently does. I just want to tell it "do an update, write down
what you see, so I can use it without brainlessly iterating around". Even
with cool short names, I'd still need to write down exactly which results I
want to expect. Starting off with what's there now is *a lot* quicker.

b) I don't want tests-noobs to be wondering how they should remove a file,
or call an "svn commit". That's such trivial stuff in shell language, and in
tests language, you need to know where to find the proper function, and how
to plug it with expected trees. I think it's very nice to pick up newcomers
where they are: in shell language.

c) I explicitly *don't* want to change one bit of how the tests currently
work. That could mean to become incompatible with all those tests that are
already there. I just want to cheat and not write everything myself.

I like it just the way it is :)

But sure, if you want to take up a proper refactoring of the tests suite,
I'd be most happy. We've got two types of tree structure, lots of little
weird things happening under the hood there... I'm not too keen. ;)

~Neels

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2363981