[TYPO3-caretaker] New Feature Retry-Test

[Martin Ficzel]

Martin Kuster schrieb:
> Hi again,
> 
> I just thinked about this retry-feature and how I could use that in my 
> tests.
> 
> One question came up: Isn't that actually already an escalation strategy?
> Because e.g. how does a test after multiple execution report the result? 
> Is a test that successes after 3 runs as successfully as one, that 
> successes after 1 execution? I think that should be configurable..
> 

Hello Martin,

currently it is just a very simple aproach to make flickering tests behave more stable and 
to prevent error message flooding.

Currently there is no visual representation of the number of tests wich was needed to run 
ok so the setting only makes sense for tests where a singular failure means nearly nothing 
and can be ignored.

For more complex rules we're thinking about implementing Aggregation Strategies as 
Services. So maybe a single test will fail but the Aggregator will only turn red if the 
test failed 5 times in a row. This has to be detailed in future. I added ticket 
http://forge.typo3.org/issues/show/5197 for this.

Regards Martin

> 
> Best regards
> Martin
> 
> 
> 
> Martin Ficzel wrote:
>> I added a maxRetryNumber configuration to the test. Now you can define 
>> that a failing test should imediatly be executed again n-times. The 
>> default value is zero. For me this made some flickering tests 
>> (especially ping and http) behave much more stable.
>>
>> Question: I implemented and comitted this as a generic feature of the 
>> test-record and the testrunner. It could be implemented as a feature 
>> of the ping and http test-service too. What do you prefer.
>>
>> Regards Martin