New deqp-runner test status
The deqp-runner tool has an important change in the test status (anholt/deqp-runner!61).
Instead of returning an UnexpectedPass
it returns an UnexpectedImprovement(...)
, where in side those parentheses we have a test state. It can be any of Pass
, Fail
, Crash
, Skip
, Flake
, KnownFlake
, Warn
, Missing
, ExpectedFail
, Timeout
.
This provides us with transition information. Knowing the state result in the expectation files, it says it changed from there to the value in the parenthesis. That's because an UnexpectedPass
was often a transition from Fail
to Pass, but also from
Crashto
Fail` confusing with the name that the test didn't pass.
The object ExpectationsProcessor
is the one in charge of understanding those tests status and act properly when inserting or removing from the expectation files is required.