Crashing tests with subtests don't show in a list of regressions

Submitted by Pavel Ondračka

Assigned to Fabian Bieler @fabe

Link to original bug (#104700)

Description

When some test with subtests starts crashing, it will not show in a piglit summary as a regression, since the actual subtests will go from pass to notrun and the failure will be only reported for the "unknown" test which will go from notrun to crash, and hence neither will end in the list of regressions. It can be still found in the changes summary, however the list of regressions is no longer reliable due to this. I'm not sure if this is a bug or a feature, however it seems somewhat unfortunate since this can lead to missed regressions.

Was probably introduced by commit 938ec48e Author: Fabian Bieler fabianbieler@fastmail.fm Date: Sat Jan 6 23:36:02 2018 +0100

framework: Handle crashing subtest.

Piglit silently ignored crashes in subtests.

It's impossible to know what subtest crashed. Theoretically it might
even be possible that a test crashes after the last call to
piglit_report_subtest_result and thus no subtest crashed. Though the
odds of that happening are probably pretty long.

If a test with subtests crashes, this commit adds an additional subtest
named "unknown" with result "crash".

The issue that subsequent subtests are not run is not touched upon by
this commit.

This is more of a work-around. A proper fix would modify the C subtest
framework to print a list of subtests it intends to run at the start of
the test. This would enable the python subtest framework to know which
subtest crashed.
But that's a lot of work (it would require modifying more than 100
tests) and this is better than nothing.

See also: https://bugs.freedesktop.org/show_bug.cgi?id=74642

Reviewed-by: Brian Paul <brianp@vmware.com>