Now that the IGT tests have a mechanism for signaling broken testing
conditions we can stop the run on the first test that has noticed it,
and possibly has triggered that state.

Traditionally run would have continued with that test failing and the
side effects would trickle down into the other tests causing a lot of
skip/fails.

v2: extra explanations, small cleanup (Petri)

Signed-off-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Reviewed-by: Petri Latvala <petri.latvala@intel.com>

See also: commit 0e6457f1bfe2 ("lib: Don't dump log buffer when
dynamic subtest failure is inherited")

This is quite an explicit top-to-bottom test that we don't get an
incorrect warn result for an innocent dynamic subtest. It is tested
here in runner_test instead of testing in lib/tests that the extra
lines are not printed, because not printing the extra lines is an
implementation detail that might change later. The issue is after all
about test results parsed by igt_runner.

v2: Squash adding the new mockup test binary to this commit

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>

Oleg Vasilev authored 5 years ago and Petri Latvala committed 5 years ago

Currently, runner already collects all subtest names into job_list.
--list-all allows to output it to stdout.

--blacklist option takes a filename as an argument and adds all regexes
from that file to the exclusion list.

v2:
 - Update exclude/include regex matches for tests without
	subtests to be matched with pigtit-like name (Petri)
 - Replace relative paths with those formatted with testdatadir (Petri)
 - Minor codestyle changes

v3:
 - Print test names in lowercase (Petri)

v4:
 - Replaced custom str_to_lower with generate_piglit_name (Petri)
 - Minor codestyle changes

Cc: Petri Latvala <petri.latvala@intel.com>
Signed-off-by: Oleg Vasilev <oleg.vasilev@intel.com>
Reviewed-by: Petri Latvala <petri.latvala@intel.com>

Main use case here is CI, which already builds using meson.

Acked-by: Petri Latvala <petri.latvala@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>

TODO: Unit tests for the results.json file contents.

v2:
 - Avoid writing the nul character to mock files
 - Properly set up tmpdirs

v3:
 - Restore the resume-related changes that were lost in rebase

v4:
 - Better teardown for temporary directories
 - Build with autotools

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>