GitLab

This adds a policy which advises when the user should reboot the system
to avoid noisy test results due to system becoming unstable, for
instance, and therefore continues testing successfully.

To do this, a new module is proposed. A class Monitoring is used for
managing the monitoring rules. Two types of rules, MonitoringFile and
MonitoringLinuxDmesg, derived from the abstract class MonitoringBase,
have been implemented. The first allow to track a pattern on standard
files or locked files. The second, derived from dmesg.LinuxDmesg, will
track a pattern on the dmesg.

The monitoring rules must be defined in piglit.conf at the section
monitored-errors. If one of the regex is found, Piglit will raise a
PiglitAbort exception, stop the test execution -terminating test thread
pool- and exit with code 3.

Then test execution resume, after rebooting the system or not, is done
like usually with command line parameter "resume".

To call it, use command line parameter: --abort-on-monitored-error

This option implies --no-concurrent

This include also a set of unit tests for this module.
Reviewed-by: Dylan Baker <dylanx.c.baker@intel.com>

This adds support for running Khronos' deqp-based conformance suite for
OpenGL with piglit.

This commits renames PIGLIT_CTS_BIN / PIGLIT_CTS_EXTRA_ARGS envvars to
PIGLIT_CTS_GLES_BIN / PIGLIT_CTS_GLES_EXTRA_ARGS respectively.

v2 (Dylan Baker):
  - Use cts_gles and cts_gl
  - Rename PIGLIT_CTS_foo to PIGLIT_CTS_GLES_foo
Reviewed-by: Dylan Baker <dylan@pnwbakers.com>

This is mostly a copy-and-paste of the original GLES3 dEQP runner with a
few added tidbits that makes running the Vulkan CTS easier.  The two
force-skip cases were very useful in the early days when lots of tests were
still in development and may no longer be needed.  However, I have a
feeling that we will yet want them.

v2 (Dylan):
 - Fix PEP8 issues
 - add and update docstrings
 - update extra_args for global deqp changes
 - update piglit.conf.example
 - remove unused functions
 - use __future__
 - use "bin" instead of "exe" (gles31 is weird that it uses "exe" not
   "bin", all other deqp modules use "bin")
Reviewed-by: Dylan Baker <dylanx.c.baker@intel.com>
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>

Sometimes there are extra arguments to deqp that a developer wants for
all of the test suites, this adds that option by putting a generic
[deqp] section and add an extra_args. This new extra args gets joined
with any suite specific extra_args
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

This adds support for running Khronos' deqp-based conformance suite with
piglit. This gives access to all of the powerful features of pigilt,
per-process tests, the junit and json backends, the summary tools, and
the familiar interface, with minimal fuss.

This is a very small change, since it is deqp-based, and piglit has a
framework for handling deqp based suites already.
Reviewed-by: Mark Janes <mark.a.janes@intel.com>
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>

This requires that there is an xz binary installed, and accessible,
obviously.

v4: - add this patch
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>

This adds bzip2 compression support to piglit. This produces a file that
is ~5% the size of the original uncompressed json file.

v4: - add more tests
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>

This adds support to compress results with gzip compression. This
reduces the size of json results significantly (from 21M to 1.6M when
running the quick profile (which is about 7% of the uncompressed size).

v4: - add additional tests
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>

This creates a framework for compressing results as part of the
FileBackend class. This allows for the simple addition of compression
formats (gz is easily possible in python2, xz in python3), but does not
actually implement any compression. zip and bz2 compression are also
possible, although they'll require a little more code.

This patch implements a framework, and tests for that framework.

The goal is that other compression methods can easily be added to piglit
simply by setting a few values in framework/backends/compression, and
then everything will just work.

This will allow junit results that are compressed to be read, but
doesn't add support to junit for compressing results, since junit is
mainly intended for consumption by jenkins, which doesn't understand
compression.

v2: - replace tests decorator set_compression with use of
      utils.set_piglit_conf. This is better because it uses a generic
      tools, and because set_compression is somewhat misleading of a
      name.
v3: - backport changes from my attempt to port this to python3. There
      are significant differences between python2 and python3 compression
      handling because of the bytes/str/unicode differences between the
      two versions. There are also code cleanups from that work here
    - Make a better effort to protect the unit tests from the
      environment around them. The goal is that the tests control the
      compression options 100%, with no chance of piglit.conf or
      environment variables affecting the tests.
    - reduce code duplication in unit tests
    - add additional unit tests as made possible by refactoring changes
v4: - add additional unit tests
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>

This adds support for running deqp's gles2 tests. This is verified
working.
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>

This doesn't currently work with mesa, since mesa doesn't expose a
GLES3.1 context
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>

Google have already added a subset of dEQP into Android CTS test,
and we believe this part of dEQP have higher priority than the rest
of dEQP test cases.
The case list is stored at some xml files. Such as:
com.drawelements.deqp.gles3.xml.
It's git repo lives in the Android tree at
[https://android.googlesource.com/platform/external/ \
deqp/+/master/android/cts/com.drawelements.deqp.gles3.xml]

This patch is based on Chad's patch(Add test profile for external
dEQP-GLES3 tests), and add an option to run the subset of dEQP-GLES3.
The only differnce for the running method is: you need to set the
environment variable PIGLIT_DEQP_MUSTPASS or the piglit.conf
option deqp-gles3.mustpasslist,then it will run the subset of dEQP
follow the test case list. If not set, it will still run the whole
dEQP.

Tested on Intel HSW. There are 45866 test cases for the whole dEQP
and 37354 test cases for the subset of dEQP using Android CTS test
case list.

V2:
1.Update the link of get test case list file.
2.Using xml.etree.cElementTree module instead of re module.
3.Fix some Piglit style issue

V3:
1.Update the output issue.
2.Fix some Piglit style issue.

V4:
1.Fix some Piglit style issue.

V5:
1.Fix some Piglit style issue.

v6 (Dylan):
- Remove trailing whitespace
Signed-off-by: Wang Shuo <shuo.wang@intel.com>
Reviewed-by: Dylan Baker <dylanx.c.baker@intel.com>

Google opensourced the drawElements Quality Product Testsuite (dEQP) as
part of the Android Lollipop release. It's git repo lives in the Android
tree at [https://android.googlesource.com/platform/external/deqp/].

This patch adds a new module, deqp_gles3.py, that runs the dEQP-GLES3
tests.  It queries the 'deqp-gles3' executable at runtime for the list
of testcases.  The module attempts queries the 'deqp-gles3' executable
only if the environment variable PIGLIT_DEQP_GLES3_EXE or the
piglit.conf option deqp-gles3.exe is set.

Why do we need to use Pigit as the testrunner for dEQP? (After all, dEQP has
its own testrunner). Because dEQP runs all tests in a single process. If test
17530 of 55409 crashes, then the dEQP testrunner crashes and the remaining
tests never run. Piglit doesn't suffer from that problem, because it runs each
test as a separate process.

Tested on Intel Ivybridge by running the command
    piglit run -t dEQP-GLES3/info/vendor \
      tests/deqp_gles3.py tests/quick.py results
with and without PIGLIT_DEQP_GLES3_EXE set and with and without
piglit.conf:deqp-gles.exe set. Also tested with various combinations of
valid and invalid values for PIGLIT_DEQP_GLES_EXTRA_ARGS and
piglit.conf:deqp-gles.extra_args.

Reviewed-by: Dylan Baker <dylanx.c.baker@intel.com> (v2)
Reviewed-by: Michael W Mason <michael.w.mason@intel.com> (v4)
Signed-off-by: Chad Versace <chad.versace@linux.intel.com>

Piglit test results are difficult to analyze with simpler JUnit
visualizers like those provided by Jenkins.  There are hundreds of
failures, but the engineer is typically interested only in NEW
failures.  Jenkins JUnit rules typically expect 100% pass rate, or
some static threshold of failures.

Specifying test names in the [expected-failures] or [expected-crashes]
sections of the config file enables JUnit reporting of just the test
failures which represent new regressions.

Test names are expected in the dotted JUnit format (eg:
"piglit.spec.ARB_fragment_program.fp-formats") and are case
insensitive.
Reviewed-by: Dylan Baker <dylanx.c.baker@intel.com>

This replaces the individual parameter in add_opencv_tests()

This patch adds support to the piglit.conf to add a default backend. If
the config is empty and it is not set then 'json' will be returned (the
standard piglit json backend everyone knows and hates).
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>

This patch adds support to piglit.conf to set a default platform.
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>

v2:
  - Code cleanups

v3:
  - More cleanups
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>

Instead of symlinking oglconform into the bin/ dir, edit piglit.conf and
set the path key under the oglconform to the root of the oglconform dir.

v2: - change config key 'bindir' to 'path' (chadv)
Reviewed-by: Chad Versace <chad.versace@linux.intel.com>
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>

Instead of symlinking igt into the bin/ directory, edit piglit.conf and
set the path key in the igt section to the root of intel-gpu-tools

v2: - change config key 'bindir' to 'path' (chadv)
Reviewed-by: Chad Versace <chad.versace@linux.intel.com>
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>

Instead of symlinking a directory into bin/ for xts, edit the path entry
under the xts section to point at the root xts directory.

v2: - replace config file key 'bindir' with 'path' (chadv)
Reviewed-by: Chad Versace <chad.versace@linux.intel.com>
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>

This enables piglit to run and interpret the results from OpenCV's
gtest based opencv_test_ocl program.

You can enable the OpenCV tests by adding the path to the
opencv_test_ocl program to your piglit.conf file.

You can also optionally specify the path to OpenCV's work
directory if you buid OpenCV in a non-standard way.  For example:

[opencv]
opencv_test_ocl_bindir=/home/user/opencv/build/bin
opencv_workdir=/home/user/opencv/samples/c/

v2:
  - Python code cleanups

v3:
  - More cleanups
  - Move opencv.py into framework
  - Use ConfigParser
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>