README.md

Piglit

1. About
2. Setup
3. How to run tests
4. Available test sets
5. How to write tests
6. Integration

1. About

Piglit is a collection of automated tests for OpenGL and OpenCL implementations.

The goal of Piglit is to help improve the quality of open source OpenGL and OpenCL drivers by providing developers with a simple means to perform regression tests.

The original tests have been taken from

• Glean ( http://glean.sf.net/ ) and
• Mesa ( http://www.mesa3d.org/ )

2. Setup

First of all, you need to make sure that the following are installed:

• Python >=3.6
• Python Mako module
• numpy (http://www.numpy.org)
• cmake (http://www.cmake.org)
• GL, glu and glut libraries and development packages (i.e. headers)
• X11 libraries and development packages (i.e. headers)
• waffle (http://www.waffle-gl.org)

Optionally, you can install the following:

• lxml. An accelerated python xml library using libxml2 (http://lxml.de/)
• simplejson. A fast C based implementation of the python json library. (https://simplejson.readthedocs.org/en/latest/)
• jsonstreams. A JSON stream writer for python. (https://jsonstreams.readthedocs.io/en/stable/)
• VkRunner. A shader script testing tool for Vulkan. (https://github.com/igalia/vkrunner)

For testing the python framework using py.test unittests/framework

• py.test. A python test framework, used for running the python framework test suite.
• tox. A tool for testing python packages against multiple configurations of python (https://tox.readthedocs.org/en/latest/index.html)
• mock. A python module for mocking other python modules. Required only for unittests (https://github.com/testing-cabal/mock)
• psutil. A portable process library for python
• jsonschema. A JSON validator library for python
• pytest-mock. A mock plugin for pytest
• pytest-pythonpath. A plugin for pytest to do automagic with sys.path
• pytest-raises. A plugin for pytest that allows decorating tests that expect failure
• pytest-warnings. A plugin for pytest that handles python warnings
• pytest-timeout. A plugin for pytest to timeout tests.

Now configure the build system:

$ ccmake .

This will start cmake's configuration tool, just follow the onscreen instructions. The default settings should be fine, but I recommend you:

• Press 'c' once (this will also check for dependencies) and then

• Set CMAKE_BUILD_TYPE to Debug Now you can press 'c' again and then 'g' to generate the build system. Now build everything:

  $ make

2.1 Cross Compiling

On Linux, if cross-compiling a 32-bit build on a 64-bit host, first make sure you didn't have CMakeCache.txt file left from 64-bit build (it would retain old flags), then you must invoke cmake with options -DCMAKE_SYSTEM_PROCESSOR=x86 -DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32.

2.2 Ubuntu

Install development packages.

$ sudo apt-get install cmake g++ mesa-common-dev libgl1-mesa-dev python3-numpy python3-mako freeglut3-dev x11proto-gl-dev libxrender-dev libwaffle-dev libudev-dev

Configure and build.

$ cmake .
$ make

2.3 Mac OS X

Install CMake. http://cmake.org/cmake/resources/software.html Download and install 'Mac OSX Universal' platform.

Install Xcode. http://developer.apple.com/xcode

Configure and build.

$ cmake .
$ make

2.4 Cygwin

Install development packages.

• cmake
• gcc4
• make
• opengl
• libGL-devel
• python
• python-numpy
• libglut-devel

Configure and build.

$ cmake .
$ make

2.5 Windows

Install Python 3. http://www.python.org/download

Install CMake. http://cmake.org/cmake/resources/software.html Download and install 'Windows' platform.

Download and install Ninja https://github.com/ninja-build/ninja/releases

Install MinGW-w64 https://mingw-w64.org/

Download OpenGL Core API and Extension Header Files. http://www.opengl.org/registry/#headers Pass -DGLEXT_INCLUDE_DIR=/path/to/headers

Install python mako.

pip install mako

Install NumPy.

pip install numpy

2.5.1 GLUT

Download freeglut for Mingw. http://www.transmissionzero.co.uk/software/freeglut-devel/

cmake -H. -Bbuild -G "Ninja" -DGLEXT_INCLUDE_DIR=\path\to\glext -DGLUT_INCLUDE_DIR=\path\to\freeglut\include -DGLUT_glut_LIBRARY=\path\to\freeglut\lib\x64\libfreeglut.a -DGLEXT_INCLUDE_DIR=\path\to\glext
ninja -C build

2.5.2 Waffle

Download and build waffle for MinGW. http://www.waffle-gl.org/

Open the Command Prompt. CD to piglit directory.

cmake -H. -Bbuild -G "Ninja" -DGLEXT_INCLUDE_DIR=\path\to\glext -DPIGLIT_USE_WAFFLE=TRUE -DWAFFLE_INCLUDE_DIRS=\path\to\waffle\include\waffle WAFFLE_LDFLAGS=\path\to\waffle\lib\libwaffle-1.a

3. How to run tests

Make sure that everything is set up correctly:

$ ./piglit run sanity results/sanity

You may include '.py' on the profile, or you may exclude it (sanity vs sanity.py), both are equally valid.

You may also preface test profiles with tests/ (or any other path you like), which may be useful for shell tab completion.

You may provide multiple profiles to be run at the same time:

$ ./piglit run quick_cl gpu deqp_gles3 results/gl-cl-combined

Use

$ ./piglit run

or

$ ./piglit run -h

To learn more about the command's syntax.

Have a look into the tests/ directory to see what test profiles are available:

$ ls tests/*.py

See also section 4.

To create some nice formatted test summaries, run

$ ./piglit summary html summary/sanity results/sanity

Hint: You can combine multiple test results into a single summary. During development, you can use this to watch for regressions:

$ ./piglit summary html summary/compare results/baseline results/current

You can combine as many testruns as you want this way (in theory; the HTML layout becomes awkward when the number of testruns increases)

Have a look at the results with a browser:

$ xdg-open summary/sanity/index.html

The summary shows the 'status' of a test:

• pass: This test has completed successfully.
• warn: The test completed successfully, but something unexpected happened. Look at the details for more information.
• fail: The test failed.
• crash: The test binary exited with a non-zero exit code.
• skip: The test was skipped.
• timeout: The test ran longer than its allotted time and was forcibly killed.

There are also dmesg-* statuses. These have the same meaning as above, but are triggered by dmesg related messages.

3.1 Environment Variables

There are a number of environment variables that control the way piglit behaves.

• PIGLIT_COMPRESSION

  Overrides the compression method used. The same values that piglit.conf allows for core:compression.

• PIGLIT_PLATFORM

  Overrides the platform run on. These allow the same values as piglit run -p. This values is honored by the tests themselves, and can be used when running a single test.

• PIGLIT_FORCE_GLSLPARSER_DESKTOP

  Force glslparser tests to be run with the desktop (non-gles) version of glslparsertest. This can be used to test ESX_compatability extensions for OpenGL

• PIGLIT_NO_FAST_SKIP

  Piglit has a mechanism run in the python layer for skipping tests with unmet OpenGL or window system dependencies without starting a new process (which is expensive). Sometimes this system doesn't work or is undesirable, setting this environment variable to True will disable this system.

• PIGLIT_NO_TIMEOUT

  When this variable is true in python then any timeouts given by tests will be ignored, and they will run until completion or they are killed.

• PIGLIT_VKRUNNER_BINARY

  Can be used to override the path to the vkrunner executable for running Vulkan shader tests. Alternatively the config option vkrunner:bin can be used instead. If neither are set then vkrunner will be searched for in the search path.