Newer Older
Nicolai Hähnle's avatar
Nicolai Hähnle committed
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

\ Initial design decisions

Before I started working on Piglit, I asked around for OpenGL testing methods.
There were basically two kinds of tests:

1. Glean, which is fully automatic and intends to test the letter of the
   OpenGL specification (at least partially).

2. Manual tests using Mesa samples or third-party software.

The weakness of Glean is that it is not flexible, not pragmatic enough for
driver development. For example, it tests for precision requirements in
blending, where a driver just cannot reasonably improve on what the hardware
delivers. As a driver developer, one wants to consider a test successful
when it reaches the optimal results that the hardware can give, even when
these results may be non-compliant.

Manual tests are not well repeatable. They require a considerable amount of
work on the part of the developer, so most of the time they aren't done at all.
On the other hand, those manual tests have sometimes been created to test for
a particular weakness in implementations, so they may be very suitable to
detect future, similar weaknesses.

Due to these weaknesses, the test coverage of open source OpenGL drivers
is suboptimal at best. My goal for Piglit is to create a useful test system
that helps driver developers in improving driver quality.

With that in mind, my sub-goals are:

1. Combine the strengths of the two kinds of tests (Glean, manual tests)
   into a single framework.

2. Provide concise, human readable summaries of the test results, with the
   option to increase the detail of the report when desired.

3. Allow easy visualization of regressions.

4. Allow easy detection of performance regressions.

I briefly considered extending Glean, but then decided for creating an
entirely new project. The most important reasons are:

1. I do not want to pollute the very clean philosophy behind Glean.

2. The model behind Glean is that one process runs all the tests.
   During driver development, one often gets bugs that cause tests to crash.
   This means that one failed test can disrupt the entire test batch.
   I want to use a more robust model, where each test runs in its own process.
   This model does not easily fit onto Glean.

3. The amount of code duplication and forking overhead is minimal because
 a) I can use Glean as a "subroutine", so no code is actually duplicated,
    there's just a tiny amount of additional Python glue code.
 b) It's unlikely that this project makes significant changes to Glean
    that need to be merged upstream.

4. While it remains reasonable to use C++ for the actual OpenGL tests,
   it is easier to use a higher level language like Python for the framework
   (summary processing etc.)

66 67
\ Coding style
Nicolai Hähnle's avatar
Nicolai Hähnle committed

Basic formatting:
Nicolai Hähnle's avatar
Nicolai Hähnle committed

71 72 73 74 75
* Indent with 8-column tabs
* Limit lines to 78 characters or less
* Function return type and name go on successive lines
* Opening function brace goes on line by itself
* Opening statement braces go on same line as the 'for' or 'else'
76 77 78
* Use /* C style comments */, not // C++ style
* Don't write 'if (condition) statement;' on one line - put the statement on
  a separate line.  Braces around a single statement are optional.
Nicolai Hähnle's avatar
Nicolai Hähnle committed

80 81
The following indent command will generally format your code for piglit's
Nicolai Hähnle's avatar
Nicolai Hähnle committed

  indent -br -i8 -npcs -ce input.c -o output.c
Nicolai Hähnle's avatar
Nicolai Hähnle committed

85 86 87
Though, that doesn't give perfect results.  It messes up the
PIGLIT_GL_TEST_CONFIG_BEGIN/END section.  And array initializers sometimes
come out funny.
Nicolai Hähnle's avatar
Nicolai Hähnle committed

89 90 91 92 93 94 95 96 97 98 99
When in doubt see other recently added piglit tests for coding style.

Code conventions:

* Use "const" qualifiers whenever possible on array declarations, pointers
  and global variables.
* Use "static const" for initialized arrays whenever possible.
* Preprocessor macros should be UPPER_CASE
* Enumeration tokens should be UPPER_CASE
* Most other identifiers are lower_case_with_underscores
100 101 102
* Library, executable, and source file names are '<base>_<api>.'
  e.g. libpiglitutil_gles2
* Test names are '<lowercasegroupname>-<testname>.'  e.g. glsl-novertexdata
* Use int, float, bool except when GL types (GLint, GLfloat) are really needed
104 105
* Always declare GL entrypoint pointer type with APIENTRY, or use piglit
  dispatch typedef
106 107 108 109 110

Test conventions:

* The goal is to find bugs and demonstrate them as simply as possible, not
  to measure performance or demonstrate features.
111 112 113 114
* Write tests that are easily read, understood and debugged.  Long, complicated
  functions are frowned upon.
* Don't try to test too much in a single test program.  Most piglit programs
  are less than 300 lines long.
115 116 117 118 119 120 121 122 123 124
* If a test doesn't render anything, it doesn't need to set the window size.
* Avoid changing an existing testing such that it might fail where it
  previously passed.  Break it into subtests and add a new subtest, or add
  a new test which supersedes the old one.
* Things that should be seen are drawn in green (or blue as a second color)
  while red is for things that shouldn't be seen.
* Calculate drawing coordinates from piglit_width/piglit_height as needed,
  instead of hard coding.
* If a test can safely run at the same time as other tests, add it as a
  concurrent test in 'all.tests' (or wherever you add it).
125 126 127 128 129 130 131

Utility code:

Piglit has a rich set of utility functions for basic drawing, setting
up shaders, probing pixels, error checking, etc.  Try to use them before
rolling your own.
Nicolai Hähnle's avatar
Nicolai Hähnle committed

133 134 135 136 137 138
Python framework:

Piglit uses python's PEP8 standard for formatting of python code; using only
spaces with no tabs for indenting.  See
http://www.python.org/dev/peps/pep-0008/ for more information.

139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164

\ Release Philosophy

Since Piglit is a test suite, it is "production software" at all times.
Test case might be incorrect, but despite that it is not useful to speak of
"stable" and "unstable" versions of a test suite, especially one that sees
a relatively small rate of change like Piglit.

For this reason, developers of OpenGL drivers and related software, and even
testers are encouraged to follow the git repository on freedesktop.org at all
times. A web interface to this repository can be found here:


Nevertheless, for purposes of marking a specific point in time for packaging
in an environment where non-developers do tests on a wide range of hardware,
it has been pointed out that it would be useful to have official releases.

For this reason, we will occasionally bump the version number in the file
RELEASE and create an appropriate tag in the git repository.

This tag is the official way of marking a release, so the tarballs provided
automatically by the cgit frontend are official release tarballs.

165 166 167 168 169 170 171 172 173 174 175 176 177

\ Contributing Patches

If you want to contribute patches, please subscribe to the piglit
mailing list (http://lists.freedesktop.org/mailman/listinfo/piglit)
and then send them to piglit@lists.freedesktop.org using "git
send-email".  One of the core piglit developers should respond with
comments and suggested improvements.  The piglit mailing list is also
a good place for general discussion about piglit development, such as
future plans for the project, and coordinating work between

178 179 180 181 182 183
Note that Piglit patches use the terms "Reviewed-by", "Tested-by", and
"Acked-by" in the same way as they are used for Linux kernel patches
(see https://www.kernel.org/doc/Documentation/SubmittingPatches).  You
are also welcome to add a "Signed-off-by" line to your patch, but it
is not required.

184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200
For developers who are new to piglit: when submitting a patch, it is
helpful to add a note (after the "---" line in the patch file)
indicating that you are new to the project and don't have commit
access; that way once your patch has been revised to meet our
standards of correctness and coding style, we will know that we should
commit it for you.  If we forget, please remind us!  Once you have
successfully contributed a handful of patches, feel free to apply for
commit access usind the process described here:

Please be patient--most of us develop graphics drivers (such as Mesa)
as our primary job, so we have limited time to respond to your patches
on the piglit mailing list.  If your patch hasn't received a reply in
a week, send a follow-up email to make sure we haven't missed it.  If
you have questions that are better discussed in real time, many piglit
developers can also be found in the #dri-devel channel on Freenode