Carlos Garcia Campos authored Dec 21, 2017 and Carlos Garcia Campos committed Dec 21, 2017

Note that running jobs are not cancelled when max failures is reached,
so we usually end up getting more failures than the maximum.

The Queue join implementation uses a non-timed wait that blocks the main
thread, making it impossible to interrupt it with CTRL+C or sending
SIGINT signal. Using any timeout value for wait would fix the problem,
but Queue doesn't allow to pass a timeout to the join method. The
Queue implementation is actually quite simple, so we can just add our
own implementation with only the things we really need and use a timeout
value when calling wait() in join().

Similar to how skipped file works, you can pass a passwords file from
the command line or add a Passwords file to your docs directory. This
file should be a python file containing a "passwords" dictionary where
the key is the test document and the value is the password required to
open that document.

Carlos Garcia Campos authored Nov 07, 2014 and Carlos Garcia Campos committed Nov 07, 2014

We are currently updating the refs only when we have checksums to
compare, but if a test used to crash there's no md5 file in refs.

Carlos Garcia Campos authored Oct 26, 2014 and Carlos Garcia Campos committed Oct 26, 2014

We are always spawning all the threads even if the documents to test is
less than the worker threads. Also optimize the case of running only one
test to not spwn any thread.

Carlos Garcia Campos authored Oct 26, 2014 and Carlos Garcia Campos committed Oct 26, 2014

Now it's possible to pass more than one argument to run-tests command and
optionally the docs directory. When more than one test is passed and the
docs directory is not provided, the common base path of all passed tests
is used as docs directory. The tests passed can be documents or
directories, using absolute paths or paths relative to the docs
directory.
This also allows us to update the refs for a group of tests.

Show a summary of tests failed per backend with the percentages and use a
new line for every test in the result instead of using a comma separated
line.

Handle them differently as new docs, and show them in the summary
suggesting to use create-refs command to include them in the test
suite.

Instead of using the number of document tested. We don't really need to
set a number to a particular document, we only want to know the progress
of the whole process. This ensures that the progress is shown in the
correct order. It also simplifies the code a bit.
Also improved the output formatting to make it easier to read.

To make sure the result corresponds to the test now that we are using
multiple threads.

To make sure they are always shown.

Show permanent information only about failed tests, without the details
about the failing pages. Previous verbose output is available passing
--verbose command line output.

It allows to update the references of failing tests.

A new command line option --skip has been added to give a file with the
list of test to skip. When --skip is not used, it look for a file named
Skipped in the tests dir. Lines starting with '#' are considered
comments and are ignored.