Skip to content

[th/test-randomize-random-seed] Revert "build: do not randomize tests by default"

Thomas Haller requested to merge th/test-randomize-random-seed into master

Tests that use nmtst_rand() randomize test inputs and code paths. The purpose is to cover more test cases (on average).

These may be test cases that we didn't even think about, or where testing all combinations (every time) is not feasable.

The idea is that if you run the tests often enough, you will hit those cases eventually. Optimally, we already perform these randomized tests in a loop and would hit always all error cases. In practice, that is not possible because we don't have time to extensivley loop and we don't explicitly know all interesting cases we'd have to test. We we would have a well-known, small set of interesting cases, we wouldn't need to randomize in the first place.

So, "random" failures due to nmtst_rand() are not something to be afraid of. You hunt them down by:

  1. first run the test in a loop with randomization enabled.

  2. once you get the failure, note the NMTST_SEED_RAND and reproduce the failure reliably.

Note that we want that failures are reliably reproducable (2) for a given seed. For that, all randomization must happen for internal reasons. Meaning: the randomization in a unit tests must not depend on external factors like the run time of a test step. If a randomized test cannot reliably reproduce the failure after setting NMTST_SEED_RAND, then the test has a bug that needs fixing.

Regardless of that, note that fixing NMTST_SEED_RAND to reproduce the failure is only guaranteed to work if you run the test in the exact same configuration. That means on the same machine, during the same boot, with the same library versions, with the same compiler options, same environment variables, and same command line ("-p TEST"). If you vary any of these, the failure may not be reliably reproducable. But that's not a problem: just run the test in a loop so that you find the offending NMTST_SEED_RAND that reproduce the issue for your current configuration.

So: fixing the NMTST_SEED_RAND to 0 by default is harmful:

a) it's not guaranteed that NMTST_SEED_RAND=0 will yield the same results in different configurations anyway. What does it help when the test "reliably" fails in gitlab-ci but you still cannot reproduce it on your system by using the same seed. Instead: you have to find a seed that reproduces the issue in your environment.

b) despite a), by fixing the seed we wrongly limit the search space of inputs and code paths that are tested. It defeats the purpose of randomization. As explained, it is a given that one fixed seed is not sufficient to find all possible failures. We must find all possible failures by running often and with different seeds.

This reverts commit 9c6ff7fe.

Merge request reports