Skip to content
  • Thomas Haller's avatar
    Revert "build: do not randomize tests by default" · 7f724b8b
    Thomas Haller authored
    Tests that use nmtst_*rand*() randomize test inputs and code paths.
    The purpose is to cover more test cases (on average).
    
    These may be test cases that we didn't even think about, or where
    testing all combinations (every time) is not feasable.
    
    The idea is that if you run the tests often enough, you will hit those
    cases eventually. Optimally, we already perform these randomized tests
    in a loop and would hit always all error cases. In practice, that is not
    possible because we don't have time to extensivley loop and we don't
    explicitly know all interesting cases we'd have to test. We we would
    have a well-known, small set of interesting cases, we wouldn't need to
    randomize in the first place.
    
    So, "random" failures due to nmtst_*rand*() are not something to be
    afraid of. You hunt them down by:
    
      1) first run the test in a loop with randomization enabled.
    
      2) once you get the failure, note the NMTST_SEED_RAND and reproduce
         the failure reliably.
    
    Note that we want that failures are reliably reproducable (2) for a
    given seed. For that, all randomization must happen for internal reasons.
    Meaning: the randomization in a unit tests must not depend on external factors
    like the run time of a test step. If a randomized test cannot reliably
    reproduce the failure after setting NMTST_SEED_RAND, then the test has a
    bug that needs fixing.
    
    Regardless of that, note that fixing NMTST_SEED_RAND to reproduce the failure
    is only guaranteed to work if you run the test in the exact same configuration.
    That means on the same machine, during the same boot, with the same
    library versions, with the same compiler options, same environment variables,
    and same command line ("-p TEST"). If you vary any of these, the failure may
    not be reliably reproducable. But that's not a problem: just run the test in a
    loop so that you find the offending NMTST_SEED_RAND that reproduce the issue for
    *your* current configuration.
    
    So: fixing the NMTST_SEED_RAND to 0 by default is harmful:
    
     a) it's not guaranteed that NMTST_SEED_RAND=0 will yield the same
        results in different configurations anyway. What does it help when the
        test "reliably" fails in gitlab-ci but you still cannot reproduce
        it on your system by using the same seed. Instead: you have to
        find a seed that reproduces the issue in your environment.
    
     b) despite a), by fixing the seed we wrongly limit the search space of
        inputs and code paths that are tested. It defeats the purpose of
        randomization. As explained, it is a given that one fixed seed is not
        sufficient to find all possible failures. We must find all possible
        failures by running often and with different seeds.
    
    This reverts commit 9c6ff7fe.
    7f724b8b