Revert "build: do not randomize tests by default"

Tests that use nmtst_*rand*() randomize test inputs and code paths.
The purpose is to cover more test cases (on average).

These may be test cases that we didn't even think about, or where
testing all combinations (every time) is not feasable.

The idea is that if you run the tests often enough, you will hit those
cases eventually. Optimally, we already perform these randomized tests
in a loop and would hit always all error cases. In practice, that is not
possible because we don't have time to extensivley loop and we don't
explicitly know all interesting cases we'd have to test. We we would
have a well-known, small set of interesting cases, we wouldn't need to
randomize in the first place.

So, "random" failures due to nmtst_*rand*() are not something to be
afraid of. You hunt them down by:

  1) first run the test in a loop with randomization enabled.

  2) once you get the failure, note the NMTST_SEED_RAND and reproduce
     the failure reliably.

Note that we want that failures are reliably reproducable (2) for a
given seed. For that, all randomization must happen for internal reasons.
Meaning: the randomization in a unit tests must not depend on external factors
like the run time of a test step. If a randomized test cannot reliably
reproduce the failure after setting NMTST_SEED_RAND, then the test has a
bug that needs fixing.

Regardless of that, note that fixing NMTST_SEED_RAND to reproduce the failure
is only guaranteed to work if you run the test in the exact same configuration.
That means on the same machine, during the same boot, with the same
library versions, with the same compiler options, same environment variables,
and same command line ("-p TEST"). If you vary any of these, the failure may
not be reliably reproducable. But that's not a problem: just run the test in a
loop so that you find the offending NMTST_SEED_RAND that reproduce the issue for
*your* current configuration.

So: fixing the NMTST_SEED_RAND to 0 by default is harmful:

 a) it's not guaranteed that NMTST_SEED_RAND=0 will yield the same
    results in different configurations anyway. What does it help when the
    test "reliably" fails in gitlab-ci but you still cannot reproduce
    it on your system by using the same seed. Instead: you have to
    find a seed that reproduces the issue in your environment.

 b) despite a), by fixing the seed we wrongly limit the search space of
    inputs and code paths that are tested. It defeats the purpose of
    randomization. As explained, it is a given that one fixed seed is not
    sufficient to find all possible failures. We must find all possible
    failures by running often and with different seeds.

This reverts commit 9c6ff7fe.
12 jobs for <span class="ref-name">th/test-randomize-random-seed</span> in 56 minutes and 25 seconds (queued for 1 second)
Status Job ID Name Coverage
  Test
passed #261677
allowed to fail
checkpatch

00:01:17

failed #261684
t_debian:9

00:11:06

passed #261686
t_debian:sid

00:12:59

manual #261685
allowed to fail manual
t_debian:testing
passed #261678
t_fedora:28

00:19:56

failed #261679
t_fedora:29

00:35:53

passed #261680
t_fedora:30

00:56:25

manual #261681
allowed to fail manual
t_fedora:rawhide
failed #261682
t_ubuntu:16.04

00:39:05

failed #261683
t_ubuntu:18.04

00:35:06

 
  External
failed NetworkManager test verification

failed NetworkManager test verification

06:11:17

 
Name Stage Failure
failed
t_ubuntu:16.04 Test
(./src/platform/tests/test-link-linux:21614): NetworkManager-DEBUG: <trace> [1556181940.3197] platform-linux: delayed-action: handle (do-request-all) refresh-all-routing-rules-ip4
# NetworkManager-DEBUG: <trace> [1556181940.3198] platform-linux: netlink: recvmsg: new message RTM_NEWROUTE, flags multi, seq 1285
(./src/platform/tests/test-link-linux:21614): NetworkManager-DEBUG: <trace> [1556181940.3198] platform-linux: netlink: recvmsg: new message RTM_NEWROUTE, flags multi, seq 1285
# NetworkManager-DEBUG: <trace> [1556181940.3198] platform-linux: event-notification: RTM_NEWROUTE, flags multi, seq 1285: ignore
(./src/platform/tests/test-link-linux:21614): NetworkManager-DEBUG: <trace> [1556181940.3198] platform-linux: event-notification: RTM_NEWROUTE, flags multi, seq 1285: ignore
# NetworkManager-DEBUG: <trace> [1556181940.3199] platform-linux: netlink: recvmsg: new message RTM_NEWROUTE, flags multi, seq 1285
(./src/platform/tests/test-link-linux:21614): NetworkManager-DEBUG: <trace> [1556181940.3199] platform-linux: netlink: recvmsg: new message RTM_NEWROUTE, flags multi, seq 1285
# NetworkManager-DEBUG: <trace> [1556181940.3199] platform-linux: event-notification: RTM_NEWROUTE, flags multi, seq 1285: ignore
(./src/platform/tests/test-link-linux:21614): NetworkManager-DEBUG: <trace> [1556181940.3199] platform-linux: event-notification: RTM_NEWROUT
Job's log exceeded limit of 8388608 bytes.
failed
t_fedora:29 Test
# NetworkManager-DEBUG: <debug> [1556181700.3645] platform-test[13.984418959]: Command finished: result=0
(./src/platform/tests/test-link-linux:1001): NetworkManager-DEBUG: 08:41:40.364: <debug> [1556181700.3645] platform-test[13.984418959]: Command finished: result=0
# NetworkManager-DEBUG: <trace> [1556181700.3647] platform-linux: delayed-action: schedule read-netlink
(./src/platform/tests/test-link-linux:1001): NetworkManager-DEBUG: 08:41:40.364: <trace> [1556181700.3647] platform-linux: delayed-action: schedule read-netlink
# NetworkManager-DEBUG: <trace> [1556181700.3648] platform-linux: delayed-action: handle read-netlink
(./src/platform/tests/test-link-linux:1001): NetworkManager-DEBUG: 08:41:40.364: <trace> [1556181700.3648] platform-linux: delayed-action: handle read-netlink
# NetworkManager-DEBUG: <trace> [1556181700.3649] platform-linux: netlink: recvmsg: new message RTM_NEWLINK, flags 0, seq 0
(./src/platform/tests/test-link-linux:1001): NetworkManager-DEBUG: 08:41:40.365: <trace> [1556181700.3649] platform-linux: netlink: recvmsg: new message RTM_NEWLINK, flags 0, seq 0
# NetworkManager-DEBUG: <trace> [1556181700.3651] platform-linux: event-notification: RTM_NEWLINK, flags 0, seq 0: 102: nm-test-device <UP,LOWER_UP;broadcast,multicast,
Job's log exceeded limit of 8388608 bytes.
failed
t_ubuntu:18.04 Test
(./src/platform/tests/test-link-linux:11012): NetworkManager-DEBUG: 08:40:41.365: <debug> [1556181641.3653] platform: (nm-test-parent) signal: route   6 removed: fe80::/64 via :: dev 99 metric 256 mss 0 rt-src rt-kernel
# NetworkManager-DEBUG: <trace> [1556181641.3653] platform-linux: netlink: recvmsg: new message RTM_DELROUTE, flags 0, seq 0
(./src/platform/tests/test-link-linux:11012): NetworkManager-DEBUG: 08:40:41.365: <trace> [1556181641.3653] platform-linux: netlink: recvmsg: new message RTM_DELROUTE, flags 0, seq 0
# NetworkManager-DEBUG: <trace> [1556181641.3654] platform-linux: event-notification: RTM_DELROUTE, flags 0, seq 0: table 255 ff00::/8 via :: dev 99 metric 256 mss 0 rt-src rt-boot
(./src/platform/tests/test-link-linux:11012): NetworkManager-DEBUG: 08:40:41.365: <trace> [1556181641.3654] platform-linux: event-notification: RTM_DELROUTE, flags 0, seq 0: table 255 ff00::/8 via :: dev 99 metric 256 mss 0 rt-src rt-boot
# NetworkManager-DEBUG: <trace> [1556181641.3655] platform-linux: update-cache-ip6-route: REMOVE: [ip6-route,0x557866303890,1,+alive,+visible; table 255 ff00::/8 via :: dev 99 metric 256 mss 0 rt-src rt-boot]
(./src/platform/tests/test-link-linux:11012): NetworkManager-DEBUG: 08:40:41.365: <trace> [1556181641.3655] platform-linux: update-cache-ip6-route: REMOVE: [ip6-route,0x557866303890,1,+alive,+visible; table 255 ff00::/8 via :: dev 99 metric 256 mss 0 rt-src rt-boot]
# NetworkManager-DEBUG: <trace> [1556181641.3655] platform: (nm-test-parent) emit signal ip6-route-changed removed: table 255 ff00::/8 via :: dev 99 metric 256 mss 0 rt-src rt-boot
(./src/platform/tests/test-link-linux:11012): NetworkManager-DE
Job's log exceeded limit of 8388608 bytes.
failed
t_debian:9 Test
test:ERROR:../src/platform/tests/test-route.c:1694:test_rule: code should not be reached
/builds/NetworkManager/NetworkManager/tools/run-nm-test.sh: line 235: 15564 Aborted (core dumped) "${NMTST_DBUS_RUN_SESSION[@]}" "$TEST" "$@"
-------

Full log written to /builds/NetworkManager/NetworkManager/build/meson-logs/testlog.txt
FAILED: meson-test
/usr/local/bin/meson test --no-rebuild --print-errorlogs
ninja: build stopped: subcommand failed.
ERROR: Job failed: exit code 1