check-local-clients-tests-test-client is flaky, fails on armel, mips64el and slower buildds
Version: 1.14.0
After the update to 1.14.0, I see test-suite failures on armel and mips64el: https://buildd.debian.org/status/package.php?p=network-manager&suite=experimental The relevant parts of the log are:
============================================================================
make[5]: Leaving directory '/<<PKGBUILDDIR>>'
dbus[22102]: Unable to set up transient service directory: XDG_RUNTIME_DIR "/run/user/2952" not available: No such file or directory
test_001 (__main__.TestNmcli) ... skipped 'Skipped asserting for localized tests test_001-002,test_001-004,test_001-008,test_001-010,test_001-012,test_001-014,test_001-016,test_001-022,test_001-024,test_001-026,test_001-028,test_001-034,test_001-036,test_001-038,test_001-040. Set NM_TEST_CLIENT_CHECK_L10N=1 to force fail.'
test_002 (__main__.TestNmcli) ... skipped 'Skipped asserting for localized tests test_002-002,test_002-004,test_002-006,test_002-008,test_002-010,test_002-012,test_002-014,test_002-016,test_002-018,test_002-020,test_002-024. Set NM_TEST_CLIENT_CHECK_L10N=1 to force fail.'
test_003 (__main__.TestNmcli) ... skipped 'Skipped asserting for localized tests test_003-008,test_003-015,test_003-017,test_003-023,test_003-025,test_003-027,test_003-029,test_003-031,test_003-038,test_003-040,test_003-046,test_003-048,test_003-050,test_003-052,test_003-054,test_003-058,test_003-062,test_003-064,test_003-066,test_003-068,test_003-072,test_003-074,test_003-076,test_003-078,test_003-080,test_003-082,test_003-084,test_003-086,test_003-088,test_003-090,test_003-092,test_003-094,test_003-096,test_003-118,test_003-122,test_003-124,test_003-126,test_003-128,test_003-132,test_003-134,test_003-136,test_003-138,test_003-140,test_003-142,test_003-144,test_003-146,test_003-148,test_003-150,test_003-152,test_003-154,test_003-156,test_003-178,test_003-182,test_003-184,test_003-186,test_003-188,test_003-192,test_003-194,test_003-196,test_003-198,test_003-200,test_003-202,test_003-204,test_003-206,test_003-208,test_003-210,test_003-212,test_003-214,test_003-216,test_003-238,test_003-242,test_003-244,test_003-246,test_003-248,test_003-252,test_003-254,test_003-256,test_003-258,test_003-260,test_003-262,test_003-264,test_003-266,test_003-268,test_003-270,test_003-272,test_003-274,test_003-276,test_003-282,test_003-292,test_003-298,test_003-302,test_003-304,test_003-306,test_003-308,test_003-312,test_003-314,test_003-316,test_003-318,test_003-320,test_003-322,test_003-324,test_003-326,test_003-328,test_003-330,test_003-332,test_003-334,test_003-336,test_003-342,test_003-352,test_003-358,test_003-362,test_003-364,test_003-366,test_003-368,test_003-372,test_003-374,test_003-376,test_003-378,test_003-380,test_003-382,test_003-384,test_003-386,test_003-388,test_003-390,test_003-392,test_003-394,test_003-396,test_003-402,test_003-412. Set NM_TEST_CLIENT_CHECK_L10N=1 to force fail.'
test_004 (__main__.TestNmcli) ...
The file '/<<PKGBUILDDIR>>/clients/tests/test-client.check-on-disk/test_004.expected' does not have the expected content:
ACTUAL OUTPUT:
[[size: 3665
location: clients/tests/test-client.py:1058:test_004()/465
cmd: $NMCLI --mode tabular --pretty --color yes -f ALL device wifi list
lang: C
returncode: 0
stdout: 3484 bytes
>>>
===========================
Wi-Fi scan list (wlan1)
===========================
NAME SSID SSID-HEX BSSID MODE CHAN FREQ RATE SIGNAL BARS SECURITY WPA-FLAGS RSN-FLAGS DEVICE ACTIVE IN-USE DBUS-PATH
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[35mAP[1][0m [35mwlan1-ap-4[0m [35m776C616E312D61702D34[0m [35m94:2B:E8:F6:D2:86[0m [35mInfra[0m [35m1[0m [35m2412 MHz[0m [35m54 Mbit/s[0m [35m48[0m [35m** [0m [35mWPA1 WPA2[0m [35mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [35mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [35mwlan1[0m [35mno[0m [35m [0m [35m/org/freedesktop/NetworkManager/AccessPoint/4[0m
===========================
Wi-Fi scan list (wlan1)
===========================
NAME SSID SSID-HEX BSSID MODE CHAN FREQ RATE SIGNAL BARS SECURITY WPA-FLAGS RSN-FLAGS DEVICE ACTIVE IN-USE DBUS-PATH
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
===========================
Wi-Fi scan list (wlan0)
===========================
NAME SSID SSID-HEX BSSID MODE CHAN FREQ RATE SIGNAL BARS SECURITY WPA-FLAGS RSN-FLAGS DEVICE ACTIVE IN-USE DBUS-PATH
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[32mAP[1][0m [32mwlan0-ap-2[0m [32m776C616E302D61702D32[0m [32mC0:E2:BE:E8:EF:B6[0m [32mInfra[0m [32m1[0m [32m2412 MHz[0m [32m54 Mbit/s[0m [32m92[0m [32m****[0m [32mWPA1 WPA2[0m [32mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [32mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [32mwlan0[0m [32mno[0m [32m [0m [32m/org/freedesktop/NetworkManager/AccessPoint/2[0m
[32mAP[2][0m [32mwlan0-ap-1[0m [32m776C616E302D61702D31[0m [32m61:95:77:AC:1E:4C[0m [32mInfra[0m [32m1[0m [32m2412 MHz[0m [32m54 Mbit/s[0m [32m81[0m [32m****[0m [32mWPA1 WPA2[0m [32mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [32mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [32mwlan0[0m [32mno[0m [32m [0m [32m/org/freedesktop/NetworkManager/AccessPoint/1[0m
[35mAP[3][0m [35mwlan0-ap-3[0m [35m776C616E302D61702D33[0m [35m9B:F6:B7:EC:97:76[0m [35mInfra[0m [35m1[0m [35m2412 MHz[0m [35m54 Mbit/s[0m [35m55[0m [35m** [0m [35mWPA1 WPA2[0m [35mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [35mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [35mwlan0[0m [35mno[0m [35m [0m [35m/org/freedesktop/NetworkManager/AccessPoint/3[0m
<<<
]]
EXPECT OUTPUT:
[[size: 3665
location: clients/tests/test-client.py:1058:test_004()/465
cmd: $NMCLI --mode tabular --pretty --color yes -f ALL device wifi list
lang: C
returncode: 0
stdout: 3484 bytes
>>>
===========================
Wi-FFAIL
======================================================================
FAIL: test_004 (__main__.TestNmcli)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./clients/tests/test-client.py", line 782, in f
self._nm_test_post()
File "./clients/tests/test-client.py", line 751, in _nm_test_post
self.fail("Unexpected output of command, expected %s. Rerun test with NM_TEST_REGENERATE=1 to regenerate files" % (filename))
AssertionError: Unexpected output of command, expected /<<PKGBUILDDIR>>/clients/tests/test-client.check-on-disk/test_004.expected. Rerun test with NM_TEST_REGENERATE=1 to regenerate files
----------------------------------------------------------------------
Ran 4 tests in 93.114s
FAILED (failures=1, skipped=3)
i scan list (wlan0)
===========================
NAME SSID SSID-HEX BSSID MODE CHAN FREQ RATE SIGNAL BARS SECURITY WPA-FLAGS RSN-FLAGS DEVICE ACTIVE IN-USE DBUS-PATH
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[32mAP[1][0m [32mwlan0-ap-2[0m [32m776C616E302D61702D32[0m [32mC0:E2:BE:E8:EF:B6[0m [32mInfra[0m [32m1[0m [32m2412 MHz[0m [32m54 Mbit/s[0m [32m92[0m [32m****[0m [32mWPA1 WPA2[0m [32mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [32mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [32mwlan0[0m [32mno[0m [32m [0m [32m/org/freedesktop/NetworkManager/AccessPoint/2[0m
[32mAP[2][0m [32mwlan0-ap-1[0m [32m776C616E302D61702D31[0m [32m61:95:77:AC:1E:4C[0m [32mInfra[0m [32m1[0m [32m2412 MHz[0m [32m54 Mbit/s[0m [32m81[0m [32m****[0m [32mWPA1 WPA2[0m [32mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [32mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [32mwlan0[0m [32mno[0m [32m [0m [32m/org/freedesktop/NetworkManager/AccessPoint/1[0m
[35mAP[3][0m [35mwlan0-ap-3[0m [35m776C616E302D61702D33[0m [35m9B:F6:B7:EC:97:76[0m [35mInfra[0m [35m1[0m [35m2412 MHz[0m [35m54 Mbit/s[0m [35m55[0m [35m** [0m [35mWPA1 WPA2[0m [35mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [35mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [35mwlan0[0m [35mno[0m [35m [0m [35m/org/freedesktop/NetworkManager/AccessPoint/3[0m
===========================
Wi-Fi scan list (wlan1)
===========================
NAME SSID SSID-HEX BSSID MODE CHAN FREQ RATE SIGNAL BARS SECURITY WPA-FLAGS RSN-FLAGS DEVICE ACTIVE IN-USE DBUS-PATH
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[35mAP[1][0m [35mwlan1-ap-4[0m [35m776C616E312D61702D34[0m [35m94:2B:E8:F6:D2:86[0m [35mInfra[0m [35m1[0m [35m2412 MHz[0m [35m54 Mbit/s[0m [35m48[0m [35m** [0m [35mWPA1 WPA2[0m [35mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [35mpair_tkip pair_ccmp group_tkip group_ccmp psk[0m [35mwlan1[0m [35mno[0m [35m [0m [35m/org/freedesktop/NetworkManager/AccessPoint/4[0m
===========================
Wi-Fi scan list (wlan1)
===========================
NAME SSID SSID-HEX BSSID MODE CHAN FREQ RATE SIGNAL BARS SECURITY WPA-FLAGS RSN-FLAGS DEVICE ACTIVE IN-USE DBUS-PATH
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
<<<
]]
Let the test write the file by rerunning with NM_TEST_REGENERATE=1
See howto in clients/tests/test-client.py for details.
make[4]: *** [Makefile:20240: check-local-clients-tests-test-client] Error 1
make[4]: Leaving directory '/<<PKGBUILDDIR>>'
make[3]: *** [Makefile:18159: check-am] Error 2
make[3]: Leaving directory '/<<PKGBUILDDIR>>'
make[2]: *** [Makefile:17355: check-recursive] Error 1
make[2]: Leaving directory '/<<PKGBUILDDIR>>'
make[1]: *** [Makefile:18161: check] Error 2
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
dh_auto_test: make -j4 check VERBOSE=1 returned exit code 2
make: *** [debian/rules:8: binary-arch] Error 2
I can reliably reproduce the problem on a armel porterbox, so if you want me to run further diagnostics, test patches etc, please let me know