Unit tests: add 'behave --dry', make Black a bit more useful, use pyproject.toml

Changes in this MR are invisible if everything is OK, however some behaviours are quite different with it!

Add pyproject.toml

Some behaviours by this MR are best achieved by configuration file. These days, the pyproject.toml seems the best way to achieve that as it is a single file with a rich but sane syntax.

Add behave --dry

The behave --dry checks all the *.feature files if they are well formed from behave's perspective and if all the steps in them are defined. The behave then prints the names of the unrecognized steps (usually steps with typos in my case...) to the standard output where they are now captured by pytest which includes them both in its console output and in JUnit file. This is achieved by:

• capturing all outputs of all tests, but
• discarding those of successful tests

Notes:

• in order to get files and lines where undefined steps are, we need to parse output of behave's steps.usage formatter in unit tests (the new print_undefs() working with captured output). I've created PRs for behave (1089, 1090) that'd allow us to get the output right from behave, let's see how maintainers see it and when (if...) they release some new version.
• a side effect of this change is that addition of a dependency in step implementations will force addition of this dependency to the .gitlab-ci.yml as well.

Black changes

The --check argument allows us to get rid of assert not proc.std(out|err) which in turn allows the pytest to treat blacks outputs the same as behave's. In black's case, stdout contains diff and stderr the list of would-be-reformatted files.

The addition of pyproject.toml also allows modification of files-to-be-checked by the unit test: we move from the hardcoded list of files to be checked to a regex of skipped files in pyproject.toml (in the exclude key). This change means that:

• new python files in the repo are subject to the test_black_code_fromatting unit test unless you add them to the exclude list in pyproject.toml
• call of black . will no longer reformat files that need to be refactored before subjecting them to the unit test

GitLab workaround

The last commit in this MR works around GitLab issue of not recognizing <system-out> and <system-err> elements in the JUnit file (and only embedding in the web UI the content of <failure> element which is quite useless in pytest-generated JUnit files as it only contains print of the testing function; GL issue).

This commit is self-contained so it can be reverted in the future, after GL learns of <system-out> and <system-err> elements.

Examples!

Console output

clean run

$ pytest
======================================= test session starts ========================================
platform linux -- Python 3.11.1, pytest-7.1.3, pluggy-1.0.0
rootdir: /var/home/djasa/src/NetworkManager-ci, configfile: pyproject.toml
collected 41 items                                                                                 

nmci/test_nmci.py ...........................ss............                                  [100%]

================================== 39 passed, 2 skipped in 7.54s ==================================

we introduce errors

expand me...

$ git switch - ; git show
Previous HEAD position was ef44db42 Unit tests: workaround GitLab's ignorance of <system-(out|err)>
Switched to branch 'dj/behave-dry-unit-test-v2'
Your branch is up to date with 'origin/dj/behave-dry-unit-test-v2'.
commit 47f4d481769cf6a807e9c08a943bc3789d7e9723 (HEAD -> dj/behave-dry-unit-test-v2, origin/dj/behave-dry-unit-test-v2)
Author: David Jasa <djasa@redhat.com>
Date:   Wed Mar 8 12:55:36 2023 +0100

    delete me: example of malformed steps

diff --git a/features/scenarios/general.feature b/features/scenarios/general.feature
index d8483932..4bf57366 100644
--- a/features/scenarios/general.feature
+++ b/features/scenarios/general.feature
@@ -10,6 +10,7 @@ Feature: nmcli - general
     @pass
     Scenario: Dummy scenario that is supposed to pass
     * Execute "nmcli --version"
+    * hello
 
 
     @last_copr_build_check
diff --git a/features/scenarios/ipv4.feature b/features/scenarios/ipv4.feature
index c8aeba16..42be575d 100644
--- a/features/scenarios/ipv4.feature
+++ b/features/scenarios/ipv4.feature
@@ -10,6 +10,7 @@ Feature: nmcli: ipv4
 
     @ipv4_method_static_no_IP
     Scenario: nmcli - ipv4 - method - static without IP
+    * we're
     * Add "ethernet" connection named "con_ipv4" for device "eth3"
     * Open editor for connection "con_ipv4"
     * Submit "set ipv4.method static" in editor
diff --git a/features/scenarios/ipv6.feature b/features/scenarios/ipv6.feature
index 0e63261c..53ba5562 100644
--- a/features/scenarios/ipv6.feature
+++ b/features/scenarios/ipv6.feature
@@ -10,6 +10,7 @@
 
     @ipv6_method_static_without_IP
     Scenario: nmcli - ipv6 - method - static without IP
+    * undefined!
     * Add "ethernet" connection named "con_ipv6" for device "eth3" with options "autoconnect no"
       * Open editor for connection "con_ipv6"
       * Submit "set ipv6.method static" in editor
diff --git a/nmci/test_nmci.py b/nmci/test_nmci.py
index 772c3cb6..ff2a161c 100755
--- a/nmci/test_nmci.py
+++ b/nmci/test_nmci.py
@@ -2033,7 +2033,8 @@ def test_behave_steps_in_feature_files(capfd):
 # of the file.
 def test_black_code_fromatting():
 
-    if os.environ.get("NMCI_NO_BLACK") == "1":
+    if os.environ.get(
+            "NMCI_NO_BLACK") == "1":
         pytest.skip("skip formatting test with python-black (NMCI_NO_BLACK=1)")
 

let's run the tests again

$ pytest
======================================= test session starts ========================================
platform linux -- Python 3.11.1, pytest-7.1.3, pluggy-1.0.0
rootdir: /var/home/djasa/src/NetworkManager-ci, configfile: pyproject.toml
collected 41 items                                                                                 

nmci/test_nmci.py ...........................ss..........FF                                  [100%]

============================================= FAILURES =============================================
________________________________ test_behave_steps_in_feature_files ________________________________

capfd = <_pytest.capture.CaptureFixture object at 0x7f202155d6d0>

    def test_behave_steps_in_feature_files(capfd):
        b_cli = ["behave", "-d", "-c", "--no-summary", "--no-snippets", "-f", "steps.usage"]
        try:
            proc = subprocess.run(b_cli)
        except FileNotFoundError:
            pytest.skip("behave is not available for check if all the steps are recognized")
    
        cap = capfd.readouterr()
        assert len(cap.out) > 0
        print_undefs(cap.out)
>       assert proc.returncode == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = CompletedProcess(args=['behave', '-d', '-c', '--no-summary', '--no-snippets', '-f', 'steps.usage'], returncode=1).returncode

nmci/test_nmci.py:2029: AssertionError
--------------------------------------- Captured stdout call ---------------------------------------
UNDEFINED STEPS[3]:
  * hello                                 # features/scenarios/general.feature:13
  * we're                                 # features/scenarios/ipv4.feature:13
  * undefined!                            # features/scenarios/ipv6.feature:13

____________________________________ test_black_code_fromatting ____________________________________

    def test_black_code_fromatting():
    
        if os.environ.get(
                "NMCI_NO_BLACK") == "1":
            pytest.skip("skip formatting test with python-black (NMCI_NO_BLACK=1)")
    
        try:
            proc = subprocess.run(["black", "--no-color", "--diff", "--check", "."])
        except FileNotFoundError:
            pytest.skip("python black is not available")
    
>       assert proc.returncode == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = CompletedProcess(args=['black', '--no-color', '--diff', '--check', '.'], returncode=1).returncode

nmci/test_nmci.py:2045: AssertionError
--------------------------------------- Captured stdout call ---------------------------------------
--- nmci/test_nmci.py	2023-03-08 12:08:45.679423 +0000
+++ nmci/test_nmci.py	2023-03-08 13:29:19.619168 +0000
@@ -2031,12 +2031,11 @@
 
 # This test should always run as last. Keep it at the bottom
 # of the file.
 def test_black_code_fromatting():
 
-    if os.environ.get(
-            "NMCI_NO_BLACK") == "1":
+    if os.environ.get("NMCI_NO_BLACK") == "1":
         pytest.skip("skip formatting test with python-black (NMCI_NO_BLACK=1)")
 
     try:
         proc = subprocess.run(["black", "--no-color", "--diff", "--check", "."])
     except FileNotFoundError:
--------------------------------------- Captured stderr call ---------------------------------------
would reformat nmci/test_nmci.py

Oh no! 💥 💔 💥
1 file would be reformatted, 40 files would be left unchanged.
===================================== short test summary info ======================================
FAILED nmci/test_nmci.py::test_behave_steps_in_feature_files - AssertionError: assert 1 == 0
FAILED nmci/test_nmci.py::test_black_code_fromatting - AssertionError: assert 1 == 0
============================= 2 failed, 37 passed, 2 skipped in 8.50s ==============================

Screenshots

---

@RunTests:pass