doc unit test: only warn when regenerated but not committed
keeping the error should make pipelines behave the same while showing only warning when you run the tests in development environment.
Before
$ sed -i '/:r\?type/d' nmci/cleanup.py $ pytest -k nmci_doc =========================================== test session starts ============================================ platform linux -- Python 3.11.1, pytest-7.1.3, pluggy-1.0.0 rootdir: /var/home/djasa/src/NetworkManager-ci, configfile: pyproject.toml collected 42 items / 41 deselected / 1 selected nmci/test_nmci.py F [100%] ================================================= FAILURES ================================================= ______________________________________________ test_nmci_doc _______________________________________________ def test_nmci_doc(): if sys.version.startswith("3.6."): pytest.skip("sphinx markdown builder does not work with python3.6 :(") print( "Running `sphinx-build -E -b markdown nmci/doc_src nmci/doc` to rebuild documentation:" ) proc = subprocess.run( ["sphinx-build", "-E", "-b", "markdown", "nmci/doc_src", "nmci/doc"] ) assert proc.returncode == 0, "Unable to build documentation" proc = subprocess.run(["git", "diff", "--quiet", "nmci/README.md"]) > assert proc.returncode == 0, "Documentation changed after rebuild." E AssertionError: Documentation changed after rebuild. E assert 1 == 0 E + where 1 = CompletedProcess(args=['git', 'diff', '--quiet', 'nmci/README.md'], returncode=1).returncode nmci/test_nmci.py:2057: AssertionError ------------------------------------------- Captured stdout call ------------------------------------------- Running `sphinx-build -E -b markdown nmci/doc_src nmci/doc` to rebuild documentation: Running Sphinx v4.2.0 building [mo]: targets for 0 po files that are out of date building [markdown]: targets for 1 source files that are out of date updating environment: [new config] 1 added, 0 changed, 0 removed reading sources... [100%] index looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [100%] index build succeeded. The markdown files are in nmci/doc. ========================================= short test summary info ========================================== FAILED nmci/test_nmci.py::test_nmci_doc - AssertionError: Documentation changed after rebuild. ===================================== 1 failed, 41 deselected in 1.62s ===================================== $ $ # again $ $ pytest -k nmci_doc =========================================== test session starts ============================================ platform linux -- Python 3.11.1, pytest-7.1.3, pluggy-1.0.0 rootdir: /var/home/djasa/src/NetworkManager-ci, configfile: pyproject.toml collected 42 items / 41 deselected / 1 selected nmci/test_nmci.py F [100%] ================================================= FAILURES ================================================= ______________________________________________ test_nmci_doc _______________________________________________ def test_nmci_doc(): if sys.version.startswith("3.6."): pytest.skip("sphinx markdown builder does not work with python3.6 :(") print( "Running `sphinx-build -E -b markdown nmci/doc_src nmci/doc` to rebuild documentation:" ) proc = subprocess.run( ["sphinx-build", "-E", "-b", "markdown", "nmci/doc_src", "nmci/doc"] ) assert proc.returncode == 0, "Unable to build documentation" proc = subprocess.run(["git", "diff", "--quiet", "nmci/README.md"]) > assert proc.returncode == 0, "Documentation changed after rebuild." E AssertionError: Documentation changed after rebuild. E assert 1 == 0 E + where 1 = CompletedProcess(args=['git', 'diff', '--quiet', 'nmci/README.md'], returncode=1).returncode nmci/test_nmci.py:2057: AssertionError ------------------------------------------- Captured stdout call ------------------------------------------- Running `sphinx-build -E -b markdown nmci/doc_src nmci/doc` to rebuild documentation: Running Sphinx v4.2.0 building [mo]: targets for 0 po files that are out of date building [markdown]: targets for 1 source files that are out of date updating environment: [new config] 1 added, 0 changed, 0 removed reading sources... [100%] index looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [100%] index build succeeded. The markdown files are in nmci/doc. ========================================= short test summary info ========================================== FAILED nmci/test_nmci.py::test_nmci_doc - AssertionError: Documentation changed after rebuild. ===================================== 1 failed, 41 deselected in 1.61s =====================================
After
$ git reset --hard HEAD is now at 173b1fdc nmci/cleanup.py: subtle bugs $ git checkout dj/nmci-doc-unit-test-warning Switched to branch 'dj/nmci-doc-unit-test-warning' Your branch and 'origin/dj/nmci-doc-unit-test-warning' have diverged, and have 3 and 1 different commits each, respectively. (use "git pull" to merge the remote branch into yours) $ sed -i '/:r\?type/d' nmci/cleanup.py $ # doc out of sync $ pytest -k nmci_doc =========================================== test session starts ============================================ platform linux -- Python 3.11.1, pytest-7.1.3, pluggy-1.0.0 rootdir: /var/home/djasa/src/NetworkManager-ci, configfile: pyproject.toml collected 42 items / 41 deselected / 1 selected nmci/test_nmci.py F [100%] ================================================= FAILURES ================================================= ______________________________________________ test_nmci_doc _______________________________________________ def test_nmci_doc(): if sys.version.startswith("3.6."): pytest.skip("sphinx markdown builder does not work with python3.6 :(") print( "Running `sphinx-build -E -b markdown nmci/doc_src nmci/doc` to rebuild documentation:" ) sph_res = run_sphinx_build_and_compare_results() assert sph_res["build_res"] == 0, "Unable to build documentation" > assert sph_res["cur_same_as_gen"], "Outdated documentation was rebuilt" E AssertionError: Outdated documentation was rebuilt E assert False nmci/test_nmci.py:2078: AssertionError ------------------------------------------- Captured stdout call ------------------------------------------- Running `sphinx-build -E -b markdown nmci/doc_src nmci/doc` to rebuild documentation: Running Sphinx v4.2.0 building [mo]: targets for 0 po files that are out of date building [markdown]: targets for 1 source files that are out of date updating environment: [new config] 1 added, 0 changed, 0 removed reading sources... [100%] index looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [100%] index build succeeded. The markdown files are in .tmp. ========================================= short test summary info ========================================== FAILED nmci/test_nmci.py::test_nmci_doc - AssertionError: Outdated documentation was rebuilt ===================================== 1 failed, 41 deselected in 1.77s ===================================== $ # doc in sync but not commited yet $ pytest -k nmci_doc =========================================== test session starts ============================================ platform linux -- Python 3.11.1, pytest-7.1.3, pluggy-1.0.0 rootdir: /var/home/djasa/src/NetworkManager-ci, configfile: pyproject.toml collected 42 items / 41 deselected / 1 selected nmci/test_nmci.py . [100%] ============================================= warnings summary ============================================= nmci/test_nmci.py::test_nmci_doc /var/home/djasa/src/NetworkManager-ci/nmci/test_nmci.py:2082: UserWarning: Documentation in nmci/README.md is current but not committed. Commit it before pushing! warnings.warn(w, UserWarning) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =============================== 1 passed, 41 deselected, 1 warning in 1.58s ================================
@RunTests:pass
Edited by David Jaša