Draft: Check unexpected files created during system test run

Tom Krizek requested to merge 4261-pytest-detect-unexpected-test-files into main

This is a work-in-progress MR to demonstrate a possible solution for #4261. I'd like to get your opinion on whether this approach and added maintenance burden is really worth the feature to detect unexpected test artifacts.

The proposed solution utilizes .gitignore files to ignore the expected test artifacts. However, this gets ugly quick, because the tests are already executed inside temporary directories which are ignored by git. This necessitates a creation of yet-another temporary directory and re-creating the file structure inside it, see f8378698.

Another issue/feature is that all .gitignore files up to git root are taken into account. This can be both useful (like placing some common patterns like named.conf into bin/tests/system/.gitignore) but also confusing (why does there need to be a test-specific rule for ans.run.prev, but not ans.run?).

Since it's a git mechanism, new pygit2 dependency is required (otherwise the check is skipped). Also, it only works in git tree.

Finally, each test has to have its clean.sh reworked into .gitignore instead. Sometimes the .gitignore turns out fairly simple, othertimes not so much.

Backporting this to v9.18 would also slightly cripple the legacy runner, since clean.sh would no longer work and git clean would have to be used instead. On the other hand, not backporting this would be even a bigger nightmare, since that would create a fairly significant difference between the test files and any future .gitignore vs clean.sh conflicts would have to be resolved during every affected v9.18 test backport, creating additional maintenance burden.


After spelling all of this out, I really think it's better to abandon the ability to detect unexpected test files and go with the simpler and cleaner solution - tests are already executed in temporary directories, so we could simply drop the clean.sh scripts for the pytest runner and not bother with listing all the test artifacts.

Merge request reports