This is a work-in-progress MR to demonstrate a possible solution for #4261. I'd like to get your opinion on whether this approach and added maintenance burden is really worth the feature to detect unexpected test artifacts.
The proposed solution utilizes
.gitignore files to ignore the expected test artifacts. However, this gets ugly quick, because the tests are already executed inside temporary directories which are ignored by git. This necessitates a creation of yet-another temporary directory and re-creating the file structure inside it, see f8378698.
Another issue/feature is that all .gitignore files up to git root are taken into account. This can be both useful (like placing some common patterns like
bin/tests/system/.gitignore) but also confusing (why does there need to be a test-specific rule for
ans.run.prev, but not
Since it's a git mechanism, new
pygit2 dependency is required (otherwise the check is skipped). Also, it only works in git tree.
Backporting this to v9.18 would also slightly cripple the legacy runner, since
clean.sh would no longer work and
git clean would have to be used instead. On the other hand, not backporting this would be even a bigger nightmare, since that would create a fairly significant difference between the test files and any future
clean.sh conflicts would have to be resolved during every affected v9.18 test backport, creating additional maintenance burden.
After spelling all of this out, I really think it's better to abandon the ability to detect unexpected test files and go with the simpler and cleaner solution - tests are already executed in temporary directories, so we could simply drop the
clean.sh scripts for the pytest runner and not bother with listing all the test artifacts.