add predictable path names for pytest runner results
The particular missing feature that I've used the most is that the old runner puts test results in a predictable place. Let's say I want to run the resolver test and examine the contents of
resolver/ns3/named.run
- with the legacy runner I can just do that, I don't have to figure out that this time the results are inresolver_tmp_pnxahq01
. If there's a way to tell pytest to put the test results in the test directory - or even just in the same place for two consecutive runs - I haven't discovered it. I often script these things; having the directory name change on every run makes that unnecessarily difficult, so I switch back tolegacy.run.sh
and then it's fine.
Add a symlink with a predictable path to the latest test run(s). It's a convenience thing that should allow easier inspection of the results.
A single system test may consist of multiple modules which are run separately, e.g. doth
has:
<Module doth/tests_gnutls.py>
<Module doth/tests_sh_doth.py>
<Module doth/tests_sslyze.py>
which may be executed independently and end up in different temporary directories. The symlink should include a module name/identifier to differentiate between them.
I think creating a directory like bin/tests/system/_last_test_run
where the stable symlinks could be placed should work well.
Related #3810, #4251 (closed)