Replace system test runner with pytest
The legacy solution for running systems test has evolved over the course of years and is currently a mix of shell & perl scripts intermingled with the build system, while some of the system tests utilize pytest. Implementing a more consistent solution using just pytest as a runner could bring following benefits:
- better test run isolation (i.e. artifacts from previous run don't interfere with current test run)
- more precise control over test selection (running just a single test case)
- getting rid of perl+shell glue scripts
- a simpler and more standard way to run and parallelize test runs
- solid foundation for future extensions (e.g. wrapping test execution inside a network/pid namespace)
For a transitory period of time, the legacy test framework should be supported, since it'd be difficult to replace everything at once. The pytest runner should be available in 9.18+, it'd be prudent to keep the legacy runner support until 9.16 reaches EOL. By that time, we should have enough insight to determine whether pytest proves to be a suitable replacement and throw away the legacy runner from supported branches at that point.
Migration plan for moving to pytest runner and dropping the legacy runner support:
- Phase I - pytest runner development, legacy runner supported
-
initial implementation of the pytest runner (#3978 (closed), !6809 (merged)) -
support out-of-tree tests (#4246 (closed)) -
resolve support on CI systems with old pytest (OpenBSD, CentOS 7) (!8193 (merged)) -
implement any missing (and desired) features from legacy runner (#4252 (closed)) -
configure make check
to invoke pytest (#4262 (closed))
-
- Phase II - deprecating legacy runner - 9.19-only
-
remove legacy runner control script(s) - legacy.run.sh, get_ports.sh ... (#4251 (closed)) -
remove no longer needed scripts from system tests (e.g. clean.sh) (#4261 (closed)) -
remove conf.sh(.common) and declare variables in pytest only (!8800 (merged)) -
remove the Makefile entanglement -
declare python and pytest-xdist as required dependencies for tests + document -
address any FUTURE
comments in the pytest runner code
-
- Phase III - cleanup after legacy runner
-
rewrite start.pl/stop.pl to python (related #3198 (closed)) -
rewrite remaining setup/teardown perl&shell scripts to python -
rewrite setup.sh/prereq.sh system tests scripts to pytest fixtures -
ensure system test documentation is up to date
-