Support using pytest to execute the system tests

Add an option to execute the entire system test suite with pytest. This should co-exist along with the legacy runner which will be removed at a later point.

For reasons, benefits and timeline, refer to the meta issue #3810.