Skip to content

Report hung system tests

Michal Nowak requested to merge mnowak/monitor-stuck-system-tests into main

At times, a problem might occur where a test is not responding, especially in the CI, determining the specific test responsible can be difficult. Fortunately, when running tests with the pytest runner, pytest sets the PYTEST_CURRENT_TEST environment variable to the current test nodeid and stage. Afterward, the variable can be examined to identify the test that has stopped responding.

The monitoring script needs to be started in the background. Still, the shell executor used for BSD and FIPS testing can't handle the background process cleanly, and the script step will wait for the background process for the entire duration of the background process (currently 3000 seconds). Therefore, run the monitoring script only when the Docker executor is used where this is not a problem.

Validation jobs:

There's a GitLab Runner issue for shell and Kubernetes executors that prevents us from using the script on shell executor (FIPS and BSD jobs). The issue is closed, but looking at the fix, it seems to me that only the Kubernetes executor was fixed.

Prereq: isc-projects/images!264

Edited by Michal Nowak

Merge request reports