Skip to content

Disable OpenBSD system test parallelism in CI

Michal Nowak requested to merge mnowak/update_TEST_PARALLEL_JOBS_for_BSD into main

OpenBSD system tests in the CI are disproportionally unstable to the rest of the platforms. Intermittent timing issues are inherent to the system test, but OpenBSD is something else.

Just today, OpenBSD system test jobs failed in 4 out of 5 daily pipelines (main, v9_18, v9_16, v9_16_sub), 7 system tests failed in them (about a half of today's overall job and test failures).

QA observed in the past that system tests on OpenBSD are relatively stable when run locally. It's possible that OpenBSD can't deal with outside IO and CPU pressure and "gives way" to other platforms when run simultaneously on the same CI Runner. There may be a way to configure the OpenBSD image to behave more closely to FreeBSD or Linux, but we lack the knowledge.

Given that we have OpenBSD in the CI mostly because we want to keep BIND 9 broadly multiplatform on UNIX and that the OpenBSD environment helps us straight with regards to maintaining system tests POSIXy, let's make it more stable by disabling system test parallelism in CI on OpenBSD, even if it means it will run 3-4× longer in the daily pipeline. It will not avoid all OpenBSD quirks, e.g., #2345, and it will still fail, but less.

With TEST_PARALLEL_JOBS: 1: Out of 9 jobs on three maintained BIND 9 branches (3181983, 3182204, 3181985, 3184221, 3184222, 3184223, 3184452, 3184449, 3184450) 4 system tests failed in 4 CI failed jobs, a moderate improvement.

Merge request reports