Cross-version testing with named configurations
In #3381 (closed) (and #3385 (closed)), we committed a backward-incompatible change to BIND 9.19.5, 9.18.7, and 9.16.33, explicitly requiring "inline-signing" for every "dnssec-policy".
We did this backward-incompatible change deliberately, knowing the consequences for users and their configurations. But if we didn't, say, we were unaware this is a backward-incompatible change and fixed failing systems test by "tweaking a nob to make the CI pass", we would not have a second look before the change hits user configurations.
"cross-version-config-tests" CI job is such a second look. It will run system tests from the latest release tag specific to the particular branch (e.g., v9.19.12 for the "main" branch) with BIND 9 binaries from the current "HEAD" (the future v9.19.13). This Frankenstein build gets conceived by altering the "TOP_BUILDDIR" variable in "bin/tests/system/conf.sh".
Caveats:
- System tests that use binary modules are "ABI sensitive" and won't work (e.g., dyndb, hooks).
- Only system test configurations are tested; no actual test code is run.
- Problems with namedN.conf configurations are not identified.
When backward-incompatible change happens, the CI job is expected to fail. If the change was deliberate, the job should have a patch line in the "script" section for the duration of the release, e.g.:
git show 5830aa9dc66bccfb564da65f678f0f8eefe7ffef d62439e2ecbab9d4af9dd509a54d2450f315c7d4 -- bin/tests/system/{nsec3,checkds,kasp}/ | patch -p1
Validation job for the no-breakage situation: https://gitlab.isc.org/isc-projects/bind9/-/jobs/3342247.
Validation job for the situation where the backward-incompatible change happened: https://gitlab.isc.org/isc-projects/bind9/-/jobs/3339938 (as expected nsec3
, checkds
, and kasp
).
Validation job for the situation where the backward-incompatible change happened, but with a temporary fix: https://gitlab.isc.org/isc-projects/bind9/-/jobs/3342416.
QA discussion from February.