Skip to content

Enable "stress" tests to be run on demand

Michał Kępień requested to merge michal/enable-stress-tests-to-be-run-on-demand into main

The BIND QA team believes it would be useful for SWENG to run the "stress" test on demand. The list of "stress" tests to run would be specified by setting CI variables at pipeline creation time.

This MR proposes one way of enabling the above. I tried to come up with a reasonable compromise between fine-grained control and ease of use. I based the approach proposed here on the assumption that we are usually more interested in specifying the test mode (authoritative vs. recursive testing) than the platforms to run the test on (and that Linux/amd64 is the primary deployment platform for BIND).

A detailed description of the approach proposed in this MR can be found in the commit log message. Here are some examples of what CI variables would need to be set to run the desired set of tests:

Run in authoritative mode on the default OS/architecture

  • BIND_STRESS_TEST_MODE=authoritative

Run in both supported modes on FreeBSD (default architecture)

  • BIND_STRESS_TEST_MODE=authoritative,recursive
  • BIND_STRESS_TEST_OS=freebsd

Run all supported variants

  • BIND_STRESS_TEST_MODE=authoritative,recursive
  • BIND_STRESS_TEST_OS=freebsd,linux
  • BIND_STRESS_TEST_ARCH=amd64,arm64

As shown above, this approach should be simple to use in the (assumed) typical cases (only one variable is required to be set) while also allowing fine-grained control when it is necessary.

Other possible approaches include:

  • conflating all parameters into a single CI variable (e.g. BIND_STRESS_TEST=authoritative/linux/amd64,recursive/freebsd/amd64); this retains fine-grained control, but is more cumbersome to use (IMHO),

  • only enabling one parameter (e.g. test mode) to be set using a CI variable and running jobs for all possible values of the other parameters (e.g. BIND_STRESS_TEST=authoritative would create jobs for both Linux and FreeBSD on both amd64 and arm64); this retains ease of use, but does not allow fine-grained control.

@bind-team: I would appreciate your opinions on whether the approach proposed in this MR is reasonable or whether you would do it differently.

@mnowak: I would like to hear back from you whether the implementation itself is clear enough to you.

Merge request reports