1. 01 May, 2020 1 commit
  2. 30 Apr, 2020 2 commits
  3. 29 Apr, 2020 1 commit
    • Ondřej Surý's avatar
      Add a pytest runner to run.sh · cf510593
      Ondřej Surý authored
      The system tests currently uses patchwork of shell scripts which doesn't
      offer proper error handling.
      This commit introduced option to write new tests in pytest framework
      that also allows easier manipulation of DNS traffic (using dnspython),
      native XML and JSON manipulation and proper error reporting.
  4. 27 Apr, 2020 1 commit
    • Ondřej Surý's avatar
      Refactor the script that gets/sets the ports for system tests · 52683934
      Ondřej Surý authored
      The current script used ephemeral port range which clashed with the
      ports used by the tools (dig, ...), and the range always started with
      the first port and there was 100 ports allocated for each system test.
      In this commit, the first port has been randomized, the get_ports.sh
      script outputs the variables (the output has to be eval'ed from run.sh)
      and there's less waste in the port range.
  5. 24 Apr, 2020 1 commit
    • Ondřej Surý's avatar
      Improve the LOG_DRIVER and LOG_COMPILER for the system tests · 58138180
      Ondřej Surý authored
      There are several improvements over the default/previous behaviour of
      the test log driver and log compiler:
       * The system-test-driver.sh was dropped (it was used incorrectly)
       * The run.sh script is now both log compiler and cli script to run
         individual tests
       * The custom-test-driver was added as extended version of the automake
         test-driver with capability to tee the test output to stdout when
         `--verbose yes` is passed to it (you can use LOG_DRIVER_FLAGS to
         add the option by default)
       * Makefile.am has been extended to honor V=1 for the system tests
         test-driver (e.g. V=1 adds `--verbose yes` to AM_LOG_DRIVER_FLAGS)
  6. 21 Apr, 2020 1 commit
    • Ondřej Surý's avatar
      Complete rewrite the BIND 9 build system · 978c7b2e
      Ondřej Surý authored
      The rewrite of BIND 9 build system is a large work and cannot be reasonable
      split into separate merge requests.  Addition of the automake has a positive
      effect on the readability and maintainability of the build system as it is more
      declarative, it allows conditional and we are able to drop all of the custom
      make code that BIND 9 developed over the years to overcome the deficiencies of
      autoconf + custom Makefile.in files.
      This squashed commit contains following changes:
      - conversion (or rather fresh rewrite) of all Makefile.in files to Makefile.am
        by using automake
      - the libtool is now properly integrated with automake (the way we used it
        was rather hackish as the only official way how to use libtool is via
      - the dynamic module loading was rewritten from a custom patchwork to libtool's
        libltdl (which includes the patchwork to support module loading on different
        systems internally)
      - conversion of the unit test executor from kyua to automake ...
  7. 11 Dec, 2019 1 commit
  8. 06 Dec, 2019 3 commits
    • Michał Kępień's avatar
      Automatically run clean.sh from run.sh · d8905b7a
      Michał Kępień authored
      The first step in all existing setup.sh scripts is to call clean.sh.  To
      reduce code duplication and ensure all system tests added in the future
      behave consistently with existing ones, invoke clean.sh from run.sh
      before calling setup.sh.
    • Michał Kępień's avatar
      Remove bin/tests/system/clean.sh · bf3eeac0
      Michał Kępień authored
      Since the role of the bin/tests/system/clean.sh script has now been
      reduced to calling a given system test's clean.sh script, remove the
      former altogether and replace its only use with a direct invocation of
      the latter.
    • Michał Kępień's avatar
      Remove the -r switch from system test scripts · b4d37878
      Michał Kępień authored
      Since files containing system test output are no longer stored in test
      subdirectories, bin/tests/system/clean.sh no longer needs to take care
      of removing the test.output file for a given test as testsummary.sh
      already takes care of that and even if a test suite terminates
      abnormally and another one is started, tee invoked without the -a
      command line switch overwrites the destination file if it exists, so
      leftover test.output.* files from previous test suite runs are not a
      concern.  Remove the -r command line switch and the code associated with
      it from the relevant scripts.
  9. 28 Nov, 2019 2 commits
  10. 26 Nov, 2019 1 commit
  11. 20 Nov, 2019 1 commit
  12. 18 Jun, 2019 1 commit
    • Michał Kępień's avatar
      Always fail a system test if crashes are detected · 7706f229
      Michał Kępień authored
      In certain situations (e.g. a named instance crashing upon shutdown in a
      system test which involves shutting down a server and restarting it
      afterwards), a system test may succeed despite a named crash being
      triggered.  This must never be the case.  Extend run.sh to mark a test
      as failed if core dumps or log lines indicating assertion failures are
      detected (the latter is only an extra measure aimed at test environments
      in which core dumps are not generated; note that some types of crashes,
      e.g. segmentation faults, will not be detected using this method alone).
  13. 03 Dec, 2018 2 commits
  14. 13 Jul, 2018 1 commit
  15. 12 Jun, 2018 1 commit
  16. 07 Mar, 2018 1 commit
  17. 23 Feb, 2018 2 commits
  18. 22 Feb, 2018 1 commit
  19. 21 Feb, 2018 1 commit
    • Michał Kępień's avatar
      Allow retaining system test output using an environment variable · 38620438
      Michał Kępień authored
      Instead of exporting an environment variable containing a command line
      argument (NOCLEAN="-n"), extend run.sh to handle a "boolean" environment
      variable (SYSTEMTEST_NO_CLEAN) itself.  The former method is buggy
      because the value of NOCLEAN is set in parallel.mk when that file is
      first created, but it is not subsequently updated upon each test run
      (because make considers parallel.mk to be up to date).
      To retain backward compatibility, the "-n" command line argument for
      run.sh is still supported (and has a higher priority than the relevant
      environment variable).
      The SYSTEMTEST_NO_CLEAN environment variable can also be used directly
      to prevent cleanup when using "make test" instead of runall.sh.
      Apart from fixing a bug, this simplifies the way runall.sh controls
      run.sh behavior due to the Makefile being bypassed.  Direct processing
      of environment variables in run.sh is more scalable in the long run,
      given that the previously utilized technique, even with its
      implementation fixed, would still require Makefile.in to be modified in
      two places each time a new flag needed to be passed from runall.sh to
  20. 16 Feb, 2018 15 commits