1. 31 Jul, 2020 1 commit
    • Michal Nowak's avatar
      Do not remove $systest on out-of-tree builds · dcecbbd1
      Michal Nowak authored
      Previously, $systest directory was removed for out-of-tree builds. For
      directories with compiled content (binaries), this breaks subsequent
      make invocations:
          make: Target 'check' not remade because of errors.
          Making all in dyndb/driver
          /bin/bash: line 20: cd: dyndb/driver: No such file or directory
          Making all in dlzexternal/driver
          /bin/bash: line 20: cd: dlzexternal/driver: No such file or directory
      Current approach removes only files known to Git and empty directories
      in $systest (including $systest).
  2. 30 Jul, 2020 2 commits
    • Michal Nowak's avatar
      Drop $SYSTEMTESTTOP from bin/tests/system/ · 093af1c0
      Michal Nowak authored
      The $SYSTEMTESTTOP shell variable if often set to .. in various shell
      scripts inside bin/tests/system/, but most of the time it is only
      used one line later, while sourcing conf.sh. This hardly improves
      code readability.
      $SYSTEMTESTTOP is also used for the purpose of referencing
      scripts/files living in bin/tests/system/, but given that the
      variable is always set to a short, relative path, we can drop it and
      replace all of its occurrences with the relative path without adversely
      affecting code readability.
    • Michał Kępień's avatar
      Only run system tests as root in developer mode · 3ef106f6
      Michał Kępień authored
      Running system tests with root privileges is potentially dangerous.
      Only allow it when explicitly requested (by building with
  3. 17 Jul, 2020 3 commits
    • Michal Nowak's avatar
      Check tests for core files regardless of test status · 1b13123c
      Michal Nowak authored
      Failed test should be checked for core files et al. and have
      backtrace generated.
    • Michal Nowak's avatar
      Rationalize backtrace logging · 05c13e50
      Michal Nowak authored
      GDB backtrace generated via "thread apply all bt full" is too long for
      standard output, lets save them to .txt file among other log files.
    • Michal Nowak's avatar
      Ensure various test issues are treated as failures · b232e858
      Michal Nowak authored
      Make sure bin/tests/system/run.sh returns a non-zero exit code if any of
      the following happens:
        - the test being run produces a core dump,
        - assertion failures are found in the test's logs,
        - ThreadSanitizer reports are found after the test completes,
        - the servers started by the test fail to shut down cleanly.
      This change is necessary to always fail a test in such cases (before the
      migration to Automake, test failures were determined based on the
      presence of "R:<test-name>:FAIL" lines in the test suite output and thus
      it was not necessary for bin/tests/system/run.sh to return a non-zero
      exit code).
  4. 09 Jun, 2020 1 commit
    • Michał Kępień's avatar
      Fix SYSTEMTESTTOP on Windows · 1861866f
      Michał Kępień authored
      Without SYSTEMTESTTOP=.. lines in tests.sh scripts, SYSTEMTESTTOP is
      being set to an absolute path.  On Windows, this means that an absolute
      Cygwin path gets passed as a command line argument to native Windows
      binaries, which cannot work and causes system tests to break.  Fix by
      passing SYSTEMTESTTOP through cygpath on Windows, which causes that
      variable to be set to an absolute "mixed mode" path (Windows path with
      forward slashes).
  5. 04 Jun, 2020 1 commit
    • Mark Andrews's avatar
      Preserve PATH and LD_LIBRARY_PATH when calling make · d9b29fcd
      Mark Andrews authored
      When ./run.sh <test> is invoked, it acts as a wrapper around
      `env - TESTS="<test>" make -e check` to preserve the ability to build
      files defined only in the `check` target.  Unfortunately, cleaning the
      full environment had a side-effect of some tests failing due to missing
      binaries and libraries.  We now preserve the two most important
      variables - PATH and LD_LIBRARY_PATH.
  6. 12 May, 2020 2 commits
  7. 01 May, 2020 2 commits
  8. 30 Apr, 2020 2 commits
  9. 29 Apr, 2020 1 commit
    • Ondřej Surý's avatar
      Add a pytest runner to run.sh · cf510593
      Ondřej Surý authored
      The system tests currently uses patchwork of shell scripts which doesn't
      offer proper error handling.
      This commit introduced option to write new tests in pytest framework
      that also allows easier manipulation of DNS traffic (using dnspython),
      native XML and JSON manipulation and proper error reporting.
  10. 27 Apr, 2020 1 commit
    • Ondřej Surý's avatar
      Refactor the script that gets/sets the ports for system tests · 52683934
      Ondřej Surý authored
      The current script used ephemeral port range which clashed with the
      ports used by the tools (dig, ...), and the range always started with
      the first port and there was 100 ports allocated for each system test.
      In this commit, the first port has been randomized, the get_ports.sh
      script outputs the variables (the output has to be eval'ed from run.sh)
      and there's less waste in the port range.
  11. 24 Apr, 2020 1 commit
    • Ondřej Surý's avatar
      Improve the LOG_DRIVER and LOG_COMPILER for the system tests · 58138180
      Ondřej Surý authored
      There are several improvements over the default/previous behaviour of
      the test log driver and log compiler:
       * The system-test-driver.sh was dropped (it was used incorrectly)
       * The run.sh script is now both log compiler and cli script to run
         individual tests
       * The custom-test-driver was added as extended version of the automake
         test-driver with capability to tee the test output to stdout when
         `--verbose yes` is passed to it (you can use LOG_DRIVER_FLAGS to
         add the option by default)
       * Makefile.am has been extended to honor V=1 for the system tests
         test-driver (e.g. V=1 adds `--verbose yes` to AM_LOG_DRIVER_FLAGS)
  12. 21 Apr, 2020 1 commit
    • Ondřej Surý's avatar
      Complete rewrite the BIND 9 build system · 978c7b2e
      Ondřej Surý authored
      The rewrite of BIND 9 build system is a large work and cannot be reasonable
      split into separate merge requests.  Addition of the automake has a positive
      effect on the readability and maintainability of the build system as it is more
      declarative, it allows conditional and we are able to drop all of the custom
      make code that BIND 9 developed over the years to overcome the deficiencies of
      autoconf + custom Makefile.in files.
      This squashed commit contains following changes:
      - conversion (or rather fresh rewrite) of all Makefile.in files to Makefile.am
        by using automake
      - the libtool is now properly integrated with automake (the way we used it
        was rather hackish as the only official way how to use libtool is via
      - the dynamic module loading was rewritten from a custom patchwork to libtool's
        libltdl (which includes the patchwork to support module loading on different
        systems internally)
      - conversion of the unit test executor from kyua to automake parallel driver
      - conversion of the system test executor from custom make/shell to automake
        parallel driver
      - The GSSAPI has been refactored, the custom SPNEGO on the basis that
        all major KRB5/GSSAPI (mit-krb5, heimdal and Windows) implementations
        support SPNEGO mechanism.
      - The various defunct tests from bin/tests have been removed:
        bin/tests/optional and bin/tests/pkcs11
      - The text files generated from the MD files have been removed, the
        MarkDown has been designed to be readable by both humans and computers
      - The xsl header is now generated by a simple sed command instead of
        perl helper
      - The <irs/platform.h> header has been removed
      - cleanups of configure.ac script to make it more simpler, addition of multiple
        macros (there's still work to be done though)
      - the tarball can now be prepared with `make dist`
      - the system tests are partially able to run in oot build
      Here's a list of unfinished work that needs to be completed in subsequent merge
      - `make distcheck` doesn't yet work (because of system tests oot run is not yet
      - documentation is not yet built, there's a different merge request with docbook
        to sphinx-build rst conversion that needs to be rebased and adapted on top of
        the automake
      - msvc build is non functional yet and we need to decide whether we will just
        cross-compile bind9 using mingw-w64 or fix the msvc build
      - contributed dlz modules are not included neither in the autoconf nor automake
  13. 11 Dec, 2019 1 commit
  14. 06 Dec, 2019 3 commits
    • Michał Kępień's avatar
      Automatically run clean.sh from run.sh · d8905b7a
      Michał Kępień authored
      The first step in all existing setup.sh scripts is to call clean.sh.  To
      reduce code duplication and ensure all system tests added in the future
      behave consistently with existing ones, invoke clean.sh from run.sh
      before calling setup.sh.
    • Michał Kępień's avatar
      Remove bin/tests/system/clean.sh · bf3eeac0
      Michał Kępień authored
      Since the role of the bin/tests/system/clean.sh script has now been
      reduced to calling a given system test's clean.sh script, remove the
      former altogether and replace its only use with a direct invocation of
      the latter.
    • Michał Kępień's avatar
      Remove the -r switch from system test scripts · b4d37878
      Michał Kępień authored
      Since files containing system test output are no longer stored in test
      subdirectories, bin/tests/system/clean.sh no longer needs to take care
      of removing the test.output file for a given test as testsummary.sh
      already takes care of that and even if a test suite terminates
      abnormally and another one is started, tee invoked without the -a
      command line switch overwrites the destination file if it exists, so
      leftover test.output.* files from previous test suite runs are not a
      concern.  Remove the -r command line switch and the code associated with
      it from the relevant scripts.
  15. 28 Nov, 2019 2 commits
  16. 26 Nov, 2019 1 commit
  17. 20 Nov, 2019 1 commit
  18. 18 Jun, 2019 1 commit
    • Michał Kępień's avatar
      Always fail a system test if crashes are detected · 7706f229
      Michał Kępień authored
      In certain situations (e.g. a named instance crashing upon shutdown in a
      system test which involves shutting down a server and restarting it
      afterwards), a system test may succeed despite a named crash being
      triggered.  This must never be the case.  Extend run.sh to mark a test
      as failed if core dumps or log lines indicating assertion failures are
      detected (the latter is only an extra measure aimed at test environments
      in which core dumps are not generated; note that some types of crashes,
      e.g. segmentation faults, will not be detected using this method alone).
  19. 03 Dec, 2018 2 commits
  20. 13 Jul, 2018 1 commit
  21. 12 Jun, 2018 1 commit
  22. 07 Mar, 2018 1 commit
  23. 23 Feb, 2018 2 commits
  24. 22 Feb, 2018 1 commit
  25. 21 Feb, 2018 1 commit
    • Michał Kępień's avatar
      Allow retaining system test output using an environment variable · 38620438
      Michał Kępień authored
      Instead of exporting an environment variable containing a command line
      argument (NOCLEAN="-n"), extend run.sh to handle a "boolean" environment
      variable (SYSTEMTEST_NO_CLEAN) itself.  The former method is buggy
      because the value of NOCLEAN is set in parallel.mk when that file is
      first created, but it is not subsequently updated upon each test run
      (because make considers parallel.mk to be up to date).
      To retain backward compatibility, the "-n" command line argument for
      run.sh is still supported (and has a higher priority than the relevant
      environment variable).
      The SYSTEMTEST_NO_CLEAN environment variable can also be used directly
      to prevent cleanup when using "make test" instead of runall.sh.
      Apart from fixing a bug, this simplifies the way runall.sh controls
      run.sh behavior due to the Makefile being bypassed.  Direct processing
      of environment variables in run.sh is more scalable in the long run,
      given that the previously utilized technique, even with its
      implementation fixed, would still require Makefile.in to be modified in
      two places each time a new flag needed to be passed from runall.sh to
  26. 16 Feb, 2018 4 commits