GitLab

The system tests currently uses patchwork of shell scripts which doesn't
offer proper error handling.

This commit introduced option to write new tests in pytest framework
that also allows easier manipulation of DNS traffic (using dnspython),
native XML and JSON manipulation and proper error reporting.

Add recursive "test" and "unit" rules, which execute "make check"
in specific directories - "make test" runs the system tests, and
"make unit" runs the unit tests.

There are several improvements over the default/previous behaviour of
the test log driver and log compiler:

 * The system-test-driver.sh was dropped (it was used incorrectly)
 * The run.sh script is now both log compiler and cli script to run
   individual tests
 * The custom-test-driver was added as extended version of the automake
   test-driver with capability to tee the test output to stdout when
   `--verbose yes` is passed to it (you can use LOG_DRIVER_FLAGS to
   add the option by default)
 * Makefile.am has been extended to honor V=1 for the system tests
   test-driver (e.g. V=1 adds `--verbose yes` to AM_LOG_DRIVER_FLAGS)

fstrm_capture is not an essential utility, but its corresponding
Makefile token needs to substituted even if it is not found in PATH or
else the "dnstap" system test will consistently fail.

The rewrite of BIND 9 build system is a large work and cannot be reasonable
split into separate merge requests.  Addition of the automake has a positive
effect on the readability and maintainability of the build system as it is more
declarative, it allows conditional and we are able to drop all of the custom
make code that BIND 9 developed over the years to overcome the deficiencies of
autoconf + custom Makefile.in files.

This squashed commit contains following changes:

- conversion (or rather fresh rewrite) of all Makefile.in files to Makefile.am
  by using automake

- the libtool is now properly integrated with automake (the way we used it
  was rather hackish as the only official way how to use libtool is via
  automake

- the dynamic module loading was rewritten from a custom patchwork to libtool's
  libltdl (which includes the patchwork to support module loading on different
  systems internally)

- conversion of the unit test executor from kyua to automake parallel driver

- conversion of the system test executor from custom make/shell to automake
  parallel driver

- The GSSAPI has been refactored, the custom SPNEGO on the basis that
  all major KRB5/GSSAPI (mit-krb5, heimdal and Windows) implementations
  support SPNEGO mechanism.

- The various defunct tests from bin/tests have been removed:
  bin/tests/optional and bin/tests/pkcs11

- The text files generated from the MD files have been removed, the
  MarkDown has been designed to be readable by both humans and computers

- The xsl header is now generated by a simple sed command instead of
  perl helper

- The <irs/platform.h> header has been removed

- cleanups of configure.ac script to make it more simpler, addition of multiple
  macros (there's still work to be done though)

- the tarball can now be prepared with `make dist`

- the system tests are partially able to run in oot build

Here's a list of unfinished work that needs to be completed in subsequent merge
requests:

- `make distcheck` doesn't yet work (because of system tests oot run is not yet
  finished)

- documentation is not yet built, there's a different merge request with docbook
  to sphinx-build rst conversion that needs to be rebased and adapted on top of
  the automake

- msvc build is non functional yet and we need to decide whether we will just
  cross-compile bind9 using mingw-w64 or fix the msvc build

- contributed dlz modules are not included neither in the autoconf nor automake

With the introduction of dnssec-policy, the aforementioned tools were
either rendered obsolete, or they will be replaced with dnssec-policy
based tools.  Remove the tools and the requirement to have Python
installed.  Python 3 is still being used for tests, so keep the autoconf
test, but make it much simpler.

Revert the change from ad03c22e as
further testing has shown that with hyper-threading disabled, named with
ISC rwlocks outperforms named with pthread rwlocks in cold cache testing
scenarios.  Since building named with pthread rwlocks might still be a
better choice for some workloads, keep the compile-time option which
enables that.

The previous commit removed the code related to the internal symbol
table.  On platforms where available, we can now use backtrace_symbols()
to print more verbose symbols table to the output.

As there's now general availability of backtrace() and
backtrace_symbols() functions (see below), the commit also removes the
usage of glibc internals and the custom stack tracing.

* backtrace(), backtrace_symbols(), and backtrace_symbols_fd() are
  provided in glibc since version 2.1.
* backtrace(), backtrace_symbols(), and backtrace_symbols_fd() first
  appeared in Mac OS X 10.5.
* The backtrace() library of functions first appeared in NetBSD 7.0 and
  FreeBSD 10.0.

libtool is a requirement to use automake (see GL #4), so this commit
removes the ability to compile BIND 9 without libtool.

The internal symbol table cannot be generated when libtool is in use,
which is going to be a mandatory in the next commit.

- the configuration summary reported zlib compression was not
  supported even when it was.
- when bind.keys.h was regenerated it violated clang-format style.

and implement a custom glob() function on Windows systems.

The change introduced by commit be159f55
was not fully complete.  Adjust ./configure summary so that it reflects
the new way the --with-tuning switch works, fixing the Autoconf variable
used for determining the value of that switch.  Fix win32utils/Configure
so that it behaves the same way as its Unix counterpart.

We were using our own versions of isc_uv_{export,import} functions
for multithreaded TCP listeners. Upcoming libuv version will
contain proper uv_{export,import} functions - use them if they're
available.

This might be removed when we stop supporting older libuv versions.

The new ISC_THREAD_LOCAL macro unifies usage of platform dependent
Thread Local Storage definition thread_local vs __thread vs
__declspec(thread) to a single macro.

The commit also unifies the required level of support for TLS as for
some parts of the code it was mandatory and for some parts of the code
it wasn't.

-Wl,-z,interpose is not supported.
-Wl,rpath=<path> is not supported use -Wl,rpath,<path> instead.
Use @SO@ for loadable extension.
Use -L <path> -l libwrap instead of libwrap.sa.

This is a replacement for the existing isc_socket and isc_socketmgr
implementation. It uses libuv for asynchronous network communication;
"networker" objects will be distributed across worker threads reading
incoming packets and sending them for processing.

UDP listener sockets automatically create an array of "child" sockets
so each worker can listen separately.

TCP sockets are shared amongst worker threads.

A TCPDNS socket is a wrapper around a TCP socket, which handles the
the two-byte length field at the beginning of DNS messages over TCP.

(Other wrapper socket types can be implemented in the future to handle
DNS over TLS, DNS over HTTPS, etc.)

this is a lockless queue based on hazard pointers.

Commit afa81ee4 omitted some spots in
the source tree which are still referencing the removed --with-cc-alg
"configure" option.  Make sure the latter is removed completely.

`/usr/share/sgml/docbook/xsl-stylesheets` and `/usr/share/dblatex` are
places where docbook-style-xsl and, respectively, dblatex packages on
Red Hat systems put their XSL templates. Unless we hint this place it
has to be added to `./configure` manually (`--with-docbook-xsl=...`):
https://src.fedoraproject.org/rpms/bind/blob/master/f/bind.spec#_691.

On Fedora 30:

Before
```
./configure
...
checking for Docbook-XSL path... auto
checking for html/docbook.xsl... "not found"
checking for xhtml/docbook.xsl... "not found"
checking for manpages/docbook.xsl... "not found"
checking for html/chunk.xsl... "not found"
checking for xhtml/chunk.xsl... "not found"
checking for html/chunktoc.xsl... "not found"
checking for xhtml/chunktoc.xsl... "not found"
checking for html/maketoc.xsl... "not found"
checking for xhtml/maketoc.xsl... "not found"
checking for xsl/docbook.xsl... "not found"
checking for xsl/latex_book_fast.xsl... "not found"
```

After:
```
./configure
...
checking for Docbook-XSL path... auto
checking for html/docbook.xsl... /usr/share/sgml/docbook/xsl-stylesheets/html/docbook.xsl
checking for xhtml/docbook.xsl... /usr/share/sgml/docbook/xsl-stylesheets/xhtml/docbook.xsl
checking for manpages/docbook.xsl... /usr/share/sgml/docbook/xsl-stylesheets/manpages/docbook.xsl
checking for html/chunk.xsl... /usr/share/sgml/docbook/xsl-stylesheets/html/chunk.xsl
checking for xhtml/chunk.xsl... /usr/share/sgml/docbook/xsl-stylesheets/xhtml/chunk.xsl
checking for html/chunktoc.xsl... /usr/share/sgml/docbook/xsl-stylesheets/html/chunktoc.xsl
checking for xhtml/chunktoc.xsl... /usr/share/sgml/docbook/xsl-stylesheets/xhtml/chunktoc.xsl
checking for html/maketoc.xsl... /usr/share/sgml/docbook/xsl-stylesheets/html/maketoc.xsl
checking for xhtml/maketoc.xsl... /usr/share/sgml/docbook/xsl-stylesheets/xhtml/maketoc.xsl
checking for xsl/docbook.xsl... /usr/share/dblatex/xsl/docbook.xsl
checking for xsl/latex_book_fast.xsl... /usr/share/dblatex/xsl/latex_book_fast.xsl
```