[master] update to atf 0.17

3648.	[test]		Updated the ATF test framework to version 0.17.
			[RT #25627]

CHANGES

+3648.	[test]		Updated the ATF test framework to version 0.17.
+			[RT #25627]
+
 3647.	[bug]		Address a race condition when shutting down a zone.
 			[RT #34750]
 

configure

@@ -23235,7 +23235,7 @@ fi
 		 /*) ;;
 		 *) srcdir="../../$srcdir";;
 		 esac
-		 ${SHELL} "${srcdir}${srcdir:+/unit/atf-src/}./configure" MISSING=: --prefix $atfdir;
+		 ${SHELL} "${srcdir}${srcdir:+/unit/atf-src/}./configure" --enable-tools --disable-shared MISSING=: --prefix $atfdir;
 		)  ;;
     "chmod":C) chmod a+x isc-config.sh doc/doxygen/doxygen-input-filter ;;

configure.in

@@ -3393,7 +3393,7 @@ if test "$atf" = yes; then
 		 /*) ;;
 		 *) srcdir="../../$srcdir";;
 		 esac
-		 ${SHELL} "${srcdir}${srcdir:+/unit/atf-src/}./configure" MISSING=: --prefix $atfdir;
+		 ${SHELL} "${srcdir}${srcdir:+/unit/atf-src/}./configure" --enable-tools --disable-shared MISSING=: --prefix $atfdir;
 		) ],
 		[atfdir=`pwd`/unit/atf])
 	AC_MSG_RESULT(building ATF from bind9/unit/atf-src)

unit/README

 These unit tests for BIND 9 are based on the NetBSD Automated Test Framework 
-release 0.12.
+release 0.17.
+
+To build an external copy of ATF for use by BIND 9:
+
+  $ cd atf-src
+  $ configure --prefix=<prefix> --enable-tools --disable-shared
+  $ make
+  $ make install
+
+Subsequently, specify the ATF prefix when building BIND 9:
+
+  $ configure --with-atf=<prefix>
+
+ATF can also be built automatically during the BIND 9 build,
+by specifying --with-atf without an argument:
+
+  $ configure --with-atf
+
+This causes BIND 9 to build ATF in the atf-src directory and
+link to it directly.

unit/atf-src/AUTHORS

@@ -20,6 +20,11 @@ Authors and contributors                        Automated Testing Framework
   initial version of the atf-check utility and started the addition of the
   ATF_REQUIRE family of macros in the C interface.
 
+* Paul Goyette <pgoyette@NetBSD.org>
+
+  Implemented timestamping of test programs and test cases so that
+  atf-report can provide timings for their execution.
+
 
 ===========================================================================
 vim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2

unit/atf-src/Atffile

@@ -6,6 +6,7 @@ tp: atf-c
 tp: atf-c++
 tp: atf-sh
 tp: test-programs
-tp: atf-config
-tp: atf-report
-tp: atf-run
+
+tp-glob: atf-config*
+tp-glob: atf-report*
+tp-glob: atf-run*

unit/atf-src/COPYING

@@ -5,7 +5,7 @@ Redistribution terms                            Automated Testing Framework
 License
 *******
 
-Copyright (c) 2007, 2008, 2009, 2010 The NetBSD Foundation, Inc.
+Copyright (c) 2007, 2008, 2009, 2010, 2011, 2012 The NetBSD Foundation, Inc.
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without
@@ -31,6 +31,36 @@ ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 POSSIBILITY OF SUCH DAMAGE.
 
 
+Copyright 2011, 2012 Google Inc.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright
+  notice, this list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright
+  notice, this list of conditions and the following disclaimer in the
+  documentation and/or other materials provided with the distribution.
+
+* Neither the name of Google Inc. nor the names of its contributors
+  may be used to endorse or promote products derived from this software
+  without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+
+
 Relicensed code
 ***************
 

unit/atf-src/INSTALL

@@ -40,16 +40,7 @@ To build and use ATF successfully you need:
 * A make(1) utility.
 
 If you are building ATF from the code on the repository, you will also need
-the following tools.  The versions listed here are the ones used to build
-the files bundled in the last formal release, but these are not strictly
-required.  Newer ones will most likely work and, maybe, some slightly older
-ones:
-
-* GNU autoconf 2.68
-
-* GNU automake 1.11.1
-
-* GNU libtool 2.2.6b
+to have GNU autoconf, automake and libtool installed.
 
 
 Regenerating the build system
@@ -186,26 +177,35 @@ script:
   and work directories for test cases.  This is just a default and can be
   overriden at run time.
 
+* GDB
+  Possible values: empty, absolute path to GNU GDB.
+  Default: empty.
+
+  Specifies the path to the GNU GDB binary that atf-run will use to gather
+  a stack trace of a crashing test program.  If empty, the configure script
+  will try to find a suitable binary for you.
+
 The following flags are specific to ATF's 'configure' script:
 
 * --enable-developer
   Possible values: yes, no
-  Default: Depends on the version number.  Stable versions define this to
-  'no' while all others have it set to 'yes'.
+  Default: 'yes' in Git HEAD builds; 'no' in formal releases.
+
+  Enables several features useful for development, such as the inclusion
+  of debugging symbols in all objects or the enforcement of compilation
+  warnings.
 
-  Enables several features useful for development, such as the inclusion of
-  debugging symbols in all objects or the enabling of warnings during
-  compilation.
+  The compiler will be executed with an exhaustive collection of warning
+  detection features regardless of the value of this flag.  However, such
+  warnings are only fatal when --enable-developer is 'yes'.
 
-* --enable-unstable-shared
+* --enable-tools
   Possible values: yes, no
   Default: no.
 
-  Forces the building of shared libraries in addition to static ones.  The
-  build of shared libraries is currently disabled because their ABIs and
-  APIs are unstable and subject to change.  This flag is provided for
-  development purposes only and will be removed once the libraries are
-  stable enough.
+  Enables the build of the deprecated atf-config, atf-report, atf-run
+  and atf-version tools.  atf-report and atf-run have been superseded by
+  Kyua, and atf-config and atf-version are unnecessary.
 
 
 Post-installation steps

unit/atf-src/Kyuafile

+syntax("kyuafile", 1)
+
+test_suite("atf")
+
+include("atf-c/Kyuafile")
+include("atf-c++/Kyuafile")
+include("atf-sh/Kyuafile")
+include("test-programs/Kyuafile")
+
+if fs.exists("atf-config/Kyuafile") then
+   include("atf-config/Kyuafile")
+end
+if fs.exists("atf-report/Kyuafile") then
+   include("atf-report/Kyuafile")
+end
+if fs.exists("atf-run/Kyuafile") then
+   include("atf-run/Kyuafile")
+end

unit/atf-src/Makefile.am

 #
 # Automated Testing Framework (atf)
 #
-# Copyright (c) 2007, 2008, 2009, 2010 The NetBSD Foundation, Inc.
+# Copyright (c) 2007 The NetBSD Foundation, Inc.
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -27,6 +27,7 @@
 # IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 #
 
+atf_aclocal_DATA =
 BUILT_SOURCES =
 CLEANFILES =
 EXTRA_DIST =
@@ -38,21 +39,27 @@ libexec_PROGRAMS =
 man_MANS =
 noinst_DATA =
 noinst_LTLIBRARIES =
+INSTALLCHECK_TARGETS =
+PHONY_TARGETS =
 
 ACLOCAL_AMFLAGS = -I m4
+AM_DISTCHECK_CONFIGURE_FLAGS = --enable-tools
 
 include admin/Makefile.am.inc
 include atf-c/Makefile.am.inc
 include atf-c++/Makefile.am.inc
-include atf-config/Makefile.am.inc
-include atf-report/Makefile.am.inc
-include atf-run/Makefile.am.inc
 include atf-sh/Makefile.am.inc
-include atf-version/Makefile.am.inc
 include bootstrap/Makefile.am.inc
 include doc/Makefile.am.inc
 include test-programs/Makefile.am.inc
 
+if ENABLE_TOOLS
+include atf-report/Makefile.am.inc
+include atf-config/Makefile.am.inc
+include atf-run/Makefile.am.inc
+include atf-version/Makefile.am.inc
+endif
+
 #
 # Top-level distfile documents.
 #
@@ -69,10 +76,11 @@ TESTS_ENVIRONMENT = PATH=$(prefix)/bin:$${PATH} \
                     PKG_CONFIG_PATH=$(prefix)/lib/pkgconfig
 
 testsdir = $(exec_prefix)/tests
-pkgtestsdir = $(testsdir)/atf
+pkgtestsdir = $(testsdir)/$(PACKAGE)
 
-installcheck-local: installcheck-atf
-.PHONY: installcheck-atf
+if ENABLE_TOOLS
+INSTALLCHECK_TARGETS += installcheck-atf
+PHONY_TARGETS += installcheck-atf
 installcheck-atf:
 	logfile=$$(pwd)/installcheck.log; \
 	fifofile=$$(pwd)/installcheck.fifo; \
@@ -89,8 +97,23 @@ installcheck-atf:
 	     "installcheck.log; exit was $${res}"; \
 	test $${res} -eq 0
 CLEANFILES += installcheck.fifo installcheck.log
-
-pkgtests_DATA = Atffile
+endif
+
+PHONY_TARGETS += installcheck-kyua
+if HAVE_KYUA
+if !ENABLE_TOOLS
+INSTALLCHECK_TARGETS += installcheck-kyua
+endif
+installcheck-kyua:
+	cd $(pkgtestsdir) && $(TESTS_ENVIRONMENT) $(KYUA) test
+endif
+
+installcheck-local: $(INSTALLCHECK_TARGETS)
+
+pkgtests_DATA = Kyuafile
+if ENABLE_TOOLS
+pkgtests_DATA += Atffile
+endif
 EXTRA_DIST += $(pkgtests_DATA)
 
 BUILD_SH_TP = \
@@ -103,19 +126,29 @@ BUILD_SH_TP = \
 # Custom targets.
 #
 
-.PHONY:	depend
-depend:
-
-.PHONY: clean-all
+dist-hook: forbid-dist
+if ENABLE_TOOLS
+forbid-dist:
+	@true
+else
+forbid-dist:
+	@echo "Sorry; cannot make dist without the tools enabled."
+	@echo "Please reconfigure with --enable-tools."
+	@false
+endif
+
+PHONY_TARGETS += clean-all
 clean-all:
-	MTN="$(MTN)" $(SH) $(srcdir)/admin/clean-all.sh
+	GIT="$(GIT)" $(SH) $(srcdir)/admin/clean-all.sh
 
-.PHONY: release
+PHONY_TARGETS += release
 release:
 	$(SH) $(srcdir)/admin/release.sh $(PACKAGE_VERSION) $(DIST_ARCHIVES)
 
-.PHONY: release-test
+PHONY_TARGETS += release-test
 release-test:
 	$(SH) $(srcdir)/admin/release-test.sh $(DIST_ARCHIVES)
 
+.PHONY: $(PHONY_TARGETS)
+
 # vim: syntax=make:noexpandtab:shiftwidth=8:softtabstop=8

unit/atf-src/NEWS

@@ -2,6 +2,187 @@ Major changes between releases                  Automated Testing Framework
 ===========================================================================
 
 
+Changes in version 0.17
+***********************
+
+Experimental version released on February 14th, 2013.
+
+* Added the atf_utils_cat_file, atf_utils_compare_file,
+  atf_utils_copy_file, atf_utils_create_file, atf_utils_file_exists,
+  atf_utils_fork, atf_utils_grep_file, atf_utils_grep_string,
+  atf_utils_readline, atf_utils_redirect and atf_utils_wait utility
+  functions to atf-c-api.  Documented the already-public
+  atf_utils_free_charpp function.
+
+* Added the cat_file, compare_file, copy_file, create_file, file_exists,
+  fork, grep_collection, grep_file, grep_string, redirect and wait
+  functions to the atf::utils namespace of atf-c++-api.  These are
+  wrappers around the same functions added to the atf-c-api library.
+
+* Added the ATF_CHECK_MATCH, ATF_CHECK_MATCH_MSG, ATF_REQUIRE_MATCH and
+  ATF_REQUIRE_MATCH_MSG macros to atf-c to simplify the validation of a
+  string against a regular expression.
+
+* Miscellaneous fixes for manpage typos and compilation problems with
+  clang.
+
+* Added caching of the results of those configure tests that rely on
+  executing a test program.  This should help crossbuild systems by
+  providing a mechanism to pre-specify what the results should be.
+
+* PR bin/45690: Make atf-report convert any non-printable characters to
+  a plain-text representation (matching their corresponding hexadecimal
+  entities) in XML output files.  This is to prevent the output of test
+  cases from breaking xsltproc later.
+
+
+Changes in version 0.16
+***********************
+
+Experimental version released on July 10th, 2012.
+
+* Added a --enable-tools flag to configure to request the build of the
+  deprecated ATF tools, whose build is now disabled by default.  In order
+  to continue running tests, you should migrate to Kyua instead of enabling
+  the build of the deprecated tools.  The kyua-atf-compat package provides
+  transitional compatibility versions of atf-run and atf-report built on
+  top of Kyua.
+
+* Tweaked the ATF_TEST_CASE macro of atf-c++ so that the compiler can
+  detect defined but unused test cases.
+
+* PR bin/45859: Fixed some XSLT bugs that resulted in the tc-time and
+  tp-time XML tags leaking into the generated HTML file.  Also improved
+  the CSS file slightly to correct alignment and color issues with the
+  timestamps column.
+
+* Optimized atf-c++/macros.hpp so that GNU G++ consumes less memory during
+  compilation with GNU G++.
+
+* Flipped the default to building shared libraries for atf-c and atf-c++,
+  and started versioning them.  As a side-effect, this removes the
+  --enable-unstable-shared flag from configure that appears to not work any
+  more (under NetBSD).  Additionally, some distributions require the use of
+  shared libraries for proper dependency tracking (e.g. Fedora), so it is
+  better if we do the right versioning upstream.
+
+* Project hosting moved from an adhoc solution (custom web site and
+  Monotone repository) to Google Code (standard wiki and Git).  ATF now
+  lives in a subcomponent of the Kyua project.
+
+
+Changes in version 0.15
+***********************
+
+Experimental version released on January 16th, 2012.
+
+* Respect stdin in atf-check.  The previous release silenced stdin for any
+  processes spawned by atf, not only test programs, which caused breakage
+  in tests that pipe data through atf-check.
+
+* Performance improvements to atf-sh.
+
+* Enabled detection of unused parameters and variables in the code and
+  fixed all warnings.
+
+* Changed the behavior of "developer mode".  Compiler warnings are now
+  enabled unconditionally regardless of whether we are in developer mode or
+  not; developer mode is now only used to perform strict warning checks and
+  to enable assertions.  Additionally, developer mode is now only
+  automatically enabled when building from the repository, not for formal
+  releases.
+
+* Added new Autoconf M4 macros (ATF_ARG_WITH, ATF_CHECK_C and
+  ATF_CHECK_CXX) to provide a consistent way of defining a --with-arg flag
+  in configure scripts and detecting the presence of any of the ATF
+  bindings.  Note that ATF_CHECK_SH was already introduced in 0.14, but it
+  has now been modified to also honor --with-atf if instantiated.
+
+* Added timing support to atf-run / atf-report.
+
+* Added support for a 'require.memory' property, to specify the minimum
+  amount of physical memory needed by the test case to yield valid results.
+
+* PR bin/45690: Force an ISO-8859-1 encoding in the XML files generated by
+  atf-report so that invalid data in the output of test cases does not
+  mangle our report.
+
+
+Changes in version 0.14
+***********************
+
+Experimental version released on June 14th, 2011.
+
+* Added a pkg-config file for atf-sh and an aclocal file to ease the
+  detection of atf-sh from autoconf scripts.
+
+* Made the default test case body defined by atf_sh fail.  This is to
+  ensure that test cases are properly defined in test programs and helps
+  in catching typos in the names of the body functions.
+
+* PR bin/44882: Made atf-run connect the stdin of test cases to /dev/zero.
+  This provides more consistent results with "normal" execution (in
+  particular, when tests are executed detached from a terminal).
+
+* Made atf-run hardcode TZ=UTC for test cases.  It used to undefine TZ, but
+  that does not take into account that libc determines the current timezone
+  from a configuration file.
+
+* All test programs will now print a warning when they are not run through
+  atf-run(1) stating that this is unsupported and may deliver incorrect
+  results.
+
+* Added support for the 'require.files' test-case property.  This allows
+  test cases to specify installed files that must be present for the test
+  case to run.
+
+
+Changes in version 0.13
+***********************
+
+Experimental version released on March 31st, 2011.
+
+This is the first release after the creation of the Kyua project, a more
+modular and reliable replacement for ATF.  From now on, ATF will change to
+accomodate the transition to this new codebase, but ATF will still continue
+to see development in the short/medium term.  Check out the project page at
+http://code.google.com/p/kyua/ for more details.
+
+The changes in this release are:
+
+* Added support to run the tests with the Kyua runtime engine (kyua-cli), a
+  new package that aims to replace atf-run and atf-report.  The ATF tests
+  can be run with the new system by issuing a 'make installcheck-kyua' from
+  the top-level directory of the project (assuming the 'kyua' binary is
+  available during the configuration stage of ATF).
+
+* atf-run and atf-report are now in maintenance mode (but *not* deprecated
+  yet!).  Kyua already implements a new, much more reliable runtime engine
+  that provides similar features to these tools.  That said, it is not
+  complete yet so all development efforts should go towards it.
+
+* If GDB is installed, atf-run dumps the stack trace of crashing test
+  programs in an attempt to aid debugging.  Contributed by Antti Kantee.
+
+* Reverted default timeout change in previous release and reset its value
+  to 5 minutes.  This was causing several issues, specially when running
+  the existing NetBSD test suite in qemu.
+
+* Fixed the 'match' output checker in atf-check to properly validate the
+  last line of a file even if it does not have a newline.
+
+* Added the ATF_REQUIRE_IN and ATF_REQUIRE_NOT_IN macros to atf-c++ to
+  check for the presence (or lack thereof) of an element in a collection.
+
+* PR bin/44176: Fixed a race condition in atf-run that would crash atf-run
+  when the cleanup of a test case triggered asynchronous modifications to
+  its work directory (e.g. killing a daemon process that cleans up a pid
+  file in the work directory).
+
+* PR bin/44301: Fixed the sample XSLT file to report bogus test programs
+  instead of just listing them as having 0 test cases.
+
+
 Changes in version 0.12
 ***********************
 

unit/atf-src/TODO

+Things to do                                    Automated Testing Framework
+===========================================================================
+
+
+Last revised: November 30th, 2010
+
+
+This document includes the list of things that need to be done in ATF that
+are most requested by the users. This information used to be available in
+an ad-hoc bug tracker but that proved to be a bad idea. I have collected
+all worthy comments in here.
+
+Please note that most work these days is going into Kyua (see
+http://code.google.com/p/kyua/). The ideas listed here apply to the
+components of ATF that have *not* been migrated to the new codebase yet.
+For bug reports or ideas that apply to the components that already have
+been migrated, please use the bug tracker in the URL above. Similarly,
+whenever a component is migrated, the ideas in this file should be revised
+and migrated to the new bug tracker where appropriate.
+
+
+---------------------------------------------------------------------------
+Add build-time checks to atf-sh
+
+The 0.7 release introduced build-time tests to atf-c and atf-c++, but not
+to atf-sh. Expose the functionality to the shell interface.
+
+This will probably require writing an atf-build utility that exposes the C
+code and can be called from the shell.
+
+---------------------------------------------------------------------------
+Revisit what to do when an Atffile lists a non-existent file
+
+---------------------------------------------------------------------------
+Add ATF_CHECK* versions to atf-c++ to support non-fatal tests
+
+---------------------------------------------------------------------------
+Implement race-condition tests
+
+gcooper:
+
+I would think that stress/negative tests would be of more value than race
+condition tests (they're similar, but not exactly the same in my mind).
+
+In particular,
+
+1. Feed through as much data as possible to determine where reporting
+   breaks down.
+2. Feed through data quickly and terminate ASAP. The data should be
+   captured. Terminate child applications with unexpected exit codes and
+   signals (in particular, SIGCHLD, SIGPIPE, exit codes that terminate,
+   etc).
+3. Open up a file descriptor in the test application, don't close the file
+   descriptor.
+4. fork(2) a process; don't wait(2) for the application to complete.
+
+There are other scenarios that could be exercised, but these are the ones
+I could think of off the topic of my head.
+
+--
+
+jmmv:
+
+1. The thing is: how do you express any of this in a portable/abstract
+   interface? How do you express that a test case "receives data"? What
+   does that exactly mean? I don't think the framework should care about
+   this: each test should be free to decide where its data is and how to
+   deal with it.
+
+2. Ditto.
+
+3. Not sure I understand your request, but testing for "unexpected exit
+   codes" is already supported. See wiki:DesignXFail for the feature
+   design details.
+
+4. What's the problem with this case? The test case exits right away after
+   terminating the execution of its body; any open file descriptors,
+   leaked memory, etc. die with it.
+
+5. forking and not waiting for a subprocess was a problem already
+   addressed.
+
+I kinda have an idea of what Antti means with "race condition tests", but
+every time I have tried to describe my understanding of matters I seem to
+be wrong. Would be nice to have a clear description of what this involves;
+in particular, what are the expectations from the framework and how should
+the feature be exposed.
+
+As of now, what I understand by "race condition test" is: a test case that
+exercises a race condition. The test case may finish without triggering
+the race, in which case it just exists with a successful status.
+Otherwise, if the race triggers, the test case gets stuck and times out.
+The result should be reported as an "expected failure" different from
+timeout.
+
+--
+
+pooka:
+
+Yup. Plus some atf-wide mechanism for the operator to supply some kind of
+guideline if the test should try to trigger the race for a second or for
+an hour.
+
+--
+
+jmmv:
+
+Alright. While mocking up some code for this, I think that your two
+requests are complementary.
+
+On the one hand, when you are talking about a "race condition" test you
+really mean an "expected race condition" test. Correct? If so, we need to
+extend the xfail mechanism to add one more case, which is to report any
+failures as a race condition error and, if there is no failure, report the
+test as successful.
+
+On the other hand, the atf-wide mechanism to support how long the test
+should run for can be thought as a "stress test" mechanism. I.e. run this
+test for X time / iterations and report its results regularly without
+involving xfail at all.
+
+So, with this in mind:
+
+* For a test that triggers an unfixed race condition, you set xfail to
+  race mode and define the test as a stress test. Any failures are
+  reported as expected failures.
+
+* For a test that verifies a supposedly-fixed race condition, you do *not*
+  set xfail to race mode, and only set the test to stress test. Any
+  failures are reported as real failures.
+