Commit a7471134 authored by Evan Hunt's avatar Evan Hunt

[master] update to atf 0.17

3648.	[test]		Updated the ATF test framework to version 0.17.
			[RT #25627]
parent 76df835d
3648. [test] Updated the ATF test framework to version 0.17.
[RT #25627]
3647. [bug] Address a race condition when shutting down a zone.
[RT #34750]
......
......@@ -23235,7 +23235,7 @@ fi
/*) ;;
*) srcdir="../../$srcdir";;
esac
${SHELL} "${srcdir}${srcdir:+/unit/atf-src/}./configure" MISSING=: --prefix $atfdir;
${SHELL} "${srcdir}${srcdir:+/unit/atf-src/}./configure" --enable-tools --disable-shared MISSING=: --prefix $atfdir;
) ;;
"chmod":C) chmod a+x isc-config.sh doc/doxygen/doxygen-input-filter ;;
......
......@@ -3393,7 +3393,7 @@ if test "$atf" = yes; then
/*) ;;
*) srcdir="../../$srcdir";;
esac
${SHELL} "${srcdir}${srcdir:+/unit/atf-src/}./configure" MISSING=: --prefix $atfdir;
${SHELL} "${srcdir}${srcdir:+/unit/atf-src/}./configure" --enable-tools --disable-shared MISSING=: --prefix $atfdir;
) ],
[atfdir=`pwd`/unit/atf])
AC_MSG_RESULT(building ATF from bind9/unit/atf-src)
......
These unit tests for BIND 9 are based on the NetBSD Automated Test Framework
release 0.12.
release 0.17.
To build an external copy of ATF for use by BIND 9:
$ cd atf-src
$ configure --prefix=<prefix> --enable-tools --disable-shared
$ make
$ make install
Subsequently, specify the ATF prefix when building BIND 9:
$ configure --with-atf=<prefix>
ATF can also be built automatically during the BIND 9 build,
by specifying --with-atf without an argument:
$ configure --with-atf
This causes BIND 9 to build ATF in the atf-src directory and
link to it directly.
......@@ -20,6 +20,11 @@ Authors and contributors Automated Testing Framework
initial version of the atf-check utility and started the addition of the
ATF_REQUIRE family of macros in the C interface.
* Paul Goyette <pgoyette@NetBSD.org>
Implemented timestamping of test programs and test cases so that
atf-report can provide timings for their execution.
===========================================================================
vim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2
......@@ -6,6 +6,7 @@ tp: atf-c
tp: atf-c++
tp: atf-sh
tp: test-programs
tp: atf-config
tp: atf-report
tp: atf-run
tp-glob: atf-config*
tp-glob: atf-report*
tp-glob: atf-run*
......@@ -5,7 +5,7 @@ Redistribution terms Automated Testing Framework
License
*******
Copyright (c) 2007, 2008, 2009, 2010 The NetBSD Foundation, Inc.
Copyright (c) 2007, 2008, 2009, 2010, 2011, 2012 The NetBSD Foundation, Inc.
All rights reserved.
Redistribution and use in source and binary forms, with or without
......@@ -31,6 +31,36 @@ ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
Copyright 2011, 2012 Google Inc.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of Google Inc. nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
Relicensed code
***************
......
......@@ -40,16 +40,7 @@ To build and use ATF successfully you need:
* A make(1) utility.
If you are building ATF from the code on the repository, you will also need
the following tools. The versions listed here are the ones used to build
the files bundled in the last formal release, but these are not strictly
required. Newer ones will most likely work and, maybe, some slightly older
ones:
* GNU autoconf 2.68
* GNU automake 1.11.1
* GNU libtool 2.2.6b
to have GNU autoconf, automake and libtool installed.
Regenerating the build system
......@@ -186,26 +177,35 @@ script:
and work directories for test cases. This is just a default and can be
overriden at run time.
* GDB
Possible values: empty, absolute path to GNU GDB.
Default: empty.
Specifies the path to the GNU GDB binary that atf-run will use to gather
a stack trace of a crashing test program. If empty, the configure script
will try to find a suitable binary for you.
The following flags are specific to ATF's 'configure' script:
* --enable-developer
Possible values: yes, no
Default: Depends on the version number. Stable versions define this to
'no' while all others have it set to 'yes'.
Default: 'yes' in Git HEAD builds; 'no' in formal releases.
Enables several features useful for development, such as the inclusion
of debugging symbols in all objects or the enforcement of compilation
warnings.
Enables several features useful for development, such as the inclusion of
debugging symbols in all objects or the enabling of warnings during
compilation.
The compiler will be executed with an exhaustive collection of warning
detection features regardless of the value of this flag. However, such
warnings are only fatal when --enable-developer is 'yes'.
* --enable-unstable-shared
* --enable-tools
Possible values: yes, no
Default: no.
Forces the building of shared libraries in addition to static ones. The
build of shared libraries is currently disabled because their ABIs and
APIs are unstable and subject to change. This flag is provided for
development purposes only and will be removed once the libraries are
stable enough.
Enables the build of the deprecated atf-config, atf-report, atf-run
and atf-version tools. atf-report and atf-run have been superseded by
Kyua, and atf-config and atf-version are unnecessary.
Post-installation steps
......
syntax("kyuafile", 1)
test_suite("atf")
include("atf-c/Kyuafile")
include("atf-c++/Kyuafile")
include("atf-sh/Kyuafile")
include("test-programs/Kyuafile")
if fs.exists("atf-config/Kyuafile") then
include("atf-config/Kyuafile")
end
if fs.exists("atf-report/Kyuafile") then
include("atf-report/Kyuafile")
end
if fs.exists("atf-run/Kyuafile") then
include("atf-run/Kyuafile")
end
#
# Automated Testing Framework (atf)
#
# Copyright (c) 2007, 2008, 2009, 2010 The NetBSD Foundation, Inc.
# Copyright (c) 2007 The NetBSD Foundation, Inc.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
......@@ -27,6 +27,7 @@
# IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
atf_aclocal_DATA =
BUILT_SOURCES =
CLEANFILES =
EXTRA_DIST =
......@@ -38,21 +39,27 @@ libexec_PROGRAMS =
man_MANS =
noinst_DATA =
noinst_LTLIBRARIES =
INSTALLCHECK_TARGETS =
PHONY_TARGETS =
ACLOCAL_AMFLAGS = -I m4
AM_DISTCHECK_CONFIGURE_FLAGS = --enable-tools
include admin/Makefile.am.inc
include atf-c/Makefile.am.inc
include atf-c++/Makefile.am.inc
include atf-config/Makefile.am.inc
include atf-report/Makefile.am.inc
include atf-run/Makefile.am.inc
include atf-sh/Makefile.am.inc
include atf-version/Makefile.am.inc
include bootstrap/Makefile.am.inc
include doc/Makefile.am.inc
include test-programs/Makefile.am.inc
if ENABLE_TOOLS
include atf-report/Makefile.am.inc
include atf-config/Makefile.am.inc
include atf-run/Makefile.am.inc
include atf-version/Makefile.am.inc
endif
#
# Top-level distfile documents.
#
......@@ -69,10 +76,11 @@ TESTS_ENVIRONMENT = PATH=$(prefix)/bin:$${PATH} \
PKG_CONFIG_PATH=$(prefix)/lib/pkgconfig
testsdir = $(exec_prefix)/tests
pkgtestsdir = $(testsdir)/atf
pkgtestsdir = $(testsdir)/$(PACKAGE)
installcheck-local: installcheck-atf
.PHONY: installcheck-atf
if ENABLE_TOOLS
INSTALLCHECK_TARGETS += installcheck-atf
PHONY_TARGETS += installcheck-atf
installcheck-atf:
logfile=$$(pwd)/installcheck.log; \
fifofile=$$(pwd)/installcheck.fifo; \
......@@ -89,8 +97,23 @@ installcheck-atf:
"installcheck.log; exit was $${res}"; \
test $${res} -eq 0
CLEANFILES += installcheck.fifo installcheck.log
pkgtests_DATA = Atffile
endif
PHONY_TARGETS += installcheck-kyua
if HAVE_KYUA
if !ENABLE_TOOLS
INSTALLCHECK_TARGETS += installcheck-kyua
endif
installcheck-kyua:
cd $(pkgtestsdir) && $(TESTS_ENVIRONMENT) $(KYUA) test
endif
installcheck-local: $(INSTALLCHECK_TARGETS)
pkgtests_DATA = Kyuafile
if ENABLE_TOOLS
pkgtests_DATA += Atffile
endif
EXTRA_DIST += $(pkgtests_DATA)
BUILD_SH_TP = \
......@@ -103,19 +126,29 @@ BUILD_SH_TP = \
# Custom targets.
#
.PHONY: depend
depend:
.PHONY: clean-all
dist-hook: forbid-dist
if ENABLE_TOOLS
forbid-dist:
@true
else
forbid-dist:
@echo "Sorry; cannot make dist without the tools enabled."
@echo "Please reconfigure with --enable-tools."
@false
endif
PHONY_TARGETS += clean-all
clean-all:
MTN="$(MTN)" $(SH) $(srcdir)/admin/clean-all.sh
GIT="$(GIT)" $(SH) $(srcdir)/admin/clean-all.sh
.PHONY: release
PHONY_TARGETS += release
release:
$(SH) $(srcdir)/admin/release.sh $(PACKAGE_VERSION) $(DIST_ARCHIVES)
.PHONY: release-test
PHONY_TARGETS += release-test
release-test:
$(SH) $(srcdir)/admin/release-test.sh $(DIST_ARCHIVES)
.PHONY: $(PHONY_TARGETS)
# vim: syntax=make:noexpandtab:shiftwidth=8:softtabstop=8
This diff is collapsed.
......@@ -2,6 +2,187 @@ Major changes between releases Automated Testing Framework
===========================================================================
Changes in version 0.17
***********************
Experimental version released on February 14th, 2013.
* Added the atf_utils_cat_file, atf_utils_compare_file,
atf_utils_copy_file, atf_utils_create_file, atf_utils_file_exists,
atf_utils_fork, atf_utils_grep_file, atf_utils_grep_string,
atf_utils_readline, atf_utils_redirect and atf_utils_wait utility
functions to atf-c-api. Documented the already-public
atf_utils_free_charpp function.
* Added the cat_file, compare_file, copy_file, create_file, file_exists,
fork, grep_collection, grep_file, grep_string, redirect and wait
functions to the atf::utils namespace of atf-c++-api. These are
wrappers around the same functions added to the atf-c-api library.
* Added the ATF_CHECK_MATCH, ATF_CHECK_MATCH_MSG, ATF_REQUIRE_MATCH and
ATF_REQUIRE_MATCH_MSG macros to atf-c to simplify the validation of a
string against a regular expression.
* Miscellaneous fixes for manpage typos and compilation problems with
clang.
* Added caching of the results of those configure tests that rely on
executing a test program. This should help crossbuild systems by
providing a mechanism to pre-specify what the results should be.
* PR bin/45690: Make atf-report convert any non-printable characters to
a plain-text representation (matching their corresponding hexadecimal
entities) in XML output files. This is to prevent the output of test
cases from breaking xsltproc later.
Changes in version 0.16
***********************
Experimental version released on July 10th, 2012.
* Added a --enable-tools flag to configure to request the build of the
deprecated ATF tools, whose build is now disabled by default. In order
to continue running tests, you should migrate to Kyua instead of enabling
the build of the deprecated tools. The kyua-atf-compat package provides
transitional compatibility versions of atf-run and atf-report built on
top of Kyua.
* Tweaked the ATF_TEST_CASE macro of atf-c++ so that the compiler can
detect defined but unused test cases.
* PR bin/45859: Fixed some XSLT bugs that resulted in the tc-time and
tp-time XML tags leaking into the generated HTML file. Also improved
the CSS file slightly to correct alignment and color issues with the
timestamps column.
* Optimized atf-c++/macros.hpp so that GNU G++ consumes less memory during
compilation with GNU G++.
* Flipped the default to building shared libraries for atf-c and atf-c++,
and started versioning them. As a side-effect, this removes the
--enable-unstable-shared flag from configure that appears to not work any
more (under NetBSD). Additionally, some distributions require the use of
shared libraries for proper dependency tracking (e.g. Fedora), so it is
better if we do the right versioning upstream.
* Project hosting moved from an adhoc solution (custom web site and
Monotone repository) to Google Code (standard wiki and Git). ATF now
lives in a subcomponent of the Kyua project.
Changes in version 0.15
***********************
Experimental version released on January 16th, 2012.
* Respect stdin in atf-check. The previous release silenced stdin for any
processes spawned by atf, not only test programs, which caused breakage
in tests that pipe data through atf-check.
* Performance improvements to atf-sh.
* Enabled detection of unused parameters and variables in the code and
fixed all warnings.
* Changed the behavior of "developer mode". Compiler warnings are now
enabled unconditionally regardless of whether we are in developer mode or
not; developer mode is now only used to perform strict warning checks and
to enable assertions. Additionally, developer mode is now only
automatically enabled when building from the repository, not for formal
releases.
* Added new Autoconf M4 macros (ATF_ARG_WITH, ATF_CHECK_C and
ATF_CHECK_CXX) to provide a consistent way of defining a --with-arg flag
in configure scripts and detecting the presence of any of the ATF
bindings. Note that ATF_CHECK_SH was already introduced in 0.14, but it
has now been modified to also honor --with-atf if instantiated.
* Added timing support to atf-run / atf-report.
* Added support for a 'require.memory' property, to specify the minimum
amount of physical memory needed by the test case to yield valid results.
* PR bin/45690: Force an ISO-8859-1 encoding in the XML files generated by
atf-report so that invalid data in the output of test cases does not
mangle our report.
Changes in version 0.14
***********************
Experimental version released on June 14th, 2011.
* Added a pkg-config file for atf-sh and an aclocal file to ease the
detection of atf-sh from autoconf scripts.
* Made the default test case body defined by atf_sh fail. This is to
ensure that test cases are properly defined in test programs and helps
in catching typos in the names of the body functions.
* PR bin/44882: Made atf-run connect the stdin of test cases to /dev/zero.
This provides more consistent results with "normal" execution (in
particular, when tests are executed detached from a terminal).
* Made atf-run hardcode TZ=UTC for test cases. It used to undefine TZ, but
that does not take into account that libc determines the current timezone
from a configuration file.
* All test programs will now print a warning when they are not run through
atf-run(1) stating that this is unsupported and may deliver incorrect
results.
* Added support for the 'require.files' test-case property. This allows
test cases to specify installed files that must be present for the test
case to run.
Changes in version 0.13
***********************
Experimental version released on March 31st, 2011.
This is the first release after the creation of the Kyua project, a more
modular and reliable replacement for ATF. From now on, ATF will change to
accomodate the transition to this new codebase, but ATF will still continue
to see development in the short/medium term. Check out the project page at
http://code.google.com/p/kyua/ for more details.
The changes in this release are:
* Added support to run the tests with the Kyua runtime engine (kyua-cli), a
new package that aims to replace atf-run and atf-report. The ATF tests
can be run with the new system by issuing a 'make installcheck-kyua' from
the top-level directory of the project (assuming the 'kyua' binary is
available during the configuration stage of ATF).
* atf-run and atf-report are now in maintenance mode (but *not* deprecated
yet!). Kyua already implements a new, much more reliable runtime engine
that provides similar features to these tools. That said, it is not
complete yet so all development efforts should go towards it.
* If GDB is installed, atf-run dumps the stack trace of crashing test
programs in an attempt to aid debugging. Contributed by Antti Kantee.
* Reverted default timeout change in previous release and reset its value
to 5 minutes. This was causing several issues, specially when running
the existing NetBSD test suite in qemu.
* Fixed the 'match' output checker in atf-check to properly validate the
last line of a file even if it does not have a newline.
* Added the ATF_REQUIRE_IN and ATF_REQUIRE_NOT_IN macros to atf-c++ to
check for the presence (or lack thereof) of an element in a collection.
* PR bin/44176: Fixed a race condition in atf-run that would crash atf-run
when the cleanup of a test case triggered asynchronous modifications to
its work directory (e.g. killing a daemon process that cleans up a pid
file in the work directory).
* PR bin/44301: Fixed the sample XSLT file to report bogus test programs
instead of just listing them as having 0 test cases.
Changes in version 0.12
***********************
......
Things to do Automated Testing Framework
===========================================================================
Last revised: November 30th, 2010
This document includes the list of things that need to be done in ATF that
are most requested by the users. This information used to be available in
an ad-hoc bug tracker but that proved to be a bad idea. I have collected
all worthy comments in here.
Please note that most work these days is going into Kyua (see
http://code.google.com/p/kyua/). The ideas listed here apply to the
components of ATF that have *not* been migrated to the new codebase yet.
For bug reports or ideas that apply to the components that already have
been migrated, please use the bug tracker in the URL above. Similarly,
whenever a component is migrated, the ideas in this file should be revised
and migrated to the new bug tracker where appropriate.
---------------------------------------------------------------------------
Add build-time checks to atf-sh
The 0.7 release introduced build-time tests to atf-c and atf-c++, but not
to atf-sh. Expose the functionality to the shell interface.
This will probably require writing an atf-build utility that exposes the C
code and can be called from the shell.
---------------------------------------------------------------------------
Revisit what to do when an Atffile lists a non-existent file
---------------------------------------------------------------------------
Add ATF_CHECK* versions to atf-c++ to support non-fatal tests
---------------------------------------------------------------------------
Implement race-condition tests
gcooper:
I would think that stress/negative tests would be of more value than race
condition tests (they're similar, but not exactly the same in my mind).
In particular,