Verified Commit 9998409c authored by Michal Nowak's avatar Michal Nowak
Browse files

Add pairwise testing of configure options

Pairwise testing is a test case generation technique based on the
observation that most faults are caused by interactions of at most
two factors. When testing partition creating software those factors may
be Type, Size, Format method, File system, and Compression. For BIND
./configure, we don't have such distinct factors. We could divide
./configure options to libraries (--with-zlib) and the rest. But this
seems too artificial.

Instead I scrape all custom BIND options introduced by AS_HELP_STRING,
clean, shuffle and divide them by group of five. While treating each
quintet as a distinct factor this model is processed by the PICT tool,
which finds an effective test vector, which is then executed in
parallel.

The test vector is limited by previous artificial grouping by five
options and thus does not provide complete test coverage of ./configure
options, but when run via the 'pairwise:configure' job enough times, all
combinations will be performed.
parent 7500fb93
......@@ -1371,3 +1371,26 @@ abi-check:
only:
- main@isc-projects/bind9
- /^v9_[1-9][0-9]$/@isc-projects/bind9
# Pairwise testing of ./configure options
pairwise:configure:
<<: *base_image
<<: *default_triggering_rules
stage: build
needs:
- job: autoreconf
artifacts: true
variables:
CC: gcc
LIBSOFTHSM_PATH: "/usr/lib/softhsm/libsofthsm2.so"
script:
# XXX Move this to the images repo. perftools packages are needed for the --with-gperftools-profiler option.
- apt-get -y install parallel libgoogle-perftools-dev libgoogle-perftools4
- curl -sSL https://github.com/microsoft/pict/archive/v3.7.1.tar.gz | tar -xz && cd pict-* && make -j${BUILD_PARALLEL_JOBS:-1} V=1 && cd ..
- util/pairwise-configure-testing.sh
artifacts:
paths:
- pairwise-configure-commands.txt
- pairwise-configure-model.txt
when: on_failure
......@@ -2176,6 +2176,7 @@
./util/nanny.pl PERL 2000,2001,2004,2007,2012,2016,2018,2019,2020
./util/new-func PERL 2005,2007,2012,2016,2018,2019,2020
./util/nt-kit SH 1999,2000,2001,2004,2007,2012,2016,2018,2019,2020
./util/pairwise-configure-testing.sh SH 2020
./util/parse_tsan.py PYTHON-BIN 2019,2020
./util/run-clang-tidy X 2020
./util/spacewhack.pl PERL 2000,2001,2004,2007,2012,2016,2018,2019,2020
......
#!/bin/sh
#
# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
#
# See the COPYRIGHT file distributed with this work for additional
# information regarding copyright ownership.
set -e
if [ -z "${LIBSOFTHSM_PATH}" ]; then
echo "ERROR: Define LIBSOFTHSM_PATH variable" >&2
exit 1
fi
if ! command -v parallel >/dev/null 2>&1; then
echo "ERROR: GNU parallel not found" >&2
exit 1
fi
grep AS_HELP_STRING configure.ac | sed \
-e 's/.*\[\(--.*\)\].*/\1/' \
-e 's/=.*//' \
-e 's/\[.*//' \
-e 's/\].*//' \
-e '/--enable-fuzzing/d' \
-e 's/--with-locktype/--with-locktype=adaptive\n--with-locktype=standard/' \
-e 's|--with-pkcs11|--with-pkcs11='"${LIBSOFTHSM_PATH}"'|' \
-e '/dnsrps/d' | sort -Ru | awk -v RS='[,\n]' '{a=$0;getline b; getline c; getline d; getline e; print a,b,c,d,e}' OFS=, > pairwise-configure-model.txt
echo "INFO: Following configure option pairs are to be tested:"
tr ',' ' ' < pairwise-configure-model.txt | nl
# XXX Fix 'pict' path once is 'pict' moved to images repo
pict-*/pict pairwise-configure-model.txt 2>/dev/null | tr '\t' ' ' | sed '1d' | sed -e "s/^/cd \$(mktemp -d -p .) \&\& ..\/configure /" > pairwise-configure-commands.txt
parallel --halt now,fail=1 --progress --tmpdir . < pairwise-configure-commands.txt
echo "INFO: No configure errors found"
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment