Commit 32d7d27a authored by Michal Nowak's avatar Michal Nowak

Add respdiff from bind-qa repo as-is

parent a04cdde4
......@@ -40,6 +40,7 @@ stages:
- push
- postcheck
- release
- post-release
### Runner Tag Templates
......@@ -1259,3 +1260,44 @@ build:coverity:sid:amd64:
- cov-analysis-linux64.md5
- cov-analysis-linux64.tgz
# Respdiff test
<<: *debian_sid_amd64_image
stage: post-release
- *setup_interfaces
- test ! -d refbind && git clone refbind
- cd refbind/
- git checkout v9_11_3
- ./configure
- cd ..
# At least python3-lmdb in Debian sid is too old for respdiff
# (reported as
- ip -6 route del default
- apt-get -y install python3-pip
- pip3 install -r
- find . -name named -type f -exec ls -ld {} \;
- cd bin/tests/respdiff/
- pwd ; find .
- bash -x -q "${PWD}/100k_mixed.txt" -c 3 -w "${PWD}/rspworkdir" "${PWD}/../../../refbind" "${PWD}/../../.."
- gcc:sid:amd64
needs: ["gcc:sid:amd64"] # FIXME: Drop.
- tags
- web
- bin/tests/respdiff
# untracked: true
expire_in: "1 day"
# paths:
# - rspworkdir
# when: on_failure
# cache:
# key:
# paths:
# - refbind
This diff is collapsed.
This diff is collapsed. A A A A A A A A A A A A A A A A A A A A
# Response Differencing
## Introduction
respdiff is a set of programs written by CZ.NIC to send queries to different
DNS servers and to compare the responses. As the answers can be different, it
allows configuration of the comparison function to exclude the answers and only
check things like flags etc. (respdiff can be found at
The files in this directory form a parameterized Jenkins job that can be used
to run BIND configured as a recursive server through a "respdiff" test. A
reference version of BIND is selected, as is a version of BIND to test. The
reference version is specified by a tag or branch in the public respository.
The version of BIND to test can be specified by either a tag/branch in the
public repository, or as a tarball that is uploaded to the Jenkins server.
The job configures respdiff to start one test nameserver and multiple copies of
the reference version. Multiple reference nameservers are a good idea as when
querying for a name, it is quite possible to get different answers to repeated
queries. With multiple reference namservers, respdiff omits from statistics
queries for which the reference nameservers do not obtain the same answer. For
those where all reference nameservers agree, it reports queries where the
answer received from the test nameserver differs.
## Specification of Parameters
The job is controlled from the file bind9-respdiff.jenkinsfile. This is a
parameterized job; the parameters are described in the file, but the main
ones are:
**TESTBIND** - Name of the branch/tag in the BIND public repository or the name
of a tarball.
**TARBALL** - Actual name of the tarball as stored in the $HOME/tarball-queue
directory of the machine on which the Jenkins job is running.
**REFBIND** - Name of the branch or tag in the BIND public repository to use
as the reference nameservers.
The rather odd requirement for tarball location is down to a restriction in
Jenkins. Ideally this job should be a simple parameterized job, but this
[bug]( in Jenkins means that
although a file parameter can be supplied for pipeline jobs, the file is never
copied to the workspace directory. (It looks from [this
report]( that the issue is
unlikely to be resolved.) For this reason, a manual job is started by running
either of the jobs bind9-parameterized-respdiff-branchtag or
bind9-parameterized-respdiff-tarball; they obtrain the revelant information,
upload the tarball (if required), then trigger this job. This job is also
started by other, regularly scheduled, jobs.
bind9-parameterized-respdiff-tarball copies the supplied tarball to the
$HOME/tarball-queue directory (changing the name to allow for multiple queued
runs of the same tarball with different parameters) and passes that name to
this job via the TARBALL parameter. The other jobs just pass the name of the
branch to this job via the TESTBIND parameter.
To avoid a Jenkins job continually cloning the cz.nic repository, a git
repository is established on the target machine the first time the script is
run. Thereafter, on every run the repository is updated with a "git pull"
## Job Details
The bulk of the logic is in the file "". After building both the
test and reference versions of BIND, it starts the instances of BIND:
* The reference versions of BIND are started on 10.53.0.{1,2,3}. (This version
of the code starts three reference instances.)
* The test version of BIND is built and started on
Prior to starting the run of respdiff, the latest version of the software is
obtained from the CZ.NIC git repository. The repository is cloned if it doesn't
exist, or thge local copy (in $HOME/respdiff) updated if it does.
The configuration files for the run are created, the nameservers started, and
respdiff is allowed to run. When completed, the instances of BIND are stopped.
The respdiff output file is parsed and a junit XML file created with the
results. The output is fairly simplistic - if the number of miksmatches
exceeds a threshold (as described in the file, the job is
assumed to have failed.
## Files
**100k_mixed.txt, 10k_a.txt, 20a.txt**
Files containing queries that respdiff can use. The job is configured to use
the 100k_mixed.txt file.
Jenkinsfile controlling the job.
Configuration templates for the reference and test instances of BIND. They are
processed before use to set the correct address and port (and, in the case of
the test instance, any supplied configuration file options).
Configuration files for the different reference instances of BIND.
respdiff configuration file. This is processed to set specific variables
before being used.
Script that actually runs the respdiff test.
options {
listen-on-v6 { none; };
listen-on port @PORT@ { @IP@; };
directory "@DIRECTORY@";
recursion yes;
dnssec-enable yes;
dnssec-validation auto;
ip =
port = 5300
transport = udp
graph_color = red
ip =
port = 5300
transport = udp
graph_color = orange
ip =
port = 5300
transport = udp
graph_color = yellow
ip =
port = 5300
transport = udp
graph_color = green
ip =
port = 5300
transport = udp
graph_color = blue
ip =
port = 5300
transport = udp
graph_color = cyan
ip =
port = 5300
transport = udp
graph_color = purple
ip =
port = 5300
transport = udp
graph_color = red
ip =
port = 5300
transport = udp
graph_color = orange
# in seconds (float)
timeout = 5
# number of queries to run simultaneously
jobs = 16
# in seconds (float); delay each query by a random time (uniformly distributed) between min and max; set max to 0 to disable
time_delay_min = 0
time_delay_max = 0
# number of maximum consecutive timeouts received from a single resolver before exiting
max_timeouts = 10
names = @SYSTEMS@
# symbolic names of DNS servers under test
# separate multiple values by ,
# each symbolic name in [servers] section refers to config section
# containing IP address and port of particular server
ip =
port = 5300
transport = udp
graph_color = black
# symbolic name of server under test
# other servers are used as reference when comparing answers from the target
target = testbind
# fields and comparison methods used when comparing two DNS messages
criteria = opcode, rcode, flags, question, answertypes, answerrrsigs, edns
# Additional fields: authority, additional, edns, nsid
# diffsum reports mismatches in field values in this order
# if particular message has multiple mismatches, it is counted only once into category with highest weight
field_weights = timeout, malformed, opcode, question, rcode, flags, answertypes, answerrrsigs, answer, authority, additional, edns, nsid
# - Run "respdiff" on the specified version of BIND
# The script builds both versions of BIND if not already built (it does this
# by testing for "bin/named/named"). It then runs "respdiff", which is written
# by cz.nic:
usage() {
cat << EOF
Usage: [-b] [-c refcount] [-h] [-o options] [-q file] [-r respdir]
[-w workdir] refdir testdir
-b If present, build BIND in the reference and test directories.
By default, the directories are assumed to contain a built version
of BIND.
-c Number of instances of the reference implementation to run. This
must be a number between 1 and 9. By default, one reference
implementation is run.
-h Print this text and exit.
-f options Only relevant if -b is specified, gives options to be passed to
"configure" for the build of the *reference* version of BIND.
Multiple options should be separated by spaces and the entire
string enclosed in quotes.
-q file File to use for the queries. If not present, the file
"respdiff-queries.txt" in the directory holding this script is
-r respdir Local git repository holding the "respdiff" code. If not present,
\$HOME/respdiff is assumed. The repository is cloned if it
doesn't exist and is updated before each run.
-t options Only relevant if -b is specified, gives options to be passed to
"configure" for the build of the *test* version of BIND.
Multiple options should be separated by spaces and the entire
string enclosed in quotes.
-w workdir Directory in which respdiff will create its various files. If
not defined, \$HOME/rpsworkdir will be used. This directory
will is created for the run and must not exist beforehand.
refdir Directory holding the reference version of BIND. It is against this
that the test version is being compared. If a build is requested, it
is built in this directory
testdir Directory holding the version of BIND under test. If a build is
requested, it is built in this directory.
# Constants
# URL of the repository holding the respdiff code
# Functions
# build_bind - Configure and Build BIND
# Arguments:
# $1 Name of the directory holding BIND
# $2 Options for the "configure" command
build_bind() {
cd $1
./configure $2
return 0
# Message functions
errmsg() {
echo "***ERROR: $*"
infomsg() {
echo "*** INFO: $*"
# Print error message and exit.
# Invocation:
# fatal [-u] "Message"
# If -u is specified, the usage message is also printed.
fatal() {
if [ "$1" = "-u" ]; then
errmsg "$1"
if [ $print_usage -eq 1 ]; then
exit 1
# Main Code
# Process the command line.
while getopts "bc:f:hq:r:t:w:" flag; do
case "$flag" in
b) build=1 ;;
c) refcount=$OPTARG ;;
f) ref_options="$OPTARG" ;;
h) usage ; exit 0 ;;
q) queryfile=$OPTARG ;;
r) rspdir=$OPTARG ;;
t) test_options="$OPTARG" ;;
w) workdir=$OPTARG ;;
shift `expr $OPTIND - 1`
# Get directory holding the miscellaneous configuration files. This is the
# directory that holds this file.
pushd `dirname $0` > /dev/null
popd > /dev/null
# Set more meaningful names for the arguments and check that the directories
# exist.
if [ "$bindref" = "" ]; then
fatal -u "must specify the reference BIND directory"
elif [ ! -d $bindref ]; then
fatal "directory $bindref does not exist"
if [ "$bindtest" = "" ]; then
fatal -u "must specify the test BIND directory"
elif [ ! -d $bindtest ]; then
fatal "directory $bindtest does not exist"
# Ensure that the "respdiff" repository is present, creating it if it doesn't
# exist.
if [ ! -d $rspdir ]; then
infomsg "cloning respdiff repository from $RESPDIFF_URL"
git clone $RESPDIFF_URL $rspdir
git checkout f771ce6e04654357f51b18bc49a7d668dd3f47f8
infomsg "updating the respdiff repository"
cd $rspdir
git clean -fdx
git checkout master
git pull
# Check that the number of reference implementations is valid.
test "$refcount" -eq "$refcount" 2> /dev/null
if [ $? -ne 0 ]; then
fatal -u "value passed to -c must be an integer"
elif [ $refcount -lt 1 ]; then
fatal "number of instances of test server must be between 1 and 9"
elif [ $refcount -gt 9 ]; then
fatal "number of instances of test server must be between 1 and 9"
# Check that the query file exists.
if [ ! -e $queryfile ]; then
fatal "file $queryfile not found"
# Create the working directory. To avoid problems with multiple runs,
# this must not exist before the script is run.
if [ -e $workdir ]; then
fatal -u "directory $workdir already exists"
mkdir $workdir
# The working directories for the BIND runs will be subdirectories of
# the workingf directory.
mkdir $testdir
while [ $count -le $refcount ]; do
mkdir ${refdir_root}${count}
count=`expr $count + 1`
# Build the two copies of BIND if requested.
if [ $build -eq 1 ]; then
infomsg "building reference version of BIND"
build_bind $bindref "$ref_options"
infomsg "building version of BIND under test"
build_bind $bindtest "$test_options"
# Create the configuration files for the runs. These will be held
# in the working directories of each server.
infomsg "creating test config file"
sed -e "s?@DIRECTORY@?$testdir?" \
-e "s?@PORT@?$TESTPORT?" \
-e "s?@IP@?$TESTIP?" \
$configdir/ > $testdir/named.conf
while [ $count -le $refcount ]; do
infomsg "creating config file for reference instance $count"
sed -e "s?@DIRECTORY@?${refdir_root}${count}?" \
-e "s?@PORT@?${REFPORT}?" \
-e "s?@IP@?${REFIP_ROOT}${count}?" \
$configdir/ > ${refdir_root}${count}/named.conf
count=`expr $count + 1`
# Create the respdiff configuration file. This resides in the directory
# holding the various programs.
while [ $count -le $refcount ]; do
SYSTEMS="${SYSTEMS}, refbind${count}"
refconfigs="$refconfigs ${configdir}/refbind${count}.cfg"
count=`expr $count + 1`
infomsg "creating respdiff config file"
sed -e "s:@SYSTEMS@:${SYSTEMS}:" \
$configdir/ > $workdir/respdiff.cfg
cat $refconfigs >> $workdir/respdiff.cfg
# Summarise the parameters before the test starts.
infomsg "summary of run configuration on `date`"
infomsg "'respdiff' repository: $rspdir"
infomsg "working directory: $workdir"
infomsg "number of reference instances: $refcount"
infomsg "query file: $queryfile"
if [ $build -eq 1 ]; then
infomsg "config options for reference version: '${ref_options}'"
infomsg "config options for test version: '${test_options}'"
infomsg "test named version: `$bindtest/bin/named/named -v`"
infomsg "run directory: $testdir"
infomsg "associated configuration file:"
cat $testdir/named.conf
while [ $count -le $refcount ]; do
infomsg "reference named (instance ${count}) version: `$bindref/bin/named/named -v`"
infomsg "run directory: ${refdir_root}${count}"
infomsg "associated configuration file:"
cat ${refdir_root}${count}/named.conf
count=`expr $count + 1`
# Start the instances of named.
while [ $count -le $refcount ]; do
infomsg "starting instance $count of the reference version of BIND"
$bindref/bin/named/named -4 -g -c ${directory}/named.conf > ${directory}/ 2>&1 &
count=`expr $count + 1`
infomsg "starting test version of BIND"
$bindtest/bin/named/named -4 -g -c $testdir/named.conf > $testdir/ 2>&1 &
# Run the various Python scripts.
cd $rspdir
infomsg "preparing the query file: $queryfile"
./ $workdir < $queryfile
infomsg "executing queries"
./ --config $workdir/respdiff.cfg --ignore-timeout $workdir
infomsg "identifying differences"
./ --config $workdir/respdiff.cfg $workdir
infomsg "difference summary"
./ --config $workdir/respdiff.cfg $workdir | tee $workdir/report.txt
# Kill the instances of BIND. Give them time to shutdown, then really kill any
# that survive.
infomsg "shutting down BIND instances"
pids=`find $workdir -name -exec cat {} \;`
if [ "$pids" != "" ]; then
infomsg "terminating named processes having PIDs ${pids}"
kill $pids
sleep 10
pids=`find $workdir -name -exec cat {} \;`
if [ "$pids" != "" ]; then
infomsg "terminating named processes having PIDs ${pids} with EXTREME prejudice"
kill -9 $pids
options {
listen-on-v6 { none; };
listen-on port @PORT@ { @IP@; };
directory "@DIRECTORY@";
recursion yes;
dnssec-enable yes;
dnssec-validation auto;
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment