All tests for "make test" returning "ERROR" even though logs show most tests passed
Summary
When I run "make test" after a successful configure/make, all of the tests throw "ERROR", but if you look at the log for each test or the xml output, it will show as passed.
BIND version affected
bind-9.18.21
Steps to reproduce
% cat /etc/redhat-release
Red Hat Enterprise Linux release 8.9 (Ootpa)
% cat /etc/oracle-release
Oracle Linux Server release 8.9
# Install required packages (though many of these are not listed as required on the BIND site! Namely the python RPMs.)
sudo yum install automake autoconf libtool libcap-devel python3-pytest python3 perl-Net-DNS python3-dns
# Download and install jemalloc RPMs from EPEL:
wget https://dl.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/j/jemalloc-5.2.1-2.el8.x86_64.rpm
wget https://dl.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/j/jemalloc-devel-5.2.1-2.el8.x86_64.rpm
sudo yum install jemalloc-5.2.1-2.el8.x86_64.rpm jemalloc-devel-5.2.1-2.el8.x86_64.rpmBuild accessory software as needed:
wget https://www.zlib.net/zlib-1.3.1.tar.gz
tar xzf zlib-1.3.1.tar.gz
cd zlib-1.3.1
./configure --prefix=/usr/local/zlib-1.3.1
make
make test
sudo make install
cd ../
sudo vi /etc/ld.so.conf.d/bind-libs.conf
# add /usr/local/zlib-1.3.1/lib
sudo ldconfig
wget https://www.openssl.org/source/openssl-3.0.13.tar.gz
tar xzf openssl-3.0.13.tar.gz
cd openssl-3.0.13
./config --prefix=/usr/local/openssl-3.0.13 --with-zlib-lib=/usr/local/zlib-1.3.1/lib
make
make test
sudo make install
cd ../
sudo vi /etc/ld.so.conf.d/bind-libs.conf
# update to /usr/local/openssl-3.0.13/lib64
sudo ldconfig
# Probably could use libuv RPM here but chose to get newest source
wget --no-check-certificate https://dist.libuv.org/dist/v1.47.0/libuv-v1.47.0.tar.gz
tar xzf libuv-v1.47.0.tar.gz
cd libuv-v1.47.0
sh autogen.sh
./configure --prefix=/usr/local/libuv-1.47.0
make
make check
sudo make install
cd ../
sudo vi /etc/ld.so.conf.d/bind-libs.conf
# add: /usr/local/libuv-1.47.0/lib
sudo ldconfig
# Now build BIND itself
wget https://downloads.isc.org/isc/bind9/9.18.21/bind-9.18.21.tar.xz
tar xJf bind-9.18.21.tar.xz
cd bind-9.18.21
export PKG_CONFIG_PATH=/usr/local/openssl-3.0.13/lib64/pkgconfig:/usr/local/zlib-1.3.1/lib/pkgconfig:/usr/local/libuv-1.47.0/lib/pkgconfig
./configure --prefix=/usr/local/bind-9.18.21 --mandir=/usr/local/bind-9.18.21/man --with-openssl=/usr/local/openssl-3.0.13 --with-jemalloc --with-gnu-ld --with-libxml2=no --with-gssapi=no --disable-auto-validation --disable-doh
make
sudo bin/tests/system/ifconfig.sh up
make testWhat is the current bug behavior?
"make test" will start generating ERROR for each test:
make check-TESTS make[6]: Entering directory '/tmp_mnt/homefs/mnt/eschultz/projects/bind-9.18.21-rhe8/bind-9.18.21/bin/tests/system' make[7]: Entering directory '/tmp_mnt/homefs/mnt/eschultz/projects/bind-9.18.21-rhe8/bind-9.18.21/bin/tests/system' ERROR: rpz ERROR: rpzrecurse ERROR: serve-stale ERROR: timeouts ERROR: upforwdBut the log itself (and .xml) shows 100% PASSED:
% tail bin/tests/system/rpz.log
2024-02-05 08:11:28 INFO:rpz I:rpz_tmp_mrhdwc8_:checking that rewriting CD=1 queries handles pending data correctly (29)
2024-02-05 08:11:28 INFO:rpz I:rpz_tmp_mrhdwc8_:status (native RPZ sub-test): 0 (pass)
2024-02-05 08:11:28 INFO:rpz I:rpz_tmp_mrhdwc8_:attempting to configure servers with DNSRPS...
2024-02-05 08:11:32 INFO:rpz I:rpz_tmp_mrhdwc8_:DNSRPS disabled at compile time
2024-02-05 08:11:32 INFO:rpz I:rpz_tmp_mrhdwc8_:DNSRPS sub-test skipped
PASSED [100%]
generated xml file: /home/eschultz/projects/bind-9.18.21-rhe8/bind-9.18.21/bin/tests/system/rpz.xml
========================== 1 passed in 98.66 seconds ===========================
ERROR rpz (exit status: 99)Every single test has exit status 99.
What is the expected correct behavior?
These tests should almost all have result PASS, when checking the logs (except for two that show 50% PASSED):
% grep PASSED bin/tests/system/test-suite.log
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [ 50%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [ 50%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]Relevant configuration files
I don't believe there is anything relevant for the build.
Relevant logs
% cat bin/tests/system/rpz.log
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-3.4.2, py-1.5.3, pluggy-0.6.0 -- /bin/python3
cachedir: ../.pytest_cache
rootdir: /tmp_mnt/homefs/mnt/eschultz/projects/bind-9.18.21-rhe8/bind-9.18.21/bin/tests/system, inifile: pytest.ini
collecting ... collected 1 item
tests_sh_rpz.py::test_rpz
-------------------------------- live log setup --------------------------------
2024-02-05 09:23:29 INFO:rpz test started: rpz/tests_sh_rpz.py
2024-02-05 09:23:29 INFO:rpz using port range: <25322, 25341>
-------------------------------- live log call ---------------------------------
2024-02-05 09:23:35 INFO:rpz I:rpz_tmp_zaujh5m9:running native RPZ sub-test
2024-02-05 09:23:35 INFO:rpz I:rpz_tmp_zaujh5m9:checking QNAME rewrites (1)
2024-02-05 09:23:42 INFO:rpz I:rpz_tmp_zaujh5m9:checking NXDOMAIN/NODATA action on QNAME trigger (2)
2024-02-05 09:23:46 INFO:rpz I:rpz_tmp_zaujh5m9:checking IP rewrites (3)
2024-02-05 09:23:48 INFO:rpz I:rpz_tmp_zaujh5m9:ns2 zone reload queued
2024-02-05 09:23:49 INFO:rpz I:rpz_tmp_zaujh5m9:ns2 zone reload queued
2024-02-05 09:23:50 INFO:rpz I:rpz_tmp_zaujh5m9:checking radix tree deletions (4)
2024-02-05 09:23:52 INFO:rpz I:rpz_tmp_zaujh5m9:checking NSDNAME rewrites (5)
2024-02-05 09:23:54 INFO:rpz I:rpz_tmp_zaujh5m9:checking NSIP rewrites (6)
2024-02-05 09:23:55 INFO:rpz I:rpz_tmp_zaujh5m9:checking walled garden NSIP rewrites (7)
2024-02-05 09:23:56 INFO:rpz I:rpz_tmp_zaujh5m9:checking policy overrides (8)
2024-02-05 09:24:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking crashes (9)
2024-02-05 09:24:04 INFO:rpz I:rpz_tmp_zaujh5m9:performance not checked; queryperf not available
2024-02-05 09:24:06 INFO:rpz I:rpz_tmp_zaujh5m9:checking that ns3 with broken rpz does not crash (10)
2024-02-05 09:24:11 INFO:rpz I:rpz_tmp_zaujh5m9:checking if rpz survives a certain class of failed reconfiguration attempts (11)
2024-02-05 09:24:12 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz failed update will keep previous rpz rules (12)
2024-02-05 09:24:13 INFO:rpz I:rpz_tmp_zaujh5m9:ns3 zone reload queued
2024-02-05 09:24:14 INFO:rpz I:rpz_tmp_zaujh5m9:checking reload of a mixed-case RPZ zone (13)
2024-02-05 09:24:14 INFO:rpz I:rpz_tmp_zaujh5m9:ns3 zone reload queued
2024-02-05 09:24:14 INFO:rpz I:rpz_tmp_zaujh5m9:checking that ttl values are not zeroed when qtype is '*' (14)
2024-02-05 09:24:14 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz updates/transfers with parent nodes added after children (15)
2024-02-05 09:24:56 INFO:rpz I:rpz_tmp_zaujh5m9:checking that going from an empty policy zone works (16)
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:ns7 zone refresh queued
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:checking that add-soa no at rpz zone level works (17)
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:checking that add-soa yes at response-policy level works (18)
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:reconfiguring server with 'add-soa no' (19)
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:checking that 'add-soa no' at response-policy level works (19)
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:checking that 'add-soa unset' works (20)
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz with delegation fails correctly (21)
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:checking policies from expired zone are no longer in effect (22)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz 'CNAME *.' (NODATA) with dns64, AAAA lookup with A-only (23)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz 'CNAME *.' (NODATA) with dns64, A lookup with A-only (24)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz 'CNAME *.' (NODATA) with dns64, AAAA lookup with no A or AAAA (25)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz 'CNAME *.' (NODATA) with dns64, A lookup with no A or AAAA (26)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz 'CNAME *.' (NODATA) with dns64, AAAA lookup with A and AAAA (27)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz 'CNAME *.' (NODATA) with dns64, A lookup with A and AAAA (28)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking that rewriting CD=1 queries handles pending data correctly (29)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:status (native RPZ sub-test): 0 (pass)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:attempting to configure servers with DNSRPS...
2024-02-05 09:25:04 INFO:rpz I:rpz_tmp_zaujh5m9:DNSRPS disabled at compile time
2024-02-05 09:25:04 INFO:rpz I:rpz_tmp_zaujh5m9:DNSRPS sub-test skipped
PASSED [100%]
generated xml file: /home/eschultz/projects/bind-9.18.21-rhe8/bind-9.18.21/bin/tests/system/rpz.xml
========================== 1 passed in 97.72 seconds ===========================
ERROR rpz (exit status: 99)% cat bin/tests/system/rpz.xml
<?xml version="1.0" encoding="utf-8"?><testsuite errors="0" failures="0" name="pytest" skips="0" tests="1" time="97.721"><testcase classname="rpz.tests_sh_rpz" file="rpz/tests_sh_rpz.py" line="12" name="test_rpz" time="97.68873071670532"></testcase></testsuite>Edited by Eric Schultz