BIND issueshttps://gitlab.isc.org/isc-projects/bind9/-/issues2024-03-26T12:49:52Zhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4655More statistics counters for average query/response size2024-03-26T12:49:52ZperlangMore statistics counters for average query/response size### Description
(Describe the problem, use cases, benefits, and/or goals.)
To get the average query/response size timely.
Is it possible to add two new statistics counters for total query/response size by byte ?
And even more, to add...### Description
(Describe the problem, use cases, benefits, and/or goals.)
To get the average query/response size timely.
Is it possible to add two new statistics counters for total query/response size by byte ?
And even more, to add a serials of counter to record the accumulate bytes for each range according to the query/response's size in bytes, such as,
```
16-31: 1482763
32-47: 170843916
48-63: 111988356
64-79: 11257767
```
I am not sure if this feature could affect the performance of the server, if it do, it's prefered to be configurable to enable or disable it at compile time or at run time.
### Request
(Describe the solution you'd like to see.)
fetch the statistics data via 'rndc stats' or 'rndc status'.
### Links / referenceshttps://gitlab.isc.org/isc-projects/bind9/-/issues/1128[ISC-support #22100] Offline KSK2024-03-26T10:42:53ZMatthijs Mekkingmatthijs@isc.org[ISC-support #22100] Offline KSKImplement offline key support.
This will require signatures for `DNSKEY`, `CDS`, and `CDNSKEY` RRsets to be generated in advance. This is done with a Key Signing Request (`KSR`). On the system with the KSK provide the `KSR` and ask to c...Implement offline key support.
This will require signatures for `DNSKEY`, `CDS`, and `CDNSKEY` RRsets to be generated in advance. This is done with a Key Signing Request (`KSR`). On the system with the KSK provide the `KSR` and ask to create signed RRsets for each period (resulting in Signed Key Responses (`SKR`)).
This also means that `DNSKEY` records for the ZSK need to be pregenerated, because ZSK rollovers may take place while the KSK is offline.
This also means that "Offline KSK" does not work in conjunction with a CSK because we need to resign the zone contents periodically.
Pregenerating keys can be done with `dnssec-keygen`, or some other method. But we need to provide a `dnssec-policy` and a duration for how long a period we need to create SKRs. For example if you have a policy where ZSKs are rolled every 3 months, and you want to keep the KSK offline for a year, the key generation utility should create 4 keys. The keys should have additional metadata that sets `Predecessor` and `Successor` key metadata and/or set the `Published` and `Remove` timing metadata.
We need to load the `SKR` files. Probably import it before hand with `rndc reconfig` or `rndc import skr` or something. Then when BIND decides to rekey it needs to lookup the correct `DNSKEY`, `CDNSKEY`, and `CDS` records from the `SKR` sign, and when BIND decides to resign it needs to lookup the signatures from the same `SKR` data.
When BIND starts a new key rollover, it should select the right key. This is determined by the metadata. The new keyset must match the `SKR` data.
If the `dnssec-policy` changes, the KSR process needs to be done again and the new SKR files should be imported.
So the following changes to the code and documentation need to be made:
- [ ] Create a tool `dnssec-ksr` for dealing with KSRs (!8188)
- [ ] Add an option to `dnssec-ksr` to generate keys given an interval and a DNSSEC policy (!8188)
- [ ] Add an option to `dnssec-ksr` to create `KSR` files given an interval, a DNSSEC policy, and some keys (!8188)
- [ ] Add an option to `dnssec-ksr` to sign `KSR` files, generating `SKR` files to be imported into BIND (!8188)
- [ ] Create signed `CDS` and `CDNSKEY` RRsets in the `SKR` files (!8188)
- [ ] Add a configuration option for `named` to set a `SKR` directory.
- [ ] Add a command line option for `rndc` to import a `SKR` file or directory.
- [ ] When rekeying, make sure that the `DNSKEY`, `CDNSKEY`, and `CDS` records match the `SKR` data.
- [ ] When resigning, lookup the signature in the `SKR` data rather than trying to generate a new signature.
- [ ] Add a checkconf check that Offline KSK cannot work with CSK.
- [ ] Add documentation about Offline KSK to the ARM and DNSSEC guide.BIND 9.19.xMatthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/4560All tests for "make test" returning "ERROR" even though logs show most tests ...2024-03-26T10:30:37ZEric SchultzAll tests for "make test" returning "ERROR" even though logs show most tests passed### Summary
When I run "make test" after a successful configure/make, all of the tests throw "ERROR", but if you look at the log for each test or the xml output, it will show as passed.
### BIND version affected
bind-9.18.21
### St...### Summary
When I run "make test" after a successful configure/make, all of the tests throw "ERROR", but if you look at the log for each test or the xml output, it will show as passed.
### BIND version affected
bind-9.18.21
### Steps to reproduce
```
% cat /etc/redhat-release
Red Hat Enterprise Linux release 8.9 (Ootpa)
% cat /etc/oracle-release
Oracle Linux Server release 8.9
# Install required packages (though many of these are not listed as required on the BIND site! Namely the python RPMs.)
sudo yum install automake autoconf libtool libcap-devel python3-pytest python3 perl-Net-DNS python3-dns
# Download and install jemalloc RPMs from EPEL:
wget https://dl.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/j/jemalloc-5.2.1-2.el8.x86_64.rpm
wget https://dl.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/j/jemalloc-devel-5.2.1-2.el8.x86_64.rpm
sudo yum install jemalloc-5.2.1-2.el8.x86_64.rpm jemalloc-devel-5.2.1-2.el8.x86_64.rpm
```
Build accessory software as needed:
```
wget https://www.zlib.net/zlib-1.3.1.tar.gz
tar xzf zlib-1.3.1.tar.gz
cd zlib-1.3.1
./configure --prefix=/usr/local/zlib-1.3.1
make
make test
sudo make install
cd ../
sudo vi /etc/ld.so.conf.d/bind-libs.conf
# add /usr/local/zlib-1.3.1/lib
sudo ldconfig
wget https://www.openssl.org/source/openssl-3.0.13.tar.gz
tar xzf openssl-3.0.13.tar.gz
cd openssl-3.0.13
./config --prefix=/usr/local/openssl-3.0.13 --with-zlib-lib=/usr/local/zlib-1.3.1/lib
make
make test
sudo make install
cd ../
sudo vi /etc/ld.so.conf.d/bind-libs.conf
# update to /usr/local/openssl-3.0.13/lib64
sudo ldconfig
# Probably could use libuv RPM here but chose to get newest source
wget --no-check-certificate https://dist.libuv.org/dist/v1.47.0/libuv-v1.47.0.tar.gz
tar xzf libuv-v1.47.0.tar.gz
cd libuv-v1.47.0
sh autogen.sh
./configure --prefix=/usr/local/libuv-1.47.0
make
make check
sudo make install
cd ../
sudo vi /etc/ld.so.conf.d/bind-libs.conf
# add: /usr/local/libuv-1.47.0/lib
sudo ldconfig
# Now build BIND itself
wget https://downloads.isc.org/isc/bind9/9.18.21/bind-9.18.21.tar.xz
tar xJf bind-9.18.21.tar.xz
cd bind-9.18.21
export PKG_CONFIG_PATH=/usr/local/openssl-3.0.13/lib64/pkgconfig:/usr/local/zlib-1.3.1/lib/pkgconfig:/usr/local/libuv-1.47.0/lib/pkgconfig
./configure --prefix=/usr/local/bind-9.18.21 --mandir=/usr/local/bind-9.18.21/man --with-openssl=/usr/local/openssl-3.0.13 --with-jemalloc --with-gnu-ld --with-libxml2=no --with-gssapi=no --disable-auto-validation --disable-doh
make
sudo bin/tests/system/ifconfig.sh up
make test
```
### What is the current *bug* behavior?
"make test" will start generating ERROR for each test:
<snip>
make check-TESTS
make[6]: Entering directory '/tmp_mnt/homefs/mnt/eschultz/projects/bind-9.18.21-rhe8/bind-9.18.21/bin/tests/system'
make[7]: Entering directory '/tmp_mnt/homefs/mnt/eschultz/projects/bind-9.18.21-rhe8/bind-9.18.21/bin/tests/system'
ERROR: rpz
ERROR: rpzrecurse
ERROR: serve-stale
ERROR: timeouts
ERROR: upforwd
<snip>
But the log itself (and .xml) shows 100% PASSED:
```
% tail bin/tests/system/rpz.log
2024-02-05 08:11:28 INFO:rpz I:rpz_tmp_mrhdwc8_:checking that rewriting CD=1 queries handles pending data correctly (29)
2024-02-05 08:11:28 INFO:rpz I:rpz_tmp_mrhdwc8_:status (native RPZ sub-test): 0 (pass)
2024-02-05 08:11:28 INFO:rpz I:rpz_tmp_mrhdwc8_:attempting to configure servers with DNSRPS...
2024-02-05 08:11:32 INFO:rpz I:rpz_tmp_mrhdwc8_:DNSRPS disabled at compile time
2024-02-05 08:11:32 INFO:rpz I:rpz_tmp_mrhdwc8_:DNSRPS sub-test skipped
PASSED [100%]
generated xml file: /home/eschultz/projects/bind-9.18.21-rhe8/bind-9.18.21/bin/tests/system/rpz.xml
========================== 1 passed in 98.66 seconds ===========================
ERROR rpz (exit status: 99)
```
Every single test has exit status 99.
### What is the expected *correct* behavior?
These tests should almost all have result PASS, when checking the logs (except for two that show 50% PASSED):
```
% grep PASSED bin/tests/system/test-suite.log
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [ 50%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [ 50%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
PASSED [100%]
```
### Relevant configuration files
I don't believe there is anything relevant for the build.
### Relevant logs
```
% cat bin/tests/system/rpz.log
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-3.4.2, py-1.5.3, pluggy-0.6.0 -- /bin/python3
cachedir: ../.pytest_cache
rootdir: /tmp_mnt/homefs/mnt/eschultz/projects/bind-9.18.21-rhe8/bind-9.18.21/bin/tests/system, inifile: pytest.ini
collecting ... collected 1 item
tests_sh_rpz.py::test_rpz
-------------------------------- live log setup --------------------------------
2024-02-05 09:23:29 INFO:rpz test started: rpz/tests_sh_rpz.py
2024-02-05 09:23:29 INFO:rpz using port range: <25322, 25341>
-------------------------------- live log call ---------------------------------
2024-02-05 09:23:35 INFO:rpz I:rpz_tmp_zaujh5m9:running native RPZ sub-test
2024-02-05 09:23:35 INFO:rpz I:rpz_tmp_zaujh5m9:checking QNAME rewrites (1)
2024-02-05 09:23:42 INFO:rpz I:rpz_tmp_zaujh5m9:checking NXDOMAIN/NODATA action on QNAME trigger (2)
2024-02-05 09:23:46 INFO:rpz I:rpz_tmp_zaujh5m9:checking IP rewrites (3)
2024-02-05 09:23:48 INFO:rpz I:rpz_tmp_zaujh5m9:ns2 zone reload queued
2024-02-05 09:23:49 INFO:rpz I:rpz_tmp_zaujh5m9:ns2 zone reload queued
2024-02-05 09:23:50 INFO:rpz I:rpz_tmp_zaujh5m9:checking radix tree deletions (4)
2024-02-05 09:23:52 INFO:rpz I:rpz_tmp_zaujh5m9:checking NSDNAME rewrites (5)
2024-02-05 09:23:54 INFO:rpz I:rpz_tmp_zaujh5m9:checking NSIP rewrites (6)
2024-02-05 09:23:55 INFO:rpz I:rpz_tmp_zaujh5m9:checking walled garden NSIP rewrites (7)
2024-02-05 09:23:56 INFO:rpz I:rpz_tmp_zaujh5m9:checking policy overrides (8)
2024-02-05 09:24:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking crashes (9)
2024-02-05 09:24:04 INFO:rpz I:rpz_tmp_zaujh5m9:performance not checked; queryperf not available
2024-02-05 09:24:06 INFO:rpz I:rpz_tmp_zaujh5m9:checking that ns3 with broken rpz does not crash (10)
2024-02-05 09:24:11 INFO:rpz I:rpz_tmp_zaujh5m9:checking if rpz survives a certain class of failed reconfiguration attempts (11)
2024-02-05 09:24:12 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz failed update will keep previous rpz rules (12)
2024-02-05 09:24:13 INFO:rpz I:rpz_tmp_zaujh5m9:ns3 zone reload queued
2024-02-05 09:24:14 INFO:rpz I:rpz_tmp_zaujh5m9:checking reload of a mixed-case RPZ zone (13)
2024-02-05 09:24:14 INFO:rpz I:rpz_tmp_zaujh5m9:ns3 zone reload queued
2024-02-05 09:24:14 INFO:rpz I:rpz_tmp_zaujh5m9:checking that ttl values are not zeroed when qtype is '*' (14)
2024-02-05 09:24:14 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz updates/transfers with parent nodes added after children (15)
2024-02-05 09:24:56 INFO:rpz I:rpz_tmp_zaujh5m9:checking that going from an empty policy zone works (16)
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:ns7 zone refresh queued
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:checking that add-soa no at rpz zone level works (17)
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:checking that add-soa yes at response-policy level works (18)
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:reconfiguring server with 'add-soa no' (19)
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:checking that 'add-soa no' at response-policy level works (19)
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:checking that 'add-soa unset' works (20)
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz with delegation fails correctly (21)
2024-02-05 09:24:59 INFO:rpz I:rpz_tmp_zaujh5m9:checking policies from expired zone are no longer in effect (22)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz 'CNAME *.' (NODATA) with dns64, AAAA lookup with A-only (23)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz 'CNAME *.' (NODATA) with dns64, A lookup with A-only (24)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz 'CNAME *.' (NODATA) with dns64, AAAA lookup with no A or AAAA (25)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz 'CNAME *.' (NODATA) with dns64, A lookup with no A or AAAA (26)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz 'CNAME *.' (NODATA) with dns64, AAAA lookup with A and AAAA (27)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking rpz 'CNAME *.' (NODATA) with dns64, A lookup with A and AAAA (28)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:checking that rewriting CD=1 queries handles pending data correctly (29)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:status (native RPZ sub-test): 0 (pass)
2024-02-05 09:25:00 INFO:rpz I:rpz_tmp_zaujh5m9:attempting to configure servers with DNSRPS...
2024-02-05 09:25:04 INFO:rpz I:rpz_tmp_zaujh5m9:DNSRPS disabled at compile time
2024-02-05 09:25:04 INFO:rpz I:rpz_tmp_zaujh5m9:DNSRPS sub-test skipped
PASSED [100%]
generated xml file: /home/eschultz/projects/bind-9.18.21-rhe8/bind-9.18.21/bin/tests/system/rpz.xml
========================== 1 passed in 97.72 seconds ===========================
ERROR rpz (exit status: 99)
```
```
% cat bin/tests/system/rpz.xml
<?xml version="1.0" encoding="utf-8"?><testsuite errors="0" failures="0" name="pytest" skips="0" tests="1" time="97.721"><testcase classname="rpz.tests_sh_rpz" file="rpz/tests_sh_rpz.py" line="12" name="test_rpz" time="97.68873071670532"></testcase></testsuite>
```March 2024 (9.16.49, 9.16.49-S1, 9.18.25, 9.18.25-S1, 9.19.22)Tom KrizekTom Krizekhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4649All TSAN-enabled builds fail in AWS-based GitLab CI jobs2024-03-25T13:45:40ZMichał KępieńAll TSAN-enabled builds fail in AWS-based GitLab CI jobs[Yesterday's mass-rebuild of Docker images][1] caused some update to be
pulled into `tsan-fedora-39-amd64` that does not play nicely with AWS
hosts because all TSAN-enabled builds now fail with an error message
like:
FATAL: ThreadSa...[Yesterday's mass-rebuild of Docker images][1] caused some update to be
pulled into `tsan-fedora-39-amd64` that does not play nicely with AWS
hosts because all TSAN-enabled builds now fail with an error message
like:
FATAL: ThreadSanitizer: unexpected memory mapping 0x7d00e0772000-0x7d00e0c00000
While it is not clear what exactly happened, here are two jobs that were
run in CI for the same commit:
- [2024-03-20 14:24, passed][2]
- [2024-03-20 16:41, failed][3]
The refreshed TSAN image was pushed to the container registry at 15:13.
The TSAN builds seemingly still work fine with the refreshed TSAN image
on our bare metal runners, which use older kernels. This is consistent
with similar reports found online:
https://stackoverflow.com/questions/77850769/fatal-threadsanitizer-unexpected-memory-mapping-when-running-on-linux-kernels
The simplest course of action is to apply the workaround mentioned in
the StackOverflow post above (`sysctl vm.mmap_rnd_bits=28`) and remove
it once the issue resolves itself as kernels and packages get updated
over time.
[1]: https://gitlab.isc.org/isc-projects/images/-/pipelines/168133
[2]: https://gitlab.isc.org/isc-projects/bind9/-/jobs/4142725
[3]: https://gitlab.isc.org/isc-projects/bind9/-/jobs/4143237April 2024 (9.16.50, 9.16.50-S1, 9.18.26, 9.18.26-S1, 9.19.23)Michal NowakMichal Nowakhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4622BIND hangs in qp_makekey2024-03-25T10:53:39ZMichal NowakBIND hangs in qp_makekeyMy home resolver (`main` - 88a5befa2578bf9969652798ff0b3ede2d19e5c8) with QP got stuck after less than an hour of runtime:
```
Mar 06 17:53:17 dns.mnowak.cz named[81103]: client @0x7f11efb74c00 86.49.239.93#37346 (fedoraproject.org): que...My home resolver (`main` - 88a5befa2578bf9969652798ff0b3ede2d19e5c8) with QP got stuck after less than an hour of runtime:
```
Mar 06 17:53:17 dns.mnowak.cz named[81103]: client @0x7f11efb74c00 86.49.239.93#37346 (fedoraproject.org): query: fedoraproject.org IN AAAA +E(0)T (23.88.124.106)
Mar 06 17:53:17 dns.mnowak.cz named[81103]: client @0x7f11d913fc00 86.49.239.93#37388 (fedoraproject.org): query: fedoraproject.org IN AAAA +E(0)T (23.88.124.106)
Mar 06 17:53:18 dns.mnowak.cz named[81103]: client @0x7f11d905c400 86.49.239.93#37265 (cdn.samsungcloudsolution.com): query: cdn.samsungcloudsolution.com IN A + (23.88.124.106)
Mar 06 17:53:18 dns.mnowak.cz named[81103]: client @0x7f11ef80dc00 86.49.239.93#37366 (cdn.samsungcloudsolution.com): query: cdn.samsungcloudsolution.com IN A + (23.88.124.106)
Mar 06 17:53:20 dns.mnowak.cz named[81103]: client @0x7f11c5f93400 86.49.239.93#37217 (cdn.samsungcloudsolution.com): query: cdn.samsungcloudsolution.com IN A + (23.88.124.106)
Mar 06 17:53:20 dns.mnowak.cz named[81103]: client @0x7f11c5e70800 86.49.239.93#37210 (cdn.samsungcloudsolution.com): query: cdn.samsungcloudsolution.com IN A + (23.88.124.106)
Mar 06 17:53:21 dns.mnowak.cz named[81103]: client @0x7f11d913fc00 86.49.239.93#37346 (download.copr.fedorainfracloud.org): query: download.copr.fedorainfracloud.org IN A +E(0)T (23.88.124.106)
Mar 06 17:53:24 dns.mnowak.cz named[81103]: client @0x7f11c5f80c00 86.49.239.93#37347 (cdn.samsungcloudsolution.com): query: cdn.samsungcloudsolution.com IN A + (23.88.124.106)
Mar 06 17:53:24 dns.mnowak.cz named[81103]: client @0x7f11c5e6f000 86.49.239.93#37298 (cdn.samsungcloudsolution.com): query: cdn.samsungcloudsolution.com IN A + (23.88.124.106)
Mar 06 17:53:26 dns.mnowak.cz named[81103]: client @0x7f11c6408800 86.49.239.93#37257 (cdn.samsungcloudsolution.com): query: cdn.samsungcloudsolution.com IN A + (23.88.124.106)
Mar 06 17:53:31 dns.mnowak.cz named[81103]: client @0x7f11c5e57400 2a02:8308:a006:2b00::901f#56962 (download.copr.fedorainfracloud.org): query: download.copr.fedorainfracloud.org IN A +E(0)T (2a01:4f8:1c17:cdb0::1)
```
I `abort`ed the server to get a core file. Here's a backtrace:
```
#0 dns_qpkey_fromname (key=0x7ffdf559a200 "\002\"%\032\002\034&\026\002 \024''\030% \"&'\002\002", name=0x7f11c6260110) at qp.c:234
#1 0x00007f11f18dd88f in qp_makekey (key=<optimized out>, uctx=<optimized out>, pval=<optimized out>, ival=<optimized out>) at qpdb.c:201
#2 0x00007f11f18bf605 in leaf_qpkey (qpr=..., n=0x7f11f02ee048, key=0x7ffdf559a200 "\002\"%\032\002\034&\026\002 \024''\030% \"&'\002\002") at /home/newman/bind9/lib/dns/qp_p.h:883
#3 fix_iterator (qp=qp@entry=0x7f11f02d82c0, iter=iter@entry=0x7ffdf559c160, start=<optimized out>,
search=search@entry=0x7ffdf559bec0 "\002\"%\032\002\031\030\027\"%\024\034!\031%\024\026\037\"(\027\002\026\"#%\002\027\"*!\037\"\024\027\002\002", searchlen=searchlen@entry=36, bit=bit@entry=37 '%',
offset=9) at qp.c:2152
#4 0x00007f11f18bfeee in dns_qp_lookup (qpr=..., name=name@entry=0x7f11d9108380, foundname=foundname@entry=0x0, iter=iter@entry=0x7ffdf559c160, chain=0x7ffdf559b4a0, chain@entry=0x0,
pval_r=pval_r@entry=0x7ffdf559d178, ival_r=0x0) at qp.c:2284
#5 0x00007f11f18d62c0 in find_coveringnsec (search=search@entry=0x7ffdf559d6b0, name=name@entry=0x7f11d9108380, nodep=nodep@entry=0x7ffdf55a0080, now=now@entry=1709744001,
foundname=foundname@entry=0x7f11d9108100, rdataset=rdataset@entry=0x7f11d9129c00, sigrdataset=0x7f11d912b480) at qp-cachedb.c:636
#6 0x00007f11f18d69ee in cache_find (db=<optimized out>, name=0x7f11d9108380, version=<optimized out>, type=1, options=16, now=<optimized out>, nodep=0x7ffdf55a0080, foundname=0x7f11d9108100,
rdataset=0x7f11d9129c00, sigrdataset=0x7f11d912b480) at qp-cachedb.c:803
#7 0x00007f11f18538f7 in dns__db_findext (db=0x7f11f0259800, name=name@entry=0x7f11d9108380, version=0x0, type=1, options=options@entry=16, now=1709744001, nodep=0x7ffdf55a0080, foundname=0x7f11d9108100,
methods=0x7ffdf559f6f0, clientinfo=0x7ffdf559f660, rdataset=0x7f11d9129c00, sigrdataset=0x7f11d912b480) at db.c:535
#8 0x00007f11f1add5b1 in query_lookup (qctx=qctx@entry=0x7ffdf559fbe0) at query.c:5969
#9 0x00007f11f1ade2f7 in ns__query_start (qctx=qctx@entry=0x7ffdf559fbe0) at query.c:5817
#10 0x00007f11f1ae3b6e in query_setup (client=client@entry=0x7f11d913fc00, qtype=qtype@entry=1) at query.c:5529
#11 0x00007f11f1ae4667 in ns_query_start (client=client@entry=0x7f11d913fc00, handle=handle@entry=0x7f11d9180700) at query.c:12107
#12 0x00007f11f1ac66fe in ns_client_request (handle=0x7f11d9180700, eresult=<optimized out>, region=<optimized out>, arg=<optimized out>) at client.c:2231
#13 0x00007f11f1b2a968 in streamdns_on_complete_dnsmessage (dnsasm=<optimized out>, region=0x7ffdf55a0850, sock=0x7f11d9161300, transphandle=0x7f11ee98d500) at netmgr/streamdns.c:147
#14 streamdns_on_dnsmessage_data_cb (dnsasm=<optimized out>, result=<optimized out>, region=0x7ffdf55a0850, cbarg=0x7f11d9161300, userarg=0x7f11ee98d500) at netmgr/streamdns.c:206
#15 0x00007f11f1b2a1c8 in isc__dnsstream_assembler_callcb (dnsasm=0x7f11efb71300, result=ISC_R_SUCCESS, region=0x7ffdf55a0850, userarg=0x0) at ./include/isc/dnsstream.h:306
#16 isc__dnsstream_assembler_handle_message (dnsasm=dnsasm@entry=0x7f11efb71300, userarg=userarg@entry=0x7f11ee98d500) at ./include/isc/dnsstream.h:353
#17 0x00007f11f1b2ab7b in isc__dnsstream_assembler_processing (dnsasm=0x7f11efb71300, userarg=0x7f11ee98d500) at ./include/isc/dnsstream.h:370
#18 isc__dnsstream_assembler_incoming_direct (dnsasm=0x7f11efb71300, userarg=0x7f11ee98d500, buf=0x7ffdf55a0970, buf_size=65) at ./include/isc/dnsstream.h:396
#19 isc_dnsstream_assembler_incoming (dnsasm=0x7f11efb71300, userarg=0x7f11ee98d500, buf=0x7ffdf55a0970, buf_size=65) at ./include/isc/dnsstream.h:508
#20 streamdns_handle_incoming_data (sock=sock@entry=0x7f11d9161300, transphandle=transphandle@entry=0x7f11ee98d500, data=0x7ffdf55a0970, len=65) at netmgr/streamdns.c:242
#21 0x00007f11f1b2ad34 in streamdns_readcb (handle=0x7f11ee98d500, result=ISC_R_SUCCESS, region=0x7ffdf55a0960, cbarg=0x7f11d9161300) at netmgr/streamdns.c:557
#22 0x00007f11f1b30b73 in tls_do_bio (sock=sock@entry=0x7f11d9162200, received_data=received_data@entry=0x7ffdf55b09f0, send_data=send_data@entry=0x0, finish=finish@entry=false) at netmgr/tlsstream.c:679
#23 0x00007f11f1b30fb5 in tls_readcb (handle=0x7f11ee987400, result=ISC_R_SUCCESS, region=0x7ffdf55b09f0, cbarg=0x7f11d9162200) at netmgr/tlsstream.c:840
#24 0x00007f11f1b2314c in isc___nm_readcb (arg=<optimized out>) at netmgr/netmgr.c:1856
#25 0x00007f11f1b2321e in isc__nm_readcb (sock=sock@entry=0x7f11d9163b00, uvreq=<optimized out>, eresult=eresult@entry=ISC_R_SUCCESS, async=async@entry=false) at netmgr/netmgr.c:1871
#26 0x00007f11f1b2ee80 in isc__nm_tcp_read_cb (stream=<optimized out>, nread=174, buf=0x7ffdf55b0a90) at netmgr/tcp.c:772
#27 0x00007f11f131c6fb in uv.read () from /lib64/libuv.so.1
#28 0x00007f11f131ca78 in uv.stream_io () from /lib64/libuv.so.1
#29 0x00007f11f132262b in uv.io_poll () from /lib64/libuv.so.1
#30 0x00007f11f1309708 in uv_run () from /lib64/libuv.so.1
#31 0x00007f11f1b48255 in loop_thread (arg=arg@entry=0x7f11f0227000) at loop.c:284
#32 0x00007f11f1b5a02e in thread_body (wrap=0x7f11f0325480) at thread.c:85
#33 0x00007f11f1b5a0a7 in isc_thread_main (func=func@entry=0x7f11f1b481ca <loop_thread>, arg=0x7f11f0227000) at thread.c:116
#34 0x00007f11f1b491ea in isc_loopmgr_run (loopmgr=0x7f11f0209a80) at loop.c:456
#35 0x0000000000426ac0 in main (argc=4, argv=0x7ffdf55b4e08) at main.c:1574
```
[bt-full.txt](/uploads/e4d8eb5744a3ea1c45f09cc9bd521294/bt-full.txt)May 2024 (9.18.27, 9.18.27-S1, 9.19.24)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/4653Are Application-layer Loop DoS Attacks relevant for bind9?2024-03-25T05:26:23ZPetr MenšíkAre Application-layer Loop DoS Attacks relevant for bind9?A new document were shared to me from our security team:
<<redacted>>
They are mentioning DNS, but it seems to be not a problem for any well behaving DNS server. Have you seen this paper already? Do you have already some stance for desc...A new document were shared to me from our security team:
<<redacted>>
They are mentioning DNS, but it seems to be not a problem for any well behaving DNS server. Have you seen this paper already? Do you have already some stance for described attacks? To me it seems this should not affect any well behaving resolver or its client.
Have you already assessed this kind of attack, whether it is relevant on bind9 in any well configured instance?
Can you confirm whether this strange thing is known to be relevant or irelevant to bind9 versions?https://gitlab.isc.org/isc-projects/bind9/-/issues/4436nsupdate segfaults in tsiggss on FreeBSD 142024-03-22T12:19:44ZMichal Nowaknsupdate segfaults in tsiggss on FreeBSD 14`nsupdate` segfaults in the `tsiggss` system test on FreeBSD 14.0 on ~"v9.18" and ~"v9.16".
Here's a first crash in the system test. There are several more crashes afterward.
```
2023-11-15 12:20:53,799 INFO:tsiggss I:tsiggss_tm...`nsupdate` segfaults in the `tsiggss` system test on FreeBSD 14.0 on ~"v9.18" and ~"v9.16".
Here's a first crash in the system test. There are several more crashes afterward.
```
2023-11-15 12:20:53,799 INFO:tsiggss I:tsiggss_tmp_dk09tbmf:testing updates to testdc1 as administrator (1)
2023-11-15 12:20:53,800 INFO:tsiggss I:tsiggss_tmp_dk09tbmf:testing update for testdc1.example.nil. A 86400 A 10.53.0.10
2023-11-15 12:20:53,840 INFO:tsiggss Segmentation fault (core dumped)
2023-11-15 12:20:53,841 INFO:tsiggss I:tsiggss_tmp_dk09tbmf:update failed for testdc1.example.nil. A 86400 A 10.53.0.10
2023-11-15 12:20:53,841 INFO:tsiggss I:Reply from SOA query:
2023-11-15 12:20:53,841 INFO:tsiggss I:;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 47069
2023-11-15 12:20:53,842 INFO:tsiggss I:;; flags: qr aa; QUESTION: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
2023-11-15 12:20:53,842 INFO:tsiggss I:;; QUESTION SECTION:
2023-11-15 12:20:53,842 INFO:tsiggss I:;testdc1.example.nil. IN SOA
2023-11-15 12:20:53,842 INFO:tsiggss I:
2023-11-15 12:20:53,842 INFO:tsiggss I:;; AUTHORITY SECTION:
2023-11-15 12:20:53,842 INFO:tsiggss I:example.nil. 0 IN SOA blu.example.nil. hostmaster.example.nil. 2010113027 172800 14400 3628800 604800
2023-11-15 12:20:53,842 INFO:tsiggss I:
2023-11-15 12:20:53,842 INFO:tsiggss I:Found zone name: example.nil
2023-11-15 12:20:53,842 INFO:tsiggss I:The primary is: blu.example.nil
2023-11-15 12:20:53,843 INFO:tsiggss I:start_gssrequest
2023-11-15 12:20:53,843 INFO:tsiggss I:Found realm from ticket: EXAMPLE.NIL
2023-11-15 12:20:53,843 INFO:tsiggss I:tsiggss_tmp_dk09tbmf:failed
```
Sample `nsupdate` backtrace:
```
Core was generated by `/root/bind9/bin/nsupdate/.libs/nsupdate -g -d ns1/update.txt'.
Program terminated with signal SIGSEGV, Segmentation fault.
Address not mapped to object.
#0 0x00000008316a1a0f in EVP_Cipher () from /lib/libcrypto.so.30
[Current thread is 1 (LWP 188477)]
#0 0x00000008316a1a0f in EVP_Cipher () from /lib/libcrypto.so.30
#1 0x000000082e96f4b6 in ?? () from /usr/lib/libkrb5.so.11
#2 0x000000082e973ac8 in krb5_encrypt_ivec () from /usr/lib/libkrb5.so.11
#3 0x000000082e973de5 in krb5_encrypt () from /usr/lib/libkrb5.so.11
#4 0x000000082e9675bf in _krb5_build_authenticator () from /usr/lib/libkrb5.so.11
#5 0x000000082dcff3f6 in ?? () from /usr/lib/libgssapi_krb5.so.10
#6 0x000000082dcfed0b in _gsskrb5_init_sec_context () from /usr/lib/libgssapi_krb5.so.10
#7 0x000000082d95bd4f in gss_init_sec_context () from /usr/lib/libgssapi.so.10
#8 0x000000083ed613b6 in ?? () from /usr/lib/libgssapi_spnego.so.10
#9 0x000000083ed5f5c0 in _gss_spnego_indicate_mechtypelist () from /usr/lib/libgssapi_spnego.so.10
#10 0x000000083ed607ee in _gss_spnego_init_sec_context () from /usr/lib/libgssapi_spnego.so.10
#11 0x000000082d95bd4f in gss_init_sec_context () from /usr/lib/libgssapi.so.10
#12 0x0000000822a308e5 in dst_gssapi_initctx (name=<optimized out>, intoken=intoken@entry=0x0, outtoken=outtoken@entry=0x83d56d700, gssctx=0x83d56e218, mctx=0x1aef866b3000, err_message=0x83d56e200) at gssapictx.c
#13 0x0000000822b0c9af in dns_tkey_buildgssquery (msg=0x1aef87203a80, name=0x2130e0 <fkname>, gname=0x1aef87234300, gname@entry=0x83d56d7a0, intoken=0x1aef872700f0, intoken@entry=0x0, lifetime=lifetime@entry=0, context=0xcf, context@entry=0x83d56e218, win2k=<optimized out>, mctx=0x1aef866b3000, err_message=0x83d56e200) at tkey.c
#14 0x000000000020e790 in start_gssrequest (primary=primary@entry=0x83d56e730) at nsupdate.c
#15 0x000000000020e33c in recvsoa (task=<optimized out>, event=0x0) at nsupdate.c
#16 0x0000000821c68370 in task_run (task=0x1aef8665c140) at task.c
#17 isc_task_run (task=0x1aef8665c140) at task.c
#18 0x0000000821c38689 in isc__nm_async_task (worker=worker@entry=0x1aef866d0000, ev0=0x1aef872700f0, ev0@entry=0x1aef8721c480) at netmgr/netmgr.c
#19 0x0000000821c32ec6 in process_netievent (worker=worker@entry=0x1aef866d0000, ievent=ievent@entry=0x1aef8721c480) at netmgr/netmgr.c
#20 0x0000000821c384f2 in process_queue (worker=worker@entry=0x1aef866d0000, type=type@entry=NETIEVENT_TASK) at netmgr/netmgr.c
#21 0x0000000821c2e6bd in process_all_queues (worker=0x1aef866d0000) at netmgr/netmgr.c
#22 async_cb (handle=0x1aef866d02d8) at netmgr/netmgr.c
#23 0x0000000829b3c871 in ?? () from /usr/local/lib/libuv.so.1
#24 0x0000000829b4e0fd in ?? () from /usr/local/lib/libuv.so.1
#25 0x0000000829b3ce60 in uv_run () from /usr/local/lib/libuv.so.1
#26 0x0000000821c2e7ab in nm_thread (worker0=0x1aef866d0000) at netmgr/netmgr.c
#27 0x0000000821c70e46 in isc__trampoline_run (arg=0x1aef8662bb90) at trampoline.c
#28 0x00000008376e0a75 in ?? () from /lib/libthr.so.3
#29 0x0000000000000000 in ?? ()
```
```
BIND 9.18.21-dev (Extended Support Version) <id:ed78bc4>
running on FreeBSD amd64 14.0-RC2 FreeBSD 14.0-RC2 #0 releng/14.0-n265317-1d2ff5639925: Fri Oct 20 06:17:03 UTC 2023 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC
built by make with '--disable-maintainer-mode' '--enable-developer' '--enable-option-checking=fatal' '--enable-dnstap' '--with-cmocka' '--with-libxml2' '--with-json-c' '--with-readline=libedit'
compiled by CLANG FreeBSD Clang 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152)
compiled with OpenSSL version: OpenSSL 3.0.11 19 Sep 2023
linked to OpenSSL version: OpenSSL 3.0.11 19 Sep 2023
compiled with libuv version: 1.46.0
linked to libuv version: 1.46.0
compiled with libnghttp2 version: 1.57.0
linked to libnghttp2 version: 1.57.0
compiled with libxml2 version: 2.10.4
linked to libxml2 version: 21004
compiled with json-c version: 0.17
linked to json-c version: 0.17
compiled with zlib version: 1.3
linked to zlib version: 1.3
linked to maxminddb version: 1.7.1
compiled with protobuf-c version: 1.4.1
linked to protobuf-c version: 1.4.1
threads support is enabled
DNSSEC algorithms: RSASHA1 NSEC3RSASHA1 RSASHA256 RSASHA512 ECDSAP256SHA256 ECDSAP384SHA384 ED25519 ED448
DS algorithms: SHA-1 SHA-256 SHA-384
HMAC algorithms: HMAC-MD5 HMAC-SHA1 HMAC-SHA224 HMAC-SHA256 HMAC-SHA384 HMAC-SHA512
TKEY mode 2 support (Diffie-Hellman): yes
TKEY mode 3 support (GSS-API): yes
default paths:
named configuration: /usr/local/etc/named.conf
rndc configuration: /usr/local/etc/rndc.conf
DNSSEC root key: /usr/local/etc/bind.keys
nsupdate session key: /usr/local/var/run/named/session.key
named PID file: /usr/local/var/run/named/named.pid
named lock file: /usr/local/var/run/named/named.lock
geoip-directory: /usr/local/share/GeoIP
```
```
checking for krb5-config... /usr/bin/krb5-config
checking for gssapi libraries... -I/usr/include -L/usr/lib -lgssapi -lgssapi_krb5 -lheimntlm -lkrb5 -lhx509 -lcom_err -lcrypto -lasn1 -lwind -lheimbase -lroken -lcrypt -pthread
checking for gssapi/gssapi.h... yes
checking for gssapi/gssapi_krb5.h... yes
checking for gssapi_krb5.h... no
checking for gss_acquire_cred... yes
checking for krb5 libraries... -I/usr/include -L/usr/lib -lkrb5 -lhx509 -lcom_err -lcrypto -lasn1 -lwind -lheimbase -lroken -lcrypt -pthread
checking for krb5/krb5.h... no
checking for krb5.h... yes
checking for krb5_init_context... yes
```
[pytest.log.txt](/uploads/ca1a092b91023024d1c3215295837dd2/pytest.log.txt)
[core.43134-backtrace.txt](/uploads/dab5cd198e0e09345030257576a91602/core.43134-backtrace.txt)
[core.43041-backtrace.txt](/uploads/f40e49e3120623d207cd0ed5b8e93d0b/core.43041-backtrace.txt)
[core.44009-backtrace.txt](/uploads/c7c66927be73250c875443cdb6c70802/core.44009-backtrace.txt)
[core.43922-backtrace.txt](/uploads/e82cfe048701645b83c74af46773e7e1/core.43922-backtrace.txt)
[core.43252-backtrace.txt](/uploads/fdfe3109fd5b13b2f9b06f22ce0da585/core.43252-backtrace.txt)
[core.43094-backtrace.txt](/uploads/9c4a6c1e2e03753c9e71cbb5b7465ff8/core.43094-backtrace.txt)
[core.42986-backtrace.txt](/uploads/c964412aa69ad3fede8fb6cf8c9c1b5d/core.42986-backtrace.txt)
[core.42931-backtrace.txt](/uploads/6402826262a2210df3fc0bb39857813e/core.42931-backtrace.txt)
[nsupdate.out6](/uploads/bdda0082ecfb34190cec4e83a9c3f1d1/nsupdate.out6)
[nsupdate.out5](/uploads/df1a9bbff55030ce6093fea3750f08f0/nsupdate.out5)
[nsupdate.out8](/uploads/58f94412f56d2136472a3305f1b0f573/nsupdate.out8)
[nsupdate.out7](/uploads/28aa35d4d517296581d05630cae16b7d/nsupdate.out7)
[nsupdate.out4](/uploads/9ec2300228db0bf43e29609b88bfef9a/nsupdate.out4)
[nsupdate.out3](/uploads/4fcbf78e8cb9e5a356618123d0e97941/nsupdate.out3)
[nsupdate.out2](/uploads/46b5c9ce602f9bb99cb1673e72ab879d/nsupdate.out2)
[nsupdate.out11](/uploads/cb41ad3685b8d0f31315a5da05308044/nsupdate.out11)
[nsupdate.out10](/uploads/c1f8cbe2f93700b395ee9547658c66cd/nsupdate.out10)
[nsupdate.out1](/uploads/f77767f9b3e2d067f15bc1b121bd56cf/nsupdate.out1)December 2023 (9.18.21, 9.18.21-S1, 9.19.19)https://gitlab.isc.org/isc-projects/bind9/-/issues/4647root "mirror" zone overrides forward zone (forward only)2024-03-22T09:42:32ZCarsten Strotmannroot "mirror" zone overrides forward zone (forward only)
### Summary
When a root zone mirror in configured in BIND 9.18.24, a forward zone (forward only) is not used.
The forward zone is for an undelegated private namespace "example.internal"
Once the root mirror is removed from the BIND ...
### Summary
When a root zone mirror in configured in BIND 9.18.24, a forward zone (forward only) is not used.
The forward zone is for an undelegated private namespace "example.internal"
Once the root mirror is removed from the BIND 9 configuration, the forward zone starts working again.
The expectation is that the forward zone is more specific, therefor it is checked first before using recursion via the root zone mirror data
### BIND version affected
```
BIND 9.18.24 (Extended Support Version) <id:6d7674f>
running on Linux x86_64 4.18.0-513.18.1.el8_9.x86_64 #1 SMP Thu Feb 1 03:51:05 EST 2024
built by make with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/opt/isc/isc-bind/root/usr' '--exec-prefix=/opt/isc/isc-bind/root/usr' '--bindir=/opt/isc/isc-bind/root/usr/bin' '--sbindir=/opt/isc/isc-bind/root/usr/sbin' '--sysconfdir=/etc/opt/isc/scls/isc-bind' '--datadir=/opt/isc/isc-bind/root/usr/share' '--includedir=/opt/isc/isc-bind/root/usr/include' '--libdir=/opt/isc/isc-bind/root/usr/lib64' '--libexecdir=/opt/isc/isc-bind/root/usr/libexec' '--localstatedir=/var/opt/isc/scls/isc-bind' '--sharedstatedir=/var/opt/isc/scls/isc-bind/lib' '--mandir=/opt/isc/isc-bind/root/usr/share/man' '--infodir=/opt/isc/isc-bind/root/usr/share/info' '--enable-warn-error' '--disable-static' '--enable-dnstap' '--with-pic' '--with-gssapi' '--with-json-c' '--with-libxml2' '--without-lmdb' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer' 'LDFLAGS=-Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -L/opt/isc/isc-bind/root/usr/lib64' 'CPPFLAGS= -I/opt/isc/isc-bind/root/usr/include' 'LT_SYS_LIBRARY_PATH=/usr/lib64' 'PKG_CONFIG_PATH=:/opt/isc/isc-bind/root/usr/lib64/pkgconfig:/opt/isc/isc-bind/root/usr/share/pkgconfig' 'SPHINX_BUILD=/builddir/build/BUILD/bind-9.18.24/sphinx/bin/sphinx-build'
compiled by GCC 8.5.0 20210514 (Red Hat 8.5.0-20)
compiled with OpenSSL version: OpenSSL 1.1.1k FIPS 25 Mar 2021
linked to OpenSSL version: OpenSSL 1.1.1k FIPS 25 Mar 2021
compiled with libuv version: 1.44.2
linked to libuv version: 1.44.2
compiled with libnghttp2 version: 1.33.0
linked to libnghttp2 version: 1.33.0
compiled with libxml2 version: 2.9.7
linked to libxml2 version: 20907
compiled with json-c version: 0.13.1
linked to json-c version: 0.13.1
compiled with zlib version: 1.2.11
linked to zlib version: 1.2.11
compiled with protobuf-c version: 1.4.1
linked to protobuf-c version: 1.4.1
threads support is enabled
DNSSEC algorithms: RSASHA1 NSEC3RSASHA1 RSASHA256 RSASHA512 ECDSAP256SHA256 ECDSAP384SHA384 ED25519 ED448
DS algorithms: SHA-1 SHA-256 SHA-384
HMAC algorithms: HMAC-MD5 HMAC-SHA1 HMAC-SHA224 HMAC-SHA256 HMAC-SHA384 HMAC-SHA512
TKEY mode 2 support (Diffie-Hellman): yes
TKEY mode 3 support (GSS-API): yes
default paths:
named configuration: /etc/opt/isc/scls/isc-bind/named.conf
rndc configuration: /etc/opt/isc/scls/isc-bind/rndc.conf
DNSSEC root key: /etc/opt/isc/scls/isc-bind/bind.keys
nsupdate session key: /var/opt/isc/scls/isc-bind/run/named/session.key
named PID file: /var/opt/isc/scls/isc-bind/run/named/named.pid
named lock file: /var/opt/isc/scls/isc-bind/run/named/named.lock
```
### Steps to reproduce
named.conf:
```
options {
directory "/var/named";
};
zone "." {
type mirror;
file "root.zone";
};
zone "example.internal" IN {
type forward;
forwarders { 192.0.2.53; };
forward only;
};
```
Test:
```
dig @<ip-of-resolver> example.internal soa
```
will return an NXDOMAIN answer (with AD flag) from the (local) root zone
Once the Root-Mirror zone is removed, the forwarding configuration works as expected (returns the SOA record for example.internal)https://gitlab.isc.org/isc-projects/bind9/-/issues/4651Add Dual Queue Low Latency Networking Support (NQB)2024-03-22T08:48:50ZJason LivingoodAdd Dual Queue Low Latency Networking Support (NQB)### Description
Add Dual Queue Low Latency Networking Support (NQB)
Please consider adding server-side support for IETF Non-Queue-Building (NQB) Per Hop Behavior (PHB) as outlined in the IETF TSVWG RFCs 9330, 9331, 9332 and https://dat...### Description
Add Dual Queue Low Latency Networking Support (NQB)
Please consider adding server-side support for IETF Non-Queue-Building (NQB) Per Hop Behavior (PHB) as outlined in the IETF TSVWG RFCs 9330, 9331, 9332 and https://datatracker.ietf.org/doc/draft-ietf-tsvwg-nqb/. Specifically, I would like the recursive resolver to set DSCP-45 marking in all packets sent back to users (stub resolvers) in DNS responses. This will have the benefit of marking DNS responses as suitable for placement in the low latency queue at bottleneck links supporting dual queue (such as a CMTS or Cable Modem).
NQB marking enables latency-sensitive traffic like DNS lookups to be handled in a separate queue from classic traffic. The result is that, even when competing with significant other LAN or access network traffic from a user, that the NQB-marked traffic will get very low working latency (usually close to what is observed for idle latency).
Comcast has tested this on resolvers in the lab as part of our low latency field trial of L4S and NQB and found it meaningfully reduced Query Response Times (QRT) under normal working conditions.
Comcast is currently the world's first ISP trialing this in the field and anticipates it being available to millions of end users in 2024.
### Request
Enable a new configuration parameter in the server enabling a resolver operator to turn on NQB support. That specifically will mean setting DSCP value 45 in the packet header. This configuration can either cover recursive responses or all outbound traffic from the server (there should be no downside to this).
### Links / references
RFC 9330 https://www.rfc-editor.org/rfc/rfc9330.html
RFC 9331 https://www.rfc-editor.org/rfc/rfc9331.html
RFC 9332 https://www.rfc-editor.org/rfc/rfc9332.html
NQB PHB Draft https://datatracker.ietf.org/doc/draft-ietf-tsvwg-nqb/
Comcast explainer for app developers https://github.com/jlivingood/IETF-L4S-Deployment/blob/main/App-Developer-Guide.md
Comcast explainer for network operators https://github.com/jlivingood/IETF-L4S-Deployment/blob/main/Network-Config-Guide.md
Comcast field trial announcement https://corporate.comcast.com/stories/comcast-kicks-off-industrys-first-low-latency-docsis-field-trialshttps://gitlab.isc.org/isc-projects/bind9/-/issues/4459[CVE-2023-50868] Preparing an NSEC3 closest encloser proof can exhaust CPU re...2024-03-22T08:30:44ZPetr Špačekpspacek@isc.org[CVE-2023-50868] Preparing an NSEC3 closest encloser proof can exhaust CPU resources| Quick Links | :link: |
| ------------------------ | ------------------------------------------------------------------------------ |
| Incident Manage...| Quick Links | :link: |
| ------------------------ | ------------------------------------------------------------------------------ |
| Incident Manager: | @pspacek |
| Deputy Incident Manager: | @ebf |
| Public Disclosure Date: | 2024-02-13 |
| CVSS Score: | [7.5][cvss_score] |
| Security Advisory: | isc-private/printing-press!93 |
| Mattermost Channel: | [CVE-2023-50868: NSEC3 closest encloser proof can exhaust CPU][mattermost_url] |
| Support Ticket: | N/A |
| Release Checklist: | #4555 |
[cvss_score]: https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H&version=3.1
[mattermost_url]: https://mattermost.isc.org/isc/channels/cve-2023-50868-nsec3-closest-encloser-proof-can-exhaust-cpu
:bulb: **Click [here][checklist_explanations] (internal resource) for general information about the security incident handling process.**
[checklist_explanations]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations
### Earlier Than T-5
- [x] [:link:][step_deputy] **(IM)** Pick a Deputy Incident Manager
- :no_entry_sign: [:link:][step_respond] **(IM)** Respond to the bug reporter - found internally by @pspacek
- [x] [:link:][step_public_mrs] **(SwEng)** Ensure there are no public merge requests which inadvertently disclose the issue
- [x] [:link:][step_assign_cve_id] **(IM)** Assign a CVE identifier
- [x] [:link:][step_note_cve_info] **(SwEng)** Update this issue with the assigned CVE identifier and the CVSS score
- [x] [:link:][step_versions_affected] **(SwEng)** Determine the range of product versions affected (including the Subscription Edition)
- [x] [:link:][step_workarounds] **(SwEng)** Determine whether workarounds for the problem exist
- [x] [:link:][step_coordinate] **(SwEng)** :warning: Coordinate with other parties :warning:
- [x] [:link:][step_earliest_prepare] **(Support)** ~~Prepare "earliest" notification text and hand it off to Marketing~~
- [x] [:link:][step_earliest_send] **(Marketing)** ~~Update "earliest" notification document in SF portal and send bulk email to earliest customers~~
- [x] [:link:][step_advisory_mr] **(Support)** [Create a merge request for the Security Advisory and include all readily available information in it](isc-private/printing-press!93)
- [x] [:link:][step_reproducer_mr] **(SwEng)** ~~[Prepare a private merge request containing a system test reproducing the problem](#note_434474)~~
- [x] [:link:][step_notify_support] **(SwEng)** ~~Notify Support when a reproducer is ready~~
- [x] [:link:][step_code_analysis] **(SwEng)** [Prepare a detailed explanation of the code flow triggering the problem](#note_434480)
- [x] [:link:][step_fix_mr] **(SwEng)** ~~[Prepare a private merge request with the fix](#note_434483)~~
- [x] [:link:][step_review_fix] **(SwEng)** ~~[Ensure the merge request with the fix is reviewed and has no outstanding discussions](#note_434483)~~
- [x] [:link:][step_review_docs] **(Support)** ~~[Review the documentation changes introduced by the merge request with the fix](#note_434483)~~
- [x] [:link:][step_backports] **(SwEng)** ~~[Prepare backports of the merge request addressing the problem for all affected (and still maintained) branches of a given product](#note_434483)~~
- [x] [:link:][step_finish_advisory] **(Support)** Finish preparing the Security Advisory
- [x] [:link:][step_meta_issue] **(QA)** Create (or update) the private issue containing links to fixes & reproducers for all CVEs fixed in a given release cycle
- [x] [:link:][step_changes] **(QA)** (BIND 9 only) Reserve a block of `CHANGES` placeholders once the complete set of vulnerabilities fixed in a given release cycle is determined
- [x] [:link:][step_merge_fixes] **(QA)** ~~[Merge the CVE fixes in CVE identifier order](#note_434483)~~
- [x] [:link:][step_patches] **(QA)** ~~[Prepare a standalone patch for the last stable release of each affected (and still maintained) product branch](#note_434483)~~
- [x] [:link:][step_asn_releases] **(QA)** Prepare ASN releases (as outlined in the Release Checklist)
### At T-5
- [x] [:link:][step_asn_documents] **(Marketing)** Update the text on the T-5 (from the Printing Press project) and "earliest" ASN documents in the SF portal
- [x] [:link:][step_asn_links] **(Marketing)** (BIND 9 only) Update the BIND -S information document in SF with download links to the new versions
- [x] [:link:][step_asn_send] **(Marketing)** Bulk email eligible customers to check the SF portal
- [x] [:link:][step_preannouncement] **(Marketing)** (BIND 9 only) Send a pre-announcement email to the *bind-announce* mailing list to alert users that the upcoming release will include security fixes
### At T-1
- [x] [:link:][step_packager_emails] **(First IM)** Send notifications to OS packagers
### On the Day of Public Disclosure
- [x] [:link:][step_clearance] **(IM)** [Grant QA & Marketing clearance to proceed with public release](https://mattermost.isc.org/isc/pl/rxzn1b4upbnjxrbq75dqx1m96o)
- [x] [:link:][step_publish] **(QA/Marketing)** Publish the releases (as outlined in the release checklist)
- [x] [:link:][step_matrix] **(Support)** (BIND 9 only) Add the new CVEs to the vulnerability matrix in the Knowledge Base
- [x] [:link:][step_publish_advisory] **(Support)** Bump Document Version for the Security Advisory and publish it in the Knowledge Base
- [x] [:link:][step_notifications] **(First IM)** Send notification emails to third parties
- [x] [:link:][step_mitre] **(First IM)** ~~[Advise MITRE about the disclosed CVEs](#note_436522)~~
- [x] [:link:][step_merge_advisory] **(First IM)** Merge the Security Advisory merge request
- [x] [:link:][step_embargo_end] **(IM)** Inform original reporter (if external) that the security disclosure process is complete
- [x] [:link:][step_asn_clear] **(Marketing)** Update the SF portal to clear the ASN
- [x] [:link:][step_customers] **(Marketing)** Email ASN recipients that the embargo is lifted
### After Public Disclosure
- [ ] [:link:][step_regression] **(QA)** Merge a regression test reproducing the bug into all affected (and still maintained) branches
[step_deputy]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#pick-a-deputy-incident-manager
[step_respond]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#respond-to-the-bug-reporter
[step_public_mrs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-there-are-no-public-merge-requests-which-inadvertently-disclose-the-issue
[step_assign_cve_id]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#assign-a-cve-identifier
[step_note_cve_info]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-this-issue-with-the-assigned-cve-identifier-and-the-cvss-score
[step_versions_affected]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-the-range-of-product-versions-affected-including-the-subscription-edition
[step_workarounds]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-whether-workarounds-for-the-problem-exist
[step_coordinate]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#if-necessary-coordinate-with-other-parties
[step_earliest_prepare]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-earliest-notification-text-and-hand-it-off-to-marketing
[step_earliest_send]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-earliest-notification-document-in-sf-portal-and-send-bulk-email-to-earliest-customers
[step_advisory_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-a-merge-request-for-the-security-advisory-and-include-all-readily-available-information-in-it
[step_reproducer_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-containing-a-system-test-reproducing-the-problem
[step_notify_support]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#notify-support-when-a-reproducer-is-ready
[step_code_analysis]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-detailed-explanation-of-the-code-flow-triggering-the-problem
[step_fix_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-with-the-fix
[step_review_fix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-the-merge-request-with-the-fix-is-reviewed-and-has-no-outstanding-discussions
[step_review_docs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#review-the-documentation-changes-introduced-by-the-merge-request-with-the-fix
[step_backports]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-backports-of-the-merge-request-addressing-the-problem-for-all-affected-and-still-maintained-branches-of-a-given-product
[step_finish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#finish-preparing-the-security-advisory
[step_meta_issue]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-or-update-the-private-issue-containing-links-to-fixes-reproducers-for-all-cves-fixed-in-a-given-release-cycle
[step_changes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-reserve-a-block-of-changes-placeholders-once-the-complete-set-of-vulnerabilities-fixed-in-a-given-release-cycle-is-determined
[step_merge_fixes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-cve-fixes-in-cve-identifier-order
[step_patches]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-standalone-patch-for-the-last-stable-release-of-each-affected-and-still-maintained-product-branch
[step_asn_releases]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-asn-releases-as-outlined-in-the-release-checklist
[step_asn_documents]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-the-text-on-the-t-5-from-the-printing-press-project-and-earliest-asn-documents-in-the-sf-portal
[step_asn_links]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-update-the-bind-s-information-document-in-sf-with-download-links-to-the-new-versions
[step_asn_send]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bulk-email-eligible-customers-to-check-the-sf-portal
[step_preannouncement]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-send-a-pre-announcement-email-to-the-bind-announce-mailing-list-to-alert-users-that-the-upcoming-release-will-include-security-fixes
[step_packager_emails]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notifications-to-os-packagers
[step_clearance]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#grant-qa-marketing-clearance-to-proceed-with-public-release
[step_publish]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#publish-the-releases-as-outlined-in-the-release-checklist
[step_matrix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-add-the-new-cves-to-the-vulnerability-matrix-in-the-knowledge-base
[step_publish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bump-document-version-for-the-security-advisory-and-publish-it-in-the-knowledge-base
[step_notifications]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notification-emails-to-third-parties
[step_mitre]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#advise-mitre-about-the-disclosed-cves
[step_merge_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-security-advisory-merge-request
[step_embargo_end]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#inform-original-reporter-if-external-that-the-security-disclosure-process-is-complete
[step_asn_clear]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-the-sf-portal-to-clear-the-asn
[step_customers]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#email-asn-recipients-that-the-embargo-is-lifted
[step_regression]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-a-regression-test-reproducing-the-bug-into-all-affected-and-still-maintained-branches
### Reproducer
1. Sign an empty zone with NSEC3, 150 iterations, and same NSEC3 salt for a good measure:
- [local.testiscorg.ch.zone](/uploads/b4a147bdabff809350e0a7a7b802758e/local.testiscorg.ch.zone)
- [Klocal.testiscorg.ch.+014+01043.key](/uploads/27aa1a99ac52e271ae1bf618c7fc4138/Klocal.testiscorg.ch.+014+01043.key)
- [Klocal.testiscorg.ch.+014+01043.private](/uploads/346295e7f71ed644dd44bb93e52ea531/Klocal.testiscorg.ch.+014+01043.private)
- `dnssec-signzone -u -3 0122345678912345 -H 150 -e 20380101000000 -S -o local.testiscorg.ch -O full -z local.testiscorg.ch.zone Klocal.testiscorg.ch.+014+01043`
- :point_right_tone1: [local.testiscorg.ch.zone.signed](/uploads/ba12811b13cc749084b6c1cef0c3a04a/local.testiscorg.ch.zone.signed)
2. Run an auth with the zone:
- [auth.conf](/uploads/51139fed8b8efe23eb58b82ce4b82379/auth.conf)
- `named -g -c auth.conf`
3. Run a resolver with the zone:
- [resolver.conf](/uploads/2b9a661105397636c55f9c5be13d8855/resolver.conf)
- `named -g -c resolver.conf`
4. Run attack using dnsperf:
- [randlabels.py](/uploads/30b54afbe090da16c06855f5561755df/randlabels.py)
- `python randlabels.py | dnsperf -s 127.0.0.1 -S1`
### Observed behavior
Around 200 QPS, one CPU maxed out. Tweaking dnsperf params can max out all CPUs with ~ 200 queries per core.
### Problem
For NSEC3 we have to hash all the labels between QNAME and zone name to find out a matching NSEC3 RR in authority section. This inflates number of hashes to potentially ~ `127 labels * <NSEC3 iterations> * <number of NSEC3 RRs in the message>`.
We have to cap this somehow. Coordination with other vendors is needed because BIND, Unbound, Knot Resolver, and PowerDNS in current versions are affected. This seems like a protocol issue so other vendors are most likely also affected, see the NSEC3 algorithm here: https://datatracker.ietf.org/doc/html/rfc5155#section-8.3February 2024 (9.16.47/9.16.48, 9.16.47/9.16.48-S1, 9.18.23/9.18.24, 9.18.23/9.18.24-S1, 9.19.21)https://gitlab.isc.org/isc-projects/bind9/-/issues/2188Bug in message.c:673: ENSURE(isc_mempool_getallocated(msg->namepool) == 0) fa...2024-03-22T08:14:06ZFstarkBug in message.c:673: ENSURE(isc_mempool_getallocated(msg->namepool) == 0) failed<!--
If the bug you are reporting is potentially security-related - for example,
if it involves an assertion failure or other crash in `named` that can be
triggered repeatedly - then please do *NOT* report it here, but send an
email to [...<!--
If the bug you are reporting is potentially security-related - for example,
if it involves an assertion failure or other crash in `named` that can be
triggered repeatedly - then please do *NOT* report it here, but send an
email to [security-officer@isc.org](security-officer@isc.org).
-->
### Summary
message.c:673: ENSURE(isc_mempool_getallocated(msg->namepool) == 0) failed, back trace
```
test@test:~/bind9/collect$ ./dns_message_parse_fuzzer id\:000000\,sig\:06\,src\:002736+001626\,time\:192782276\,op\:splice\,rep\:128
INFO: Seed: 1666455395
INFO: Loaded 1 modules (61310 inline 8-bit counters): 61310 [0x100d2b0, 0x101c22e),
INFO: Loaded 1 PC tables (61310 PCs): 61310 [0x101c230,0x110ba10),
./dns_message_parse_fuzzer: Running 1 inputs 1 time(s) each.
Running: id:000000,sig:06,src:002736+001626,time:192782276,op:splice,rep:128
message.c:673: ENSURE(isc_mempool_getallocated(msg->namepool) == 0) failed, back trace
./dns_message_parse_fuzzer() [0xab474a]
./dns_message_parse_fuzzer() [0xab43d0]
./dns_message_parse_fuzzer() [0xab422a]
./dns_message_parse_fuzzer() [0x566c5f]
./dns_message_parse_fuzzer() [0x566da7]
./dns_message_parse_fuzzer() [0x551bc4]
./dns_message_parse_fuzzer() [0x550f98]
./dns_message_parse_fuzzer() [0x45a0c2]
./dns_message_parse_fuzzer() [0x445843]
./dns_message_parse_fuzzer() [0x44b89f]
./dns_message_parse_fuzzer() [0x473213]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f0090f5eb97]
./dns_message_parse_fuzzer() [0x41fed9]
==23768== ERROR: libFuzzer: deadly signal
#0 0x527611 in __sanitizer_print_stack_trace /src/llvm-project/compiler-rt/lib/asan/asan_stack.cpp:86:3
#1 0x472a38 in fuzzer::PrintStackTrace() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtil.cpp:210:5
#2 0x458b63 in fuzzer::Fuzzer::CrashCallback() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:232:3
#3 0x7f009196489f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1289f)
#4 0x7f0090f7bf46 in __libc_signal_restore_set /build/glibc-2ORdQG/glibc-2.27/signal/../sysdeps/unix/sysv/linux/nptl-signals.h:80
#5 0x7f0090f7bf46 in gsignal /build/glibc-2ORdQG/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c:48
#6 0x7f0090f7d8b0 in abort /build/glibc-2ORdQG/glibc-2.27/stdlib/abort.c:79
#7 0xab4233 in isc_assertion_failed /src/bind9/lib/isc/assertions.c:47:2
#8 0x566c5e in msgreset /src/bind9/lib/dns/message.c:673:2
#9 0x566da6 in dns_message_destroy /src/bind9/lib/dns/message.c:801:2
#10 0x551bc3 in render_message /src/bind9/fuzz/dns_message_parse.c:131:2
#11 0x550f97 in LLVMFuzzerTestOneInput /src/bind9/fuzz/dns_message_parse.c:162:11
#12 0x45a0c1 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:558:15
#13 0x445842 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:296:6
#14 0x44b89e in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:796:9
#15 0x473212 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:19:10
#16 0x7f0090f5eb96 in __libc_start_main /build/glibc-2ORdQG/glibc-2.27/csu/../csu/libc-start.c:310
#17 0x41fed8 in _start (/home/test/bind9/collect/dns_message_parse_fuzzer+0x41fed8)
NOTE: libFuzzer has rudimentary signal handlers.
Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
```
### BIND version used
master-git
### Steps to reproduce
./fuzzer POC
[bind9.zip](/uploads/58bf9e655b37622954937535b1e69bd6/bind9.zip)
### What is the current *bug* behavior?
crash
### Relevant logs and/or screenshots
File in zip
### Possible fixes
(If you can, link to the line of code that might be responsible for the
problem.)October 2020 (9.11.24, 9.11.24-S1, 9.16.8, 9.16.8-S1, 9.17.6)Mark AndrewsMark Andrewshttps://gitlab.isc.org/isc-projects/bind9/-/issues/4605re-enable enginepkcs11 system test2024-03-21T16:36:27ZTom Krizekre-enable enginepkcs11 system testThe `enginepkcs11` test was accidentally disabled by a wrong `prereq.sh` condition in https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/5924/diffs?commit_id=2e9fd6d0c11d0648589da5779baeefb5b98e855e#8b557d8387b0ad5e06dad7a7c2c6f6...The `enginepkcs11` test was accidentally disabled by a wrong `prereq.sh` condition in https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/5924/diffs?commit_id=2e9fd6d0c11d0648589da5779baeefb5b98e855e#8b557d8387b0ad5e06dad7a7c2c6f6521febff5e_16_16.
Once [enabled](https://gitlab.isc.org/isc-projects/bind9/-/commit/f3d402d1aa2e359caa741ce2b4742795a4a37224), the test has a couple of [failures](https://gitlab.isc.org/isc-projects/bind9/-/jobs/4068956) that need to be addressed:
```
2024-02-26 17:25:39 INFO:enginepkcs11.test_enginepkcs11 I:Test SOA is signed for ecdsap256sha256.same-policy.views in view1 (65)
2024-02-26 17:25:42 INFO:enginepkcs11.test_enginepkcs11 I:failed (SOA RRset not signed)
2024-02-26 17:25:42 INFO:enginepkcs11.test_enginepkcs11 I:Test DNSKEY is signed for ecdsap256sha256.same-policy.views in view1 (66)
2024-02-26 17:25:45 INFO:enginepkcs11.test_enginepkcs11 I:failed (DNSKEY RRset not signed)
```April 2024 (9.16.50, 9.16.50-S1, 9.18.26, 9.18.26-S1, 9.19.23)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/1565"autosign" system test still fails occasionally due to apparent randomness is...2024-03-21T15:29:42ZMichał Kępień"autosign" system test still fails occasionally due to apparent randomness issuesThe `autosign` system test still [fails occasionally][1] due to the number of daily categories into which signature expiration times are divided is not high enough (4, which is less than the required minimum of 5):
```
I:autosign:checki...The `autosign` system test still [fails occasionally][1] due to the number of daily categories into which signature expiration times are divided is not high enough (4, which is less than the required minimum of 5):
```
I:autosign:checking expired signatures were jittered correctly (13)
I:autosign:404 20200121
I:autosign:202 20200123
I:autosign:599 20200124
I:autosign:405 20200125
I:autosign:error: not enough categories
I:autosign:failed
```
We should investigate our options here to prevent this test from failing as it is happening too often to just gloss over it. Assuming nothing is wrong with the implementation itself (or the test code checking the number of categories), we should either further lower the minimum required number of categories or ignore this issue completely - otherwise we are going to develop a bad reflex of ignoring all failures of this test.
[1]: https://gitlab.isc.org/isc-private/bind9/-/jobs/572751April 2024 (9.16.50, 9.16.50-S1, 9.18.26, 9.18.26-S1, 9.19.23)https://gitlab.isc.org/isc-projects/bind9/-/issues/4247Autosign resigning too fast after dnssec-settime calls2024-03-21T15:29:09ZMark AndrewsAutosign resigning too fast after dnssec-settime callsJob [#3570991](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3570991) failed for ce1db0017ee498c3c9e0d632b83113a344d25dda:
I've seen autosign fail in the ZSK roll test several times and it appears to be because dnssec-signzone is bei...Job [#3570991](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3570991) failed for ce1db0017ee498c3c9e0d632b83113a344d25dda:
I've seen autosign fail in the ZSK roll test several times and it appears to be because dnssec-signzone is being called in the same second as dnssec-settime. dnssec-signzone uses the phase `is set and is in the past` as the determinant of when things change. Should this be made `is set and is now or in the past`? Which behaviour is consistent with named's behaviour? Which behaviour is less error prone? If we don't change the behaviour we need to add sleeps to ensure the tests succeed.
```
-S This option enables smart signing, which instructs dnssec-signzone to search the key repository for keys that match the zone
being signed, and to include them in the zone if appropriate.
When a key is found, its timing metadata is examined to determine how it should be used, according to the following rules.
Each successive rule takes priority over the prior ones:
If no timing metadata has been set for the key, the key is published in the zone and used to sign the zone.
If the key's publication date is set and is in the past, the key is published in the zone.
If the key's activation date is set and is in the past, the key is published (regardless of publication date) and used to
sign the zone.
If the key's revocation date is set and is in the past, and the key is published, then the key is revoked, and the revoked
key is used to sign the zone.
If either the key's unpublication or deletion date is set and in the past, the key is NOT published or used to sign the
zone, regardless of any other metadata.
If the key's sync publication date is set and is in the past, synchronization records (type CDS and/or CDNSKEY) are
created.
If the key's sync deletion date is set and is in the past, synchronization records (type CDS and/or CDNSKEY) are removed.
```
```
2023-08-08 00:59:52 INFO:autosign I:autosign_tmp_x_6wkr8z:preparing ZSK roll
2023-08-08 00:59:52 INFO:autosign I:autosign_tmp_x_6wkr8z:ns1 A zone reload and thaw was started.
2023-08-08 00:59:52 INFO:autosign I:autosign_tmp_x_6wkr8z:ns1 Check the logs to see the result.
2023-08-08 00:59:56 INFO:autosign I:autosign_tmp_x_6wkr8z:revoking key to duplicated key ID
2023-08-08 00:59:56 INFO:autosign dnssec-settime: warning: Permissions on the file ns2/Kbar.+013+59973.private have changed from 0644 to 0600 as a result of this operation.
2023-08-08 00:59:56 INFO:autosign I:autosign_tmp_x_6wkr8z:ns2 A zone reload and thaw was started.
2023-08-08 00:59:56 INFO:autosign I:autosign_tmp_x_6wkr8z:ns2 Check the logs to see the result.
2023-08-08 00:59:56 INFO:autosign I:autosign_tmp_x_6wkr8z:waiting for changes to take effect
2023-08-08 01:00:01 INFO:autosign I:autosign_tmp_x_6wkr8z:checking former standby key 5259 is now active (53)
2023-08-08 01:00:01 INFO:autosign I:autosign_tmp_x_6wkr8z:failed
```
```
echo_i "preparing ZSK roll"
starttime=$($PERL -e 'print time(), "\n";')
oldfile=$(cat active.key)
oldid=$(keyfile_to_key_id "$(cat active.key)")
newfile=$(cat standby.key)
newid=$(keyfile_to_key_id "$(cat standby.key)")
$SETTIME -K ns1 -I now -D now+25 $oldfile > settime.out.test$n.1 || ret=1
$SETTIME -K ns1 -i 0 -S $oldfile $newfile > settime.out.test$n.2 || ret=1
# note previous zone serial number
oldserial=$($DIG $DIGOPTS +short soa . @10.53.0.1 | awk '{print $3}')
($RNDCCMD 10.53.0.1 freeze . 2>&1 | sed 's/^/ns1 /' | cat_i) || ret=1
cp ns1/root.db.signed ns1/root.db.1
$SIGNER -S -o . -O full -K ns1 -f ns1/root.db.signed ns1/root.db.1 > signing.root.out$n 2>&1 || ret=1
($RNDCCMD 10.53.0.1 thaw . 2>&1 | sed 's/^/ns1 /' | cat_i) || ret=1
sleep 4
echo_i "revoking key to duplicated key ID"
$SETTIME -R now -K ns2 Kbar.+013+59973.key > settime.out.test$n.3 || ret=1
($RNDCCMD 10.53.0.2 freeze bar. 2>&1 | sed 's/^/ns2 /' | cat_i) || ret=1
cp ns2/bar.db.signed ns2/bar.db
$SIGNER -S -o bar. -O full -K ns2 ns2/bar.db > signing.bar.out$n 2>&1 || ret=1
($RNDCCMD 10.53.0.2 thaw bar. 2>&1 | sed 's/^/ns2 /' | cat_i) || ret=1
echo_i "waiting for changes to take effect"
sleep 5
echo_i "checking former standby key $newid is now active ($n)"
ret=0
$DIG $DIGOPTS dnskey . @10.53.0.1 > dig.out.ns1.test$n || ret=1
grep 'RRSIG.*'" $newid "'\. ' dig.out.ns1.test$n > /dev/null || ret=1
n=$((n + 1))
if [ $ret != 0 ]; then echo_i "failed"; fi
status=$((status + ret))
```
```
08-Aug-2023 00:59:52.469 allocate new control connection
08-Aug-2023 00:59:52.469 received control channel command 'freeze .'
08-Aug-2023 00:59:52.469 loop exclusive mode: starting
08-Aug-2023 00:59:52.469 loop exclusive mode: started
08-Aug-2023 00:59:52.469 zone_dump: zone ./IN: enter
08-Aug-2023 00:59:52.469 loop exclusive mode: ending
08-Aug-2023 00:59:52.469 loop exclusive mode: ended
08-Aug-2023 00:59:52.469 freezing zone './IN': success
08-Aug-2023 00:59:52.469 freeing control connection
08-Aug-2023 00:59:52.497 dump_done: zone ./IN: enter
08-Aug-2023 00:59:52.497 zone_journal_compact: zone ./IN: target journal size 7662
08-Aug-2023 00:59:52.497 zone ./IN: dns_journal_compact: success
08-Aug-2023 00:59:52.509 allocate new control connection
08-Aug-2023 00:59:52.509 received control channel command 'thaw .'
08-Aug-2023 00:59:52.509 loop exclusive mode: starting
08-Aug-2023 00:59:52.509 loop exclusive mode: started
08-Aug-2023 00:59:52.509 zone ./IN: starting load
08-Aug-2023 00:59:52.509 zone_startload: zone ./IN: enter
08-Aug-2023 00:59:52.509 loop exclusive mode: ending
08-Aug-2023 00:59:52.509 loop exclusive mode: ended
08-Aug-2023 00:59:52.509 thawing zone './IN': success
08-Aug-2023 00:59:52.509 freeing control connection
08-Aug-2023 00:59:52.509 zone_loaddone: zone ./IN: enter
08-Aug-2023 00:59:52.509 zone ./IN: number of nodes in database: 5
08-Aug-2023 00:59:52.509 zone ./IN: loaded; checking validity
08-Aug-2023 00:59:52.509 dns_zone_verifydb: zone ./IN: enter
08-Aug-2023 00:59:52.509 zone ./IN: zone serial (2000042101) unchanged. zone may fail to transfer to secondaries.
08-Aug-2023 00:59:52.509 zone ./IN: replacing zone database
08-Aug-2023 00:59:52.509 calling free_rbtdb(.)
08-Aug-2023 00:59:52.509 done free_rbtdb(.)
08-Aug-2023 00:59:52.509 zone ./IN: loaded serial 2000042101 (DNSSEC signed)
08-Aug-2023 00:59:52.509 zone_postload: zone ./IN: done
08-Aug-2023 00:59:52.509 zone__settimer: zone ./IN: enter
```April 2024 (9.16.50, 9.16.50-S1, 9.18.26, 9.18.26-S1, 9.19.23)https://gitlab.isc.org/isc-projects/bind9/-/issues/4633Undefined behaviour in rdataslab.c2024-03-21T12:20:05ZMark AndrewsUndefined behaviour in rdataslab.cmemmove can be called with a NULL source pointer if the rdata length is zero. This is triggers undefined behaviour.memmove can be called with a NULL source pointer if the rdata length is zero. This is triggers undefined behaviour.April 2024 (9.16.50, 9.16.50-S1, 9.18.26, 9.18.26-S1, 9.19.23)https://gitlab.isc.org/isc-projects/bind9/-/issues/4648pytest failure oraclelinux8 in rpz/tests_sh_rpz_dnsrps.py2024-03-21T12:09:20ZMark Andrewspytest failure oraclelinux8 in rpz/tests_sh_rpz_dnsrps.pyJob [#4143909](https://gitlab.isc.org/isc-projects/bind9/-/jobs/4143909) failed for ecb043fc7b1a99a7e2ffb3d34974d16c00348471:
```
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/flaky/flaky_pytest_plugin.py", line 142, in...Job [#4143909](https://gitlab.isc.org/isc-projects/bind9/-/jobs/4143909) failed for ecb043fc7b1a99a7e2ffb3d34974d16c00348471:
```
INTERNALERROR> File "/usr/local/lib/python3.6/site-packages/flaky/flaky_pytest_plugin.py", line 142, in _call_runtest_hook
INTERNALERROR> reraise = (runner.Exit,)
INTERNALERROR> AttributeError: module '_pytest.runner' has no attribute 'Exit'
INTERNALERROR> Traceback (most recent call last):
```https://gitlab.isc.org/isc-projects/bind9/-/issues/2099Implement ZoneMD signature generation and verification.2024-03-21T03:37:01ZMark AndrewsImplement ZoneMD signature generation and verification.ZoneMD is described in <https://datatracker.ietf.org/doc/html/rfc8976>.
The record type currently implemented #867.
ZONEMD generation and verification needs to be add to both dnssec-signzone and named.
ZONEMD verification needs to be ...ZoneMD is described in <https://datatracker.ietf.org/doc/html/rfc8976>.
The record type currently implemented #867.
ZONEMD generation and verification needs to be add to both dnssec-signzone and named.
ZONEMD verification needs to be added to dnssec-verify. Does this need a seperate flag to say to expect ZONEMD?
There needs to be a way to signal to named that ZONEMD generation is to be performed for a UPDATABLE zones. This generation will need to be performed in the post update stage and must be completed before the UPDATE request is responded to. This needs to occur after the zone's serial is computed. The NSEC and NSEC3 records generation for the zone apex needs to aware of whether ZONEMD is to be generated to not. If ZONEMD generation ends up requiring the zone to be walked incrementally we will need to delay other updates to the zone until ZONEMD completes. ZONEMD must be included in each delta for a zone that is being updated.
There needs to be a way to signal to named that ZONEMD should be generated for inline zones. Similar requirements to UPDATABLE zones apply to inline zones as well.
There needs to be a way to signal to named that ZONEMD validation needs to be performed for a zone. This needs to complete before the zones contents are made visible to clients. This needs to be performed for both IXFR and AXFR. This may need to be performed incrementally. If it is being performed incrementally other transfers of the zone need to be deferred.
For IXFR do we need to check ZONEMD for each delta or only at the end of the final delta of a IXFR?
For dnssec-signzone we need a way to signal that ZONEMD should be generated.
For dnssec-signzone do we need a seperate flag to verify the ZONEMD or do we use the existing flag.
For dnssec-signzone what is the behaviour if the existing zone has a ZONEMD? Do we have a "auto" state?
What impact does this have on kasp?BIND 9.19.xhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4640Checkzone in system test leaks queries2024-03-21T02:40:22ZMark AndrewsCheckzone in system test leaks queriesThe checkzone "checking with max ttl (text)" test leaks queries.The checkzone "checking with max ttl (text)" test leaks queries.April 2024 (9.16.50, 9.16.50-S1, 9.18.26, 9.18.26-S1, 9.19.23)https://gitlab.isc.org/isc-projects/bind9/-/issues/4580Add resolver.arpa to the built in empty zones2024-03-21T00:23:52ZMark AndrewsAdd resolver.arpa to the built in empty zonesRFC 9462 adds resolver.arpa to the lists of empty zones.
```
6.4. Handling Non-DDR Queries for resolver.arpa
DNS resolvers that support DDR by responding to queries for _dns.resolver.arpa. MUST treat resolver.arpa as a
locally served z...RFC 9462 adds resolver.arpa to the lists of empty zones.
```
6.4. Handling Non-DDR Queries for resolver.arpa
DNS resolvers that support DDR by responding to queries for _dns.resolver.arpa. MUST treat resolver.arpa as a
locally served zone per [RFC6303]. In practice, this means that resolvers SHOULD respond to queries of any
type other than SVCB for _dns.resolver.arpa. with NODATA and queries of any type for any domain name under
resolver.arpa with NODATA.
```May 2024 (9.18.27, 9.18.27-S1, 9.19.24)https://gitlab.isc.org/isc-projects/bind9/-/issues/4591TTL-based cache cleaning in rbtdb ineffective (M/N problem)2024-03-20T13:50:45ZOndřej SurýTTL-based cache cleaning in rbtdb ineffective (M/N problem)Copying stuff from MatterMost for permanent record:
How it started?
> @ondrej: But I don't know actually - if there's something blocking the top of the heap to be cleaned up (dunno why yet), it could block cleaning everything else behin...Copying stuff from MatterMost for permanent record:
How it started?
> @ondrej: But I don't know actually - if there's something blocking the top of the heap to be cleaned up (dunno why yet), it could block cleaning everything else behind it
So, here's a summary of our finding
The TTL based cleaning is opportunistic. When we are inserting a new node, we look at the top of the heap ("list sorted by TTL-value") and if TTL of the top of the heap is expired, we try to eradicate the data it stores from the cache. The code that decides whether we will do the TTL-based cleaning looks like this:
```c
header = isc_heap_element(rbtdb->heaps[rbtnode->locknum], 1);
if (header != NULL) {
dns_ttl_t rdh_ttl = header->rdh_ttl;
/* Only account for stale TTL if cache is not overmem */
if (!cache_is_overmem) {
rdh_ttl += STALE_TTL(header, rbtdb);
}
if (rdh_ttl < now - RBTDB_VIRTUAL) {
expire_header(rbtdb, header, tree_locked,
expire_ttl);
}
}
```
Now, the RBTDB_VIRTUAL is this:
```
/*
* Allow clients with a virtual time of up to 5 minutes in the past to see
* records that would have otherwise have expired.
*/
#define RBTDB_VIRTUAL 300
```
Which means that even with 0 TTL records, the TTL-based cleaning will not kick-in until 5 minutes has passed.
Now what happens if you insert N records (where N could be large) within the first 5 minutes (cold bootstrap)?
After 5 minutes the TTL based cleaning should kick-in, but that's only if everything in the cache has expired, but even with that.
If we insert M records in the next minute, only M records will gets cleaned at maximum.
But anything that has longer TTL will just add to N, so the backlog of the records that are not subject to the TTL-based cleaning will build up over time until we are overmem and LRU based cleaning kicks in.
Additionally, @ondrej found out that there are records with `rdh_ttl` set to `0` if you just run `dnsperf` on a cold cache resolver, which made us puzzled - shouldn't we be cleaning records only after 5 minutes?
@michal did some excellent work and found out following:
so, i looked at those "zero TTL" frees that happen very shortly into the start of regular resolver operation
turns out that these are "superseded" headers
now, i don't understand why RBTDB does that. but it's this branch in add32() that expires these headers: https://gitlab.isc.org/isc-projects/bind9/-/blob/8fb49c5a8acdb37d4f4be0c5f5b19645d0103f48/lib/dns/rbtdb.c#L6677-6714
basically, AFAICT, what happens here is:
resolver: hey, RBTDB, add this shiny new SOA rdataset header to the cache
RBTDB: oh okay, let me see if i have a SOA header on that node already...
RBTDB: yup, got one. i'll just expire the one i had in there before (topheader) and insert the new one you gave me (newheader).
what i don't understand is why it just doesn't leave that old (identical, AFAICT) rdataset intact. but i'll move on with the reasoning for now.
so, the "previously-topmost" SOA header has its TTL set to zero, so it lands on the top of one of the TTL heaps
once we add a new rdataset, it should be cleaned up. but, as we already established yesterday, that only happens if the reference count of its node is zero.
meanwhile, in my very basic test (literally seconds into firing up dnsperf with the query set we use for respdiff tests, at some 1 kQPS), those rdatasets at the top of the heap are for nodes like net. or com. whose reference counts are around 60, or maybe 80 references. and until that count goes to zero, the first rdataset on the TTL heap will not disappear from there, blocking any rdataset headers on the same TTL heap from being cleaned up as well (backlog)
so, i thought "how bad is it and can we measure it?" and oh, boy, he could:
```
20-Feb-2024 10:46:29.561 connection refused resolving 'geo.fortinet.net/NS/IN': 104.198.248.186#53
20-Feb-2024 10:46:29.564 DNS format error from 203.205.195.75#53 resolving ns3.qq.com/AAAA for <unknown>: Name qq.com (SOA) not subdomain of zone ns3.qq.com -- invalid response
20-Feb-2024 10:46:29.564 FORMERR resolving 'ns3.qq.com/AAAA/IN': 203.205.195.75#53
20-Feb-2024 10:46:29.564 DNS format error from 108.136.87.44#53 resolving api.cloud.huawei.com/AAAA for 127.0.0.1#34840: Name huawei.com (SOA) not subdomain of zone cloud.huawei.com -- invalid response
20-Feb-2024 10:46:29.564 FORMERR resolving 'api.cloud.huawei.com/AAAA/IN': 108.136.87.44#53
-- 0 msecs since expiry
-- 0 msecs since expiry
-- 106 msecs since expiry
-- 106 msecs since expiry
-- 29 msecs since expiry
-- 29 msecs since expiry
20-Feb-2024 10:46:29.574 success resolving 'a.karma.sc2.fsapi.com/A' after disabling qname minimization due to 'ncache nxdomain'
-- 243 msecs since expiry
-- 246 msecs since expiry
-- 249 msecs since expiry
-- 433 msecs since expiry
-- 806 msecs since expiry
-- 813 msecs since expiry
-- 243 msecs since expiry
-- 246 msecs since expiry
-- 249 msecs since expiry
-- 433 msecs since expiry
-- 806 msecs since expiry
-- 0 msecs since expiry
-- 816 msecs since expiry
-- 816 msecs since expiry
-- 123 msecs since expiry
-- 323 msecs since expiry
-- 123 msecs since expiry
-- 323 msecs since expiry
20-Feb-2024 10:46:29.601 success resolving 'F00DCFRIBM41.zf.if.atcsg.net/A' after disabling qname minimization due to 'ncache nxdomain'
20-Feb-2024 10:46:29.624 REFUSED unexpected RCODE resolving 'cloud.Jensen.com.cn/NS/IN': 47.118.199.194#53
-- 0 msecs since expiry
-- 0 msecs since expiry
20-Feb-2024 10:46:29.677 DNS format error from 3.97.163.50#53 resolving api.cloud.huawei.com/AAAA for 127.0.0.1#34840: Name huawei.com (SOA) not subdomain of zone cloud.huawei.com -- invalid response
20-Feb-2024 10:46:29.677 FORMERR resolving 'api.cloud.huawei.com/AAAA/IN': 3.97.163.50#53
-- 6 msecs since expiry
-- 3 msecs since expiry
-- 9 msecs since expiry
-- 0 msecs since expiry
20-Feb-2024 10:46:29.801 REFUSED unexpected RCODE resolving 'cloud.Jensen.com.cn/NS/IN': 39.96.153.33#53
-- 3 msecs since expiry
-- 3 msecs since expiry
-- 46 msecs since expiry
-- 19 msecs since expiry
-- 0 msecs since expiry
-- 39 msecs since expiry
-- 0 msecs since expiry
-- 0 msecs since expiry
-- 3 msecs since expiry
-- 3 msecs since expiry
-- 126 msecs since expiry
-- 0 msecs since expiry
-- 0 msecs since expiry
-- 0 msecs since expiry
20-Feb-2024 10:46:30.104 REFUSED unexpected RCODE resolving 'cloud.Jensen.com.cn/NS/IN': 139.224.142.104#53
-- 286 msecs since expiry
-- 0 msecs since expiry
-- 69 msecs since expiry
-- 13 msecs since expiry
20-Feb-2024 10:46:30.307 success resolving 'hcdnd.csfw.c.cdnhwc6.com/AAAA' after disabling qname minimization due to 'ncache nxdomain'
-- 0 msecs since expiry
-- 199 msecs since expiry
-- 269 msecs since expiry
-- 656 msecs since expiry
-- 689 msecs since expiry
-- 746 msecs since expiry
-- 789 msecs since expiry
-- 816 msecs since expiry
-- 819 msecs since expiry
-- 819 msecs since expiry
-- 839 msecs since expiry
-- 853 msecs since expiry
-- 873 msecs since expiry
-- 876 msecs since expiry
-- 889 msecs since expiry
-- 896 msecs since expiry
-- 896 msecs since expiry
-- 916 msecs since expiry
-- 936 msecs since expiry
-- 956 msecs since expiry
-- 969 msecs since expiry
-- 979 msecs since expiry
-- 983 msecs since expiry
-- 999 msecs since expiry
-- 999 msecs since expiry
-- 1006 msecs since expiry
-- 1006 msecs since expiry
-- 1053 msecs since expiry
-- 1053 msecs since expiry
-- 1063 msecs since expiry
-- 1066 msecs since expiry
-- 1073 msecs since expiry
-- 1079 msecs since expiry
-- 1083 msecs since expiry
-- 1089 msecs since expiry
-- 1096 msecs since expiry
-- 1133 msecs since expiry
-- 1139 msecs since expiry
-- 1156 msecs since expiry
-- 1163 msecs since expiry
-- 1166 msecs since expiry
-- 1179 msecs since expiry
-- 1183 msecs since expiry
-- 1189 msecs since expiry
-- 1193 msecs since expiry
-- 1196 msecs since expiry
-- 1199 msecs since expiry
-- 1223 msecs since expiry
-- 1223 msecs since expiry
-- 1249 msecs since expiry
-- 1259 msecs since expiry
-- 1276 msecs since expiry
-- 1293 msecs since expiry
-- 1336 msecs since expiry
-- 1363 msecs since expiry
-- 1369 msecs since expiry
-- 1379 msecs since expiry
-- 1393 msecs since expiry
-- 1396 msecs since expiry
-- 1399 msecs since expiry
-- 1403 msecs since expiry
-- 1423 msecs since expiry
-- 1429 msecs since expiry
-- 1436 msecs since expiry
-- 1436 msecs since expiry
-- 1439 msecs since expiry
-- 1446 msecs since expiry
-- 1446 msecs since expiry
-- 1466 msecs since expiry
-- 1466 msecs since expiry
-- 1469 msecs since expiry
-- 1473 msecs since expiry
-- 1486 msecs since expiry
-- 1503 msecs since expiry
-- 1506 msecs since expiry
-- 1516 msecs since expiry
-- 1539 msecs since expiry
-- 1539 msecs since expiry
-- 1556 msecs since expiry
-- 1566 msecs since expiry
-- 1569 msecs since expiry
-- 1589 msecs since expiry
-- 1623 msecs since expiry
-- 1623 msecs since expiry
-- 1629 msecs since expiry
-- 1636 msecs since expiry
-- 1636 msecs since expiry
-- 1649 msecs since expiry
-- 1679 msecs since expiry
-- 1686 msecs since expiry
-- 1703 msecs since expiry
-- 1703 msecs since expiry
-- 1716 msecs since expiry
-- 1726 msecs since expiry
-- 1736 msecs since expiry
-- 1773 msecs since expiry
-- 1796 msecs since expiry
-- 1796 msecs since expiry
-- 1799 msecs since expiry
-- 1803 msecs since expiry
-- 1829 msecs since expiry
-- 1839 msecs since expiry
-- 1876 msecs since expiry
-- 1879 msecs since expiry
-- 1883 msecs since expiry
-- 1889 msecs since expiry
-- 1889 msecs since expiry
-- 1889 msecs since expiry
-- 1899 msecs since expiry
-- 1946 msecs since expiry
-- 1956 msecs since expiry
-- 1983 msecs since expiry
-- 2013 msecs since expiry
-- 2013 msecs since expiry
-- 2023 msecs since expiry
-- 2049 msecs since expiry
-- 2059 msecs since expiry
-- 2063 msecs since expiry
-- 2079 msecs since expiry
-- 2106 msecs since expiry
-- 2109 msecs since expiry
-- 2119 msecs since expiry
-- 2136 msecs since expiry
-- 2156 msecs since expiry
-- 2183 msecs since expiry
-- 2199 msecs since expiry
-- 2203 msecs since expiry
-- 2219 msecs since expiry
-- 2223 msecs since expiry
-- 2229 msecs since expiry
-- 2243 msecs since expiry
-- 2253 msecs since expiry
-- 2263 msecs since expiry
-- 2286 msecs since expiry
-- 2303 msecs since expiry
-- 2343 msecs since expiry
-- 2356 msecs since expiry
-- 2376 msecs since expiry
-- 2376 msecs since expiry
-- 2379 msecs since expiry
-- 2386 msecs since expiry
-- 2416 msecs since expiry
-- 2426 msecs since expiry
-- 2433 msecs since expiry
-- 2449 msecs since expiry
-- 2466 msecs since expiry
-- 2479 msecs since expiry
-- 2509 msecs since expiry
-- 2516 msecs since expiry
-- 2523 msecs since expiry
-- 2529 msecs since expiry
-- 2536 msecs since expiry
-- 2539 msecs since expiry
-- 2539 msecs since expiry
-- 2543 msecs since expiry
-- 2553 msecs since expiry
-- 2559 msecs since expiry
-- 2569 msecs since expiry
-- 2593 msecs since expiry
-- 2609 msecs since expiry
-- 2616 msecs since expiry
-- 2623 msecs since expiry
-- 2629 msecs since expiry
-- 2656 msecs since expiry
-- 2659 msecs since expiry
-- 2673 msecs since expiry
-- 2716 msecs since expiry
-- 2766 msecs since expiry
-- 2773 msecs since expiry
-- 2773 msecs since expiry
-- 2776 msecs since expiry
-- 2783 msecs since expiry
-- 2789 msecs since expiry
-- 2823 msecs since expiry
-- 2843 msecs since expiry
-- 2879 msecs since expiry
-- 2879 msecs since expiry
-- 2886 msecs since expiry
-- 2899 msecs since expiry
-- 2919 msecs since expiry
-- 2926 msecs since expiry
-- 2949 msecs since expiry
-- 2956 msecs since expiry
-- 2956 msecs since expiry
-- 2956 msecs since expiry
-- 2979 msecs since expiry
-- 2983 msecs since expiry
-- 2983 msecs since expiry
-- 2986 msecs since expiry
-- 3033 msecs since expiry
-- 3039 msecs since expiry
-- 3049 msecs since expiry
-- 3066 msecs since expiry
-- 3086 msecs since expiry
-- 3126 msecs since expiry
-- 3129 msecs since expiry
-- 3156 msecs since expiry
-- 3169 msecs since expiry
-- 3173 msecs since expiry
-- 3176 msecs since expiry
-- 3226 msecs since expiry
-- 3253 msecs since expiry
-- 3266 msecs since expiry
-- 3269 msecs since expiry
-- 3279 msecs since expiry
-- 3279 msecs since expiry
-- 3299 msecs since expiry
-- 3323 msecs since expiry
-- 3323 msecs since expiry
-- 3336 msecs since expiry
-- 3336 msecs since expiry
-- 3363 msecs since expiry
-- 3366 msecs since expiry
-- 3396 msecs since expiry
-- 3399 msecs since expiry
-- 3409 msecs since expiry
-- 3449 msecs since expiry
-- 3463 msecs since expiry
-- 3466 msecs since expiry
-- 3499 msecs since expiry
-- 0 msecs since expiry
-- 199 msecs since expiry
-- 269 msecs since expiry
-- 656 msecs since expiry
-- 689 msecs since expiry
-- 746 msecs since expiry
-- 789 msecs since expiry
-- 816 msecs since expiry
-- 819 msecs since expiry
-- 819 msecs since expiry
-- 839 msecs since expiry
-- 853 msecs since expiry
-- 873 msecs since expiry
-- 876 msecs since expiry
-- 889 msecs since expiry
-- 896 msecs since expiry
-- 896 msecs since expiry
-- 916 msecs since expiry
-- 936 msecs since expiry
-- 956 msecs since expiry
-- 969 msecs since expiry
-- 979 msecs since expiry
-- 983 msecs since expiry
-- 999 msecs since expiry
-- 999 msecs since expiry
-- 1006 msecs since expiry
-- 1006 msecs since expiry
-- 1053 msecs since expiry
-- 1053 msecs since expiry
-- 1063 msecs since expiry
-- 1066 msecs since expiry
-- 1073 msecs since expiry
-- 1079 msecs since expiry
-- 1083 msecs since expiry
-- 1089 msecs since expiry
-- 1096 msecs since expiry
-- 1133 msecs since expiry
-- 1139 msecs since expiry
-- 1156 msecs since expiry
-- 1163 msecs since expiry
-- 1166 msecs since expiry
-- 1179 msecs since expiry
-- 1183 msecs since expiry
-- 1189 msecs since expiry
-- 1196 msecs since expiry
-- 1199 msecs since expiry
-- 1203 msecs since expiry
-- 1226 msecs since expiry
-- 1226 msecs since expiry
-- 1253 msecs since expiry
-- 1263 msecs since expiry
-- 1279 msecs since expiry
-- 1296 msecs since expiry
-- 1339 msecs since expiry
-- 1366 msecs since expiry
-- 1373 msecs since expiry
-- 1383 msecs since expiry
-- 1396 msecs since expiry
-- 1399 msecs since expiry
-- 1403 msecs since expiry
-- 1406 msecs since expiry
-- 1426 msecs since expiry
-- 1433 msecs since expiry
-- 1439 msecs since expiry
-- 1439 msecs since expiry
-- 1443 msecs since expiry
-- 1449 msecs since expiry
-- 1449 msecs since expiry
-- 1469 msecs since expiry
-- 1469 msecs since expiry
-- 1473 msecs since expiry
-- 1476 msecs since expiry
-- 1489 msecs since expiry
-- 1506 msecs since expiry
-- 1509 msecs since expiry
-- 1519 msecs since expiry
-- 1543 msecs since expiry
-- 1543 msecs since expiry
-- 1559 msecs since expiry
-- 1569 msecs since expiry
-- 1573 msecs since expiry
-- 1593 msecs since expiry
-- 1626 msecs since expiry
-- 1626 msecs since expiry
-- 1633 msecs since expiry
-- 1639 msecs since expiry
-- 1639 msecs since expiry
-- 1653 msecs since expiry
-- 1683 msecs since expiry
-- 1689 msecs since expiry
-- 1706 msecs since expiry
-- 1706 msecs since expiry
-- 1719 msecs since expiry
-- 1729 msecs since expiry
-- 1739 msecs since expiry
-- 1776 msecs since expiry
-- 1799 msecs since expiry
-- 1799 msecs since expiry
-- 1803 msecs since expiry
-- 1806 msecs since expiry
-- 1833 msecs since expiry
-- 1843 msecs since expiry
-- 1879 msecs since expiry
-- 1883 msecs since expiry
-- 1886 msecs since expiry
-- 1893 msecs since expiry
-- 1893 msecs since expiry
-- 1893 msecs since expiry
-- 1903 msecs since expiry
-- 1949 msecs since expiry
-- 1959 msecs since expiry
-- 1986 msecs since expiry
-- 2016 msecs since expiry
-- 2016 msecs since expiry
-- 2026 msecs since expiry
-- 2053 msecs since expiry
-- 2063 msecs since expiry
-- 2066 msecs since expiry
-- 2083 msecs since expiry
-- 2109 msecs since expiry
-- 2113 msecs since expiry
-- 2123 msecs since expiry
-- 2139 msecs since expiry
-- 2159 msecs since expiry
-- 2186 msecs since expiry
-- 2203 msecs since expiry
-- 2206 msecs since expiry
-- 2223 msecs since expiry
-- 2226 msecs since expiry
-- 2233 msecs since expiry
-- 2246 msecs since expiry
-- 2256 msecs since expiry
-- 2266 msecs since expiry
-- 2289 msecs since expiry
-- 2306 msecs since expiry
-- 2346 msecs since expiry
-- 2359 msecs since expiry
-- 2379 msecs since expiry
-- 2379 msecs since expiry
-- 2383 msecs since expiry
-- 2389 msecs since expiry
-- 2419 msecs since expiry
-- 2429 msecs since expiry
-- 2436 msecs since expiry
-- 2453 msecs since expiry
-- 2469 msecs since expiry
-- 2483 msecs since expiry
-- 2513 msecs since expiry
-- 2519 msecs since expiry
-- 2526 msecs since expiry
-- 2533 msecs since expiry
-- 2539 msecs since expiry
-- 2543 msecs since expiry
-- 2543 msecs since expiry
-- 2546 msecs since expiry
-- 2556 msecs since expiry
-- 2563 msecs since expiry
-- 2573 msecs since expiry
-- 2596 msecs since expiry
-- 2613 msecs since expiry
-- 2619 msecs since expiry
-- 2626 msecs since expiry
-- 2633 msecs since expiry
-- 2659 msecs since expiry
-- 2663 msecs since expiry
-- 2676 msecs since expiry
-- 2719 msecs since expiry
-- 2769 msecs since expiry
-- 2776 msecs since expiry
-- 2776 msecs since expiry
-- 2779 msecs since expiry
-- 2786 msecs since expiry
-- 2793 msecs since expiry
-- 2826 msecs since expiry
-- 2846 msecs since expiry
-- 2883 msecs since expiry
-- 2883 msecs since expiry
-- 2889 msecs since expiry
-- 2903 msecs since expiry
-- 2923 msecs since expiry
-- 2929 msecs since expiry
-- 2956 msecs since expiry
-- 2959 msecs since expiry
-- 2959 msecs since expiry
-- 2959 msecs since expiry
-- 2983 msecs since expiry
-- 2986 msecs since expiry
-- 2986 msecs since expiry
-- 2989 msecs since expiry
-- 3036 msecs since expiry
-- 3043 msecs since expiry
-- 3053 msecs since expiry
-- 3069 msecs since expiry
-- 3089 msecs since expiry
-- 3129 msecs since expiry
-- 3133 msecs since expiry
-- 3159 msecs since expiry
-- 3173 msecs since expiry
-- 3176 msecs since expiry
-- 3179 msecs since expiry
-- 3229 msecs since expiry
-- 3256 msecs since expiry
-- 3269 msecs since expiry
-- 3273 msecs since expiry
-- 3283 msecs since expiry
-- 3283 msecs since expiry
-- 3303 msecs since expiry
-- 3326 msecs since expiry
-- 3326 msecs since expiry
-- 3339 msecs since expiry
-- 3339 msecs since expiry
-- 3366 msecs since expiry
-- 3369 msecs since expiry
-- 3399 msecs since expiry
-- 3403 msecs since expiry
-- 3413 msecs since expiry
-- 3453 msecs since expiry
-- 3466 msecs since expiry
-- 3469 msecs since expiry
-- 3 msecs since expiry
20-Feb-2024 10:46:30.324 REFUSED unexpected RCODE resolving 'cloud.Jensen.com.cn/NS/IN': 39.96.153.53#53
-- 0 msecs since expiry
-- 0 msecs since expiry
20-Feb-2024 10:46:30.524 REFUSED unexpected RCODE resolving 'cloud.Jensen.com.cn/NS/IN': 39.96.153.32#53
-- 0 msecs since expiry
```
So, there are multiple issues circling around the TTL-based cleaning.March 2024 (9.16.49, 9.16.49-S1, 9.18.25, 9.18.25-S1, 9.19.22)