ISC Open Source Projects issueshttps://gitlab.isc.org/groups/isc-projects/-/issues2023-09-01T09:16:46Zhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4266Documentation: Please document the "lifetime" options for dnssec policy keys ...2023-09-01T09:16:46ZDan MahoneyDocumentation: Please document the "lifetime" options for dnssec policy keys statements.Looking through the ARM, see timing options for the key lifetimes such as:
```
keys {
ksk key-directory lifetime unlimited algorithm rsasha256 2048;
zsk lifetime P30D algorithm 8;
csk lifetime P6MT12H3M15S algorithm ecdsa25...Looking through the ARM, see timing options for the key lifetimes such as:
```
keys {
ksk key-directory lifetime unlimited algorithm rsasha256 2048;
zsk lifetime P30D algorithm 8;
csk lifetime P6MT12H3M15S algorithm ecdsa256;
};
```
And while the ARM spells out what that last (weird) key argument does, there's no link to what the actual format of the lifetime statement is -- what the P and T stand for (also, nothing in our KB's.)
named-checkconf also does not emit warnings if you just say "lifetime 30d" so I cannot tell if this behavior is somehow different.
In fact, I cannot at all find a syntax for the "keys" sub-statement for dnssec-policy in the ARM.
Finally, in the rendered version of the ARM, there's a statement that says:
"Here is an example (for illustration purposes only) of some possible entries in a [keys] list:", and that links to the
wrong "keys" statement. (It links to https://bind9.readthedocs.io/en/v9.18.18/reference.html#namedconf-statement-keys
where it specifically says "these are NOT the dnssec-policy keys")September 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/4263Deprecate the "dnssec-must-be-secure" feature2023-12-07T10:23:40ZOndřej SurýDeprecate the "dnssec-must-be-secure" featureThe `dnssec-must-be-secure` feature was added in the early days of BIND 9 and DNSSEC and it makes sense only as a debugging feature. Remove it to simplify the code.
See #4482 for the removal issue.The `dnssec-must-be-secure` feature was added in the early days of BIND 9 and DNSSEC and it makes sense only as a debugging feature. Remove it to simplify the code.
See #4482 for the removal issue.September 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)Ondřej SurýOndřej Surýhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4262invoke pytest runner with `make check`2023-10-30T15:13:35ZTom Krizekinvoke pytest runner with `make check`Switch `make check` to use the pytest system test runner instead of using the legacy one.
- keep `TESTS=x make -e check` working if possible
- control how many processes to spawn with `make -j` - this might be a bit more tricky, but it ...Switch `make check` to use the pytest system test runner instead of using the legacy one.
- keep `TESTS=x make -e check` working if possible
- control how many processes to spawn with `make -j` - this might be a bit more tricky, but it should be [possible](https://stackoverflow.com/questions/5303553/gnu-make-extracting-argument-to-j-within-makefile/67247743#67247743) to detect the value of `-j` and then pass it to the `pytest -n` invocation
Related #3810November 2023 (9.16.45, 9.16.45-S1, 9.18.20, 9.18.20-S1, 9.19.18)Tom KrizekTom Krizekhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4261Detect unexpected files created during system test run with pytest runner2023-12-06T15:51:28ZTom KrizekDetect unexpected files created during system test run with pytest runner> we should have a check whether the named haven't produced any unexpected files. But that's only tangential to cleaning up the cruft. Perhaps this can take a form of .gitignore(?) with expected files for each test.
https://gitlab.isc.o...> we should have a check whether the named haven't produced any unexpected files. But that's only tangential to cleaning up the cruft. Perhaps this can take a form of .gitignore(?) with expected files for each test.
https://gitlab.isc.org/isc-projects/bind9/-/issues/4246#note_395492
Related #3810Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4260initial AXFR fails in the inline test2023-11-15T13:59:57ZTom Krizekinitial AXFR fails in the inline testIn [`system:gcc:tarball`](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3584073) the `inline` test failed:
```
2023-08-15 00:17:47 INFO:inline I:inline_tmp_dffqwzo0:checking that the zone is signed on initial transfer (3)
2023-08...In [`system:gcc:tarball`](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3584073) the `inline` test failed:
```
2023-08-15 00:17:47 INFO:inline I:inline_tmp_dffqwzo0:checking that the zone is signed on initial transfer (3)
2023-08-15 00:17:47 INFO:inline Zone contains no DNSSEC keys
2023-08-15 00:17:48 INFO:inline Zone contains no DNSSEC keys
2023-08-15 00:17:49 INFO:inline Zone contains no DNSSEC keys
2023-08-15 00:17:50 INFO:inline Zone contains no DNSSEC keys
2023-08-15 00:17:51 INFO:inline Zone contains no DNSSEC keys
2023-08-15 00:17:52 INFO:inline Zone contains no DNSSEC keys
2023-08-15 00:17:53 INFO:inline Zone contains no DNSSEC keys
2023-08-15 00:17:54 INFO:inline Zone contains no DNSSEC keys
2023-08-15 00:17:55 INFO:inline Zone contains no DNSSEC keys
2023-08-15 00:17:56 INFO:inline Zone contains no DNSSEC keys
2023-08-15 00:17:56 INFO:inline I:inline_tmp_dffqwzo0:failed
2023-08-15 00:17:56 INFO:inline I:inline_tmp_dffqwzo0:checking expired signatures are updated on load (4)
2023-08-15 00:17:56 INFO:inline I:inline_tmp_dffqwzo0:checking removal of private type record via 'rndc signing -clear' (5)
2023-08-15 00:18:06 INFO:inline I:inline_tmp_dffqwzo0:failed
2023-08-15 00:18:06 INFO:inline I:inline_tmp_dffqwzo0:checking private type was properly signed (6)
2023-08-15 00:18:06 INFO:inline I:inline_tmp_dffqwzo0:failed
```
The dig output says the transfer failed:
```
; <<>> DiG 9.19.17-dev <<>> +tcp +dnssec -p 18989 @10.53.0.3 bits. AXFR
; (1 server found)
;; global options: +cmd
; Transfer failed.
```
In [ns3/named.run](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3584073/artifacts/raw/bind-9.19.17-dev/bin/tests/system/inline_tmp_dffqwzo0/ns3/named.run?inline=false), there's a message that `zone transfer setup failed`:
```
15-Aug-2023 00:17:54.716 client @0x7f7a78c46000 (no-peer): allocate new client
15-Aug-2023 00:17:54.716 client @0x7f7a78c46000 10.53.0.1#35491: TCP request
15-Aug-2023 00:17:54.716 client @0x7f7a78c46000 10.53.0.1#35491: using view '_default'
15-Aug-2023 00:17:54.716 client @0x7f7a78c46000 10.53.0.1#35491: request is not signed
15-Aug-2023 00:17:54.716 client @0x7f7a78c46000 10.53.0.1#35491: recursion not available (recursion not enabled for view)
15-Aug-2023 00:17:54.716 query client=0x7f7a78c46000 thread=0x7f7a7a47e700(<unknown-query>): ns_query_start
15-Aug-2023 00:17:54.716 client @0x7f7a78c46000 10.53.0.1#35491 (bits): AXFR request
15-Aug-2023 00:17:54.716 client @0x7f7a78c46000 10.53.0.1#35491 (bits): zone transfer setup failed
15-Aug-2023 00:17:54.716 client @0x7f7a78c46000 10.53.0.1#35491 (bits): reset client
15-Aug-2023 00:17:54.716 query client=0x7f7a78c46000 thread=0x7f7a7a47e700(bits/AXFR): query_reset
```November 2023 (9.16.45, 9.16.45-S1, 9.18.20, 9.18.20-S1, 9.19.18)https://gitlab.isc.org/isc-projects/bind9/-/issues/4259Statschannel leftovers2023-08-31T15:10:25ZMark AndrewsStatschannel leftoversJob [#3583906](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3583906) failed for 5097911f510851c9dfee235e6450caa585cfae89:
```
$ if git rev-parse > /dev/null 2>&1; then ( ! grep "^I:.*:file.*not removed$" *.log ); fi
statschannel.log:...Job [#3583906](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3583906) failed for 5097911f510851c9dfee235e6450caa585cfae89:
```
$ if git rev-parse > /dev/null 2>&1; then ( ! grep "^I:.*:file.*not removed$" *.log ); fi
statschannel.log:I:statschannel:file statschannel/bind9.xsl.1 not removed
statschannel.log:I:statschannel:file statschannel/bind9.xsl.2 not removed
```September 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)https://gitlab.isc.org/isc-projects/bind9/-/issues/4255suspicious message IDs and lost answers in a long DoT stream2023-09-06T09:26:12ZPetr Špačekpspacek@isc.orgsuspicious message IDs and lost answers in a long DoT stream### Summary
dnsperf in DoT mode sometimes reports unexpected message IDs and timeouts when talking to BIND, even in test runs _without_ retransmits, timeouts, and message ID reuse.
### Versions used
* ~"Affects v9.18": 5a1d465 (fix fo...### Summary
dnsperf in DoT mode sometimes reports unexpected message IDs and timeouts when talking to BIND, even in test runs _without_ retransmits, timeouts, and message ID reuse.
### Versions used
* ~"Affects v9.18": 5a1d465 (fix for #4242)
* Not affected/not reproducible on: ~"v9.19"
* dnsperf: 2.13.0 - commit 00f5a188aaabab7cc3b2fffa016944c20402b5e6
### Steps to reproduce
* Enable DoT
* Run BIND with `SSLKEYLOGFILE=/tmp/tlskeys` env variable to capture TLS keys so we can open PCAPs later
* Capture PCAP: `sudo tcpdump -i lo -w /tmp/pcap 'port 853'`
* Run `dnsperf` couple times: `dnsperf -d /tmp/qlist -l0.1 -q 1000 -m dot -s 127.0.0.1 -p 853 -c1`
* Watch for "response with an unexpected (maybe timed out) id" and "[Timeout]" messages
Above are minimal parameters for `-q` and `-l` which allow me to reproduce the problem on my machine. Bump `-q` and `-l` as needed.
### What is the current *bug* behavior?
```
$ dnsperf -d /tmp/qlist -l0.1 -q 1000 -m dot -s 127.0.0.1 -p 853 -c1
DNS Performance Testing Tool
Version 2.13.0
[Status] Command line: dnsperf -d /tmp/qlist -l0.1 -q 1000 -m dot -s 127.0.0.1 -p 853 -c1
[Status] Sending queries (to 127.0.0.1:853)
[Status] Started at: Thu Aug 10 14:59:52 2023
[Status] Stopping after 0.100000 seconds
```
:warning:
```
Warning: received a response with an unexpected (maybe timed out) id: 2014
[Timeout] Query timed out: msg id 2015
```
:warning:
```
[Status] Testing complete (time limit)
Statistics:
Queries sent: 4160
Queries completed: 4159 (99.98%)
Queries lost: 1 (0.02%)
Response codes: NOERROR 4159 (100.00%)
Average packet size: request 35, response 51
Run time (s): 0.106694
Queries per second: 38980.636212
Average Latency (s): 0.016029 (min 0.000095, max 0.027541)
Latency StdDev (s): 0.009930
Connection Statistics:
Reconnections: 0 (0.00% of 1 connections)
Average Latency (s): 0.002322 (min 0.002322, max 0.002322)
```
### What is the expected *correct* behavior?
No reports of unexpected or lost messages.
### Relevant configuration files
* [named.conf](/uploads/ae631f3b0be630a7ba2553ad9bdd8d10/named.conf)
* [rsasha1.example.db](/uploads/7373e5f88373379497f26597f5b8bb44/rsasha1.example.db)
### Relevant logs and/or screenshots
* TLS key log file for Wireshark: [tlskeys](/uploads/5930c7282d7dbb2421659ce368fde356/tlskeys)
* PCAP: [pcap](/uploads/930ae4a36df9714331294549362eaabd/pcap)
Look for message IDs 2014 and 2015. They should be there just once in each direction, but 2014 is duplicated on *response from BIND* and 2015 has missing response.
Helpful wireshark display filters:
* `dns.id == 2014`
* `dns.id == 2015`
* `dns.retransmission == 1`September 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)Artem BoldarievArtem Boldarievhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4254zonechecks died mid test2023-08-10T13:14:02ZMark Andrewszonechecks died mid testJob [#3577320](https://gitlab.isc.org/isc-private/bind9/-/jobs/3577320) failed for isc-private/bind9@51f47c4d045f50dd0e72573cd573a5031261fed4:
zonechecks died mid test possibly fallout from setting `-e`.
```
2023-08-10 07:15:03 INFO:zo...Job [#3577320](https://gitlab.isc.org/isc-private/bind9/-/jobs/3577320) failed for isc-private/bind9@51f47c4d045f50dd0e72573cd573a5031261fed4:
zonechecks died mid test possibly fallout from setting `-e`.
```
2023-08-10 07:15:03 INFO:zonechecks I:zonechecks_tmp_w6aef5m0:checking that we detect a NS which looks like a AAAA record (fail)
2023-08-10 07:15:03 INFO:zonechecks I:zonechecks_tmp_w6aef5m0:checking that we detect a NS which looks like a AAAA record (warn=default)
2023-08-10 07:15:03 INFO:zonechecks I:zonechecks_tmp_w6aef5m0:checking that we detect a NS which looks like a AAAA record (ignore)
2023-08-10 07:15:03 INFO:zonechecks I:zonechecks_tmp_w6aef5m0:checking 'rdnc zonestatus' output
2023-08-10 07:15:03 INFO:zonechecks I:zonechecks_tmp_w6aef5m0:ns1 zone reload queued
```Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4252add predictable path names for pytest runner results2023-09-20T14:27:25ZTom Krizekadd predictable path names for pytest runner results> The particular missing feature that I've used the most is that the old runner puts test results in a predictable place. Let's say I want to run the resolver test and examine the contents of `resolver/ns3/named.run` - with the legacy ru...> The particular missing feature that I've used the most is that the old runner puts test results in a predictable place. Let's say I want to run the resolver test and examine the contents of `resolver/ns3/named.run` - with the legacy runner I can just do that, I don't have to figure out that this time the results are in `resolver_tmp_pnxahq01`. If there's a way to tell pytest to put the test results in the test directory - or even just in the same place for two consecutive runs - I haven't discovered it. I often script these things; having the directory name change on every run makes that unnecessarily difficult, so I switch back to `legacy.run.sh` and then it's fine.
https://gitlab.isc.org/isc-projects/bind9/-/issues/4246#note_394780
---
Add a symlink with a predictable path to the latest test run(s). It's a convenience thing that should allow easier inspection of the results.
A single system test may consist of multiple modules which are run separately, e.g. `doth` has:
```
<Module doth/tests_gnutls.py>
<Module doth/tests_sh_doth.py>
<Module doth/tests_sslyze.py>
```
which may be executed independently and end up in different temporary directories. The symlink should include a module name/identifier to differentiate between them.
I think creating a directory like `bin/tests/system/_last_test_run` where the stable symlinks could be placed should work well.
Related #3810, #4251September 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)Tom KrizekTom Krizekhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4250remove support for running python system tests with legacy test runner2023-10-04T13:26:28ZTom Krizekremove support for running python system tests with legacy test runnerCurrently, python system tests must be designed in a way that is compatible with two different modes of operation - the legacy runner and the pytest runner. Their design philosophies are sufficiently different to induce a lot of friction...Currently, python system tests must be designed in a way that is compatible with two different modes of operation - the legacy runner and the pytest runner. Their design philosophies are sufficiently different to induce a lot of friction points and various compatibility issues.
In order for us to be able efficiently write and extend the capabilities of the pytest runner as well as take advantage of all its possibilities, it's not feasible to keep the compatibility layer which enables the python tests to be executed with the legacy runner.
Instead, python tests should be exclusively executed by the pytest runner, providing a predictable environment and behavior.
Legacy runner can still be used to run shell system tests.
Related #3810November 2023 (9.16.45, 9.16.45-S1, 9.18.20, 9.18.20-S1, 9.19.18)Tom KrizekTom Krizekhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4249compile test binaries/libraries during make2023-09-08T14:05:18ZTom Krizekcompile test binaries/libraries during makeCurrently, test files in `bin/tests/system` which need to be compiled are not part of the typical `make` invocation. Instead, they're compiled when `make check` is invoked.
While this worked well with the legacy test runner, it makes th...Currently, test files in `bin/tests/system` which need to be compiled are not part of the typical `make` invocation. Instead, they're compiled when `make check` is invoked.
While this worked well with the legacy test runner, it makes things needlessly complicated for the pytest runner, and prevents use-cases such as running out-of-tree tests easily.
Compile the required test files as `noinst_` instead of `check_` to ensure they're compiled and available after `make` invocation.
Related #4246, #3810September 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)Tom KrizekTom Krizekhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4247Autosign resigning too fast after dnssec-settime calls2024-03-21T15:29:09ZMark AndrewsAutosign resigning too fast after dnssec-settime callsJob [#3570991](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3570991) failed for ce1db0017ee498c3c9e0d632b83113a344d25dda:
I've seen autosign fail in the ZSK roll test several times and it appears to be because dnssec-signzone is bei...Job [#3570991](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3570991) failed for ce1db0017ee498c3c9e0d632b83113a344d25dda:
I've seen autosign fail in the ZSK roll test several times and it appears to be because dnssec-signzone is being called in the same second as dnssec-settime. dnssec-signzone uses the phase `is set and is in the past` as the determinant of when things change. Should this be made `is set and is now or in the past`? Which behaviour is consistent with named's behaviour? Which behaviour is less error prone? If we don't change the behaviour we need to add sleeps to ensure the tests succeed.
```
-S This option enables smart signing, which instructs dnssec-signzone to search the key repository for keys that match the zone
being signed, and to include them in the zone if appropriate.
When a key is found, its timing metadata is examined to determine how it should be used, according to the following rules.
Each successive rule takes priority over the prior ones:
If no timing metadata has been set for the key, the key is published in the zone and used to sign the zone.
If the key's publication date is set and is in the past, the key is published in the zone.
If the key's activation date is set and is in the past, the key is published (regardless of publication date) and used to
sign the zone.
If the key's revocation date is set and is in the past, and the key is published, then the key is revoked, and the revoked
key is used to sign the zone.
If either the key's unpublication or deletion date is set and in the past, the key is NOT published or used to sign the
zone, regardless of any other metadata.
If the key's sync publication date is set and is in the past, synchronization records (type CDS and/or CDNSKEY) are
created.
If the key's sync deletion date is set and is in the past, synchronization records (type CDS and/or CDNSKEY) are removed.
```
```
2023-08-08 00:59:52 INFO:autosign I:autosign_tmp_x_6wkr8z:preparing ZSK roll
2023-08-08 00:59:52 INFO:autosign I:autosign_tmp_x_6wkr8z:ns1 A zone reload and thaw was started.
2023-08-08 00:59:52 INFO:autosign I:autosign_tmp_x_6wkr8z:ns1 Check the logs to see the result.
2023-08-08 00:59:56 INFO:autosign I:autosign_tmp_x_6wkr8z:revoking key to duplicated key ID
2023-08-08 00:59:56 INFO:autosign dnssec-settime: warning: Permissions on the file ns2/Kbar.+013+59973.private have changed from 0644 to 0600 as a result of this operation.
2023-08-08 00:59:56 INFO:autosign I:autosign_tmp_x_6wkr8z:ns2 A zone reload and thaw was started.
2023-08-08 00:59:56 INFO:autosign I:autosign_tmp_x_6wkr8z:ns2 Check the logs to see the result.
2023-08-08 00:59:56 INFO:autosign I:autosign_tmp_x_6wkr8z:waiting for changes to take effect
2023-08-08 01:00:01 INFO:autosign I:autosign_tmp_x_6wkr8z:checking former standby key 5259 is now active (53)
2023-08-08 01:00:01 INFO:autosign I:autosign_tmp_x_6wkr8z:failed
```
```
echo_i "preparing ZSK roll"
starttime=$($PERL -e 'print time(), "\n";')
oldfile=$(cat active.key)
oldid=$(keyfile_to_key_id "$(cat active.key)")
newfile=$(cat standby.key)
newid=$(keyfile_to_key_id "$(cat standby.key)")
$SETTIME -K ns1 -I now -D now+25 $oldfile > settime.out.test$n.1 || ret=1
$SETTIME -K ns1 -i 0 -S $oldfile $newfile > settime.out.test$n.2 || ret=1
# note previous zone serial number
oldserial=$($DIG $DIGOPTS +short soa . @10.53.0.1 | awk '{print $3}')
($RNDCCMD 10.53.0.1 freeze . 2>&1 | sed 's/^/ns1 /' | cat_i) || ret=1
cp ns1/root.db.signed ns1/root.db.1
$SIGNER -S -o . -O full -K ns1 -f ns1/root.db.signed ns1/root.db.1 > signing.root.out$n 2>&1 || ret=1
($RNDCCMD 10.53.0.1 thaw . 2>&1 | sed 's/^/ns1 /' | cat_i) || ret=1
sleep 4
echo_i "revoking key to duplicated key ID"
$SETTIME -R now -K ns2 Kbar.+013+59973.key > settime.out.test$n.3 || ret=1
($RNDCCMD 10.53.0.2 freeze bar. 2>&1 | sed 's/^/ns2 /' | cat_i) || ret=1
cp ns2/bar.db.signed ns2/bar.db
$SIGNER -S -o bar. -O full -K ns2 ns2/bar.db > signing.bar.out$n 2>&1 || ret=1
($RNDCCMD 10.53.0.2 thaw bar. 2>&1 | sed 's/^/ns2 /' | cat_i) || ret=1
echo_i "waiting for changes to take effect"
sleep 5
echo_i "checking former standby key $newid is now active ($n)"
ret=0
$DIG $DIGOPTS dnskey . @10.53.0.1 > dig.out.ns1.test$n || ret=1
grep 'RRSIG.*'" $newid "'\. ' dig.out.ns1.test$n > /dev/null || ret=1
n=$((n + 1))
if [ $ret != 0 ]; then echo_i "failed"; fi
status=$((status + ret))
```
```
08-Aug-2023 00:59:52.469 allocate new control connection
08-Aug-2023 00:59:52.469 received control channel command 'freeze .'
08-Aug-2023 00:59:52.469 loop exclusive mode: starting
08-Aug-2023 00:59:52.469 loop exclusive mode: started
08-Aug-2023 00:59:52.469 zone_dump: zone ./IN: enter
08-Aug-2023 00:59:52.469 loop exclusive mode: ending
08-Aug-2023 00:59:52.469 loop exclusive mode: ended
08-Aug-2023 00:59:52.469 freezing zone './IN': success
08-Aug-2023 00:59:52.469 freeing control connection
08-Aug-2023 00:59:52.497 dump_done: zone ./IN: enter
08-Aug-2023 00:59:52.497 zone_journal_compact: zone ./IN: target journal size 7662
08-Aug-2023 00:59:52.497 zone ./IN: dns_journal_compact: success
08-Aug-2023 00:59:52.509 allocate new control connection
08-Aug-2023 00:59:52.509 received control channel command 'thaw .'
08-Aug-2023 00:59:52.509 loop exclusive mode: starting
08-Aug-2023 00:59:52.509 loop exclusive mode: started
08-Aug-2023 00:59:52.509 zone ./IN: starting load
08-Aug-2023 00:59:52.509 zone_startload: zone ./IN: enter
08-Aug-2023 00:59:52.509 loop exclusive mode: ending
08-Aug-2023 00:59:52.509 loop exclusive mode: ended
08-Aug-2023 00:59:52.509 thawing zone './IN': success
08-Aug-2023 00:59:52.509 freeing control connection
08-Aug-2023 00:59:52.509 zone_loaddone: zone ./IN: enter
08-Aug-2023 00:59:52.509 zone ./IN: number of nodes in database: 5
08-Aug-2023 00:59:52.509 zone ./IN: loaded; checking validity
08-Aug-2023 00:59:52.509 dns_zone_verifydb: zone ./IN: enter
08-Aug-2023 00:59:52.509 zone ./IN: zone serial (2000042101) unchanged. zone may fail to transfer to secondaries.
08-Aug-2023 00:59:52.509 zone ./IN: replacing zone database
08-Aug-2023 00:59:52.509 calling free_rbtdb(.)
08-Aug-2023 00:59:52.509 done free_rbtdb(.)
08-Aug-2023 00:59:52.509 zone ./IN: loaded serial 2000042101 (DNSSEC signed)
08-Aug-2023 00:59:52.509 zone_postload: zone ./IN: done
08-Aug-2023 00:59:52.509 zone__settimer: zone ./IN: enter
```April 2024 (9.16.50, 9.16.50-S1, 9.18.26, 9.18.26-S1, 9.19.23)https://gitlab.isc.org/isc-projects/bind9/-/issues/4246use pytest runner to run system tests for out-of-tree tests2023-08-21T14:19:40ZTom Krizekuse pytest runner to run system tests for out-of-tree testsImplement support to run system tests for out of tree builds with the pytest system test runner.
This is one of the last uses of the legacy runner in our CI (along with OpenBSD and CentOS7 on ~"v9.18").
Related #3810Implement support to run system tests for out of tree builds with the pytest system test runner.
This is one of the last uses of the legacy runner in our CI (along with OpenBSD and CentOS7 on ~"v9.18").
Related #3810September 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)Tom KrizekTom Krizekhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4245Incorrect return values in rpz's addr and drop functions2023-08-31T15:03:41ZMark AndrewsIncorrect return values in rpz's addr and drop functions`addr` and `drop` should `return 0` after calling `setret` rather than `return 1` as the error has already been logged. This is fallout from the `set -e` changes.`addr` and `drop` should `return 0` after calling `setret` rather than `return 1` as the error has already been logged. This is fallout from the `set -e` changes.September 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)https://gitlab.isc.org/isc-projects/bind9/-/issues/4243_wait_for_stats errors not detected in ixfr system test2023-08-08T00:43:48ZMark Andrews_wait_for_stats errors not detected in ixfr system testSeptember 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)Mark AndrewsMark Andrewshttps://gitlab.isc.org/isc-projects/stork/-/issues/1125Update AUTHORS file2023-10-02T10:17:54ZSlawek FigielUpdate AUTHORS fileThe issue was found by @marcin during [1.12 sanity checks](https://gitlab.isc.org/isc-projects/stork/-/issues/1122#note_393096).
The AUTHORS file may require an update. For example: there are new features, like hooks, that Slawek implem...The issue was found by @marcin during [1.12 sanity checks](https://gitlab.isc.org/isc-projects/stork/-/issues/1122#note_393096).
The AUTHORS file may require an update. For example: there are new features, like hooks, that Slawek implemented but aren't listed under his name. There can be more.1.13Tomek MrugalskiTomek Mrugalskihttps://gitlab.isc.org/isc-projects/bind9/-/issues/4242[CVE-2023-4236] named may terminate unexpectedly under high DNS-over-TLS quer...2023-10-18T15:59:10ZCathy Almond[CVE-2023-4236] named may terminate unexpectedly under high DNS-over-TLS query load| Quick Links | :link: |
| ------------------------ | ------------------------------------ |
| Incident Manager: | @pspacek |
| Deputy Incident Manager: | @gre...| Quick Links | :link: |
| ------------------------ | ------------------------------------ |
| Incident Manager: | @pspacek |
| Deputy Incident Manager: | @greg |
| Public Disclosure Date: | 2023-09-20 |
| CVSS Score: | [7.5][cvss_score] |
| Security Advisory: | isc-private/printing-press!66 |
| Mattermost Channel: | [CVE-2023-4236][mattermost_url] |
| Support Ticket: | Salesforce case \#1086 |
| Release Checklist: | #4289 |
| Post-mortem Etherpad: | [postmortem-2023-09][postmortem_url] |
[cvss_score]: https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H&version=3.1
[mattermost_url]: https://mattermost.isc.org/isc/channels/cve-2023-4236-crash-after-tls-connection-termination
[postmortem_url]: https://pad.isc.org/p/postmortem-2023-09
:bulb: **Click [here][checklist_explanations] (internal resource) for general information about the security incident handling process.**
[checklist_explanations]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations
### Earlier Than T-5
- [x] [:link:][step_deputy] **(IM)** Pick a Deputy Incident Manager
- [x] [:link:][step_respond] **(IM)** Respond to the bug reporter
- [x] [:link:][step_etherpad] **(IM)** Create an Etherpad for post-mortem
- [x] [:link:][step_public_mrs] **(SwEng)** Ensure there are no public merge requests which inadvertently disclose the issue
- [x] [:link:][step_assign_cve_id] **(IM)** [Assign a CVE identifier](#note_394286)
- [x] [:link:][step_note_cve_info] **(SwEng)** Update this issue with the assigned CVE identifier and the CVSS score: [MM](https://mattermost.isc.org/isc/pl/6bq1nynyaff68drzmt8bq8stda)
- [x] [:link:][step_versions_affected] **(SwEng)** Determine the range of product [versions affected](https://gitlab.isc.org/isc-projects/bind9/-/issues/4242#note_393519) (including the Subscription Edition)
- [X] [:link:][step_workarounds] **(SwEng)** Determine whether workarounds for the problem exist
- :no_entry_sign: ~~**(SwEng)** If necessary, coordinate with other parties~~
- [x] [:link:][step_earliest] **(Support)** Prepare and send out "earliest" notifications
- [x] [:link:][step_advisory_mr] **(Support)** [Create a merge request for the Security Advisory and include all readily available information in it](isc-private/printing-press!66)
- :no_entry_sign: ~~**(SwEng)** Prepare a private merge request containing a system test reproducing the problem~~
- :no_entry_sign: ~~**(SwEng)** Notify Support when a reproducer is ready~~
- [x] [:link:][step_code_analysis] **(SwEng)** Prepare a detailed explanation of the code flow triggering the problem: see [here](https://gitlab.isc.org/isc-private/bind9/-/merge_requests/565)
- [x] [:link:][step_fix_mr] **(SwEng)** Prepare a private merge request with the [fix](https://gitlab.isc.org/isc-private/bind9/-/merge_requests/565)
- [x] [:link:][step_review_fix] **(SwEng)** Ensure the merge request with the fix is reviewed and has no outstanding discussions
- [x] [:link:][step_review_docs] **(Support)** [Review the documentation changes introduced by the merge request with the fix](https://gitlab.isc.org/isc-private/bind9/-/merge_requests/565#note_400226)
- :no_entry_sign: ~~**(SwEng)** Prepare backports of the merge request addressing the problem for all affected (and still maintained) branches of a given product~~
- [x] [:link:][step_finish_advisory] **(Support)** Finish preparing the Security Advisory
- [x] [:link:][step_meta_issue] **(QA)** [Create (or update) the private issue containing links to fixes & reproducers for all CVEs fixed in a given release cycle](isc-private/bind9#70)
- [x] [:link:][step_changes] **(QA)** (BIND 9 only) Reserve a block of `CHANGES` placeholders once the complete set of vulnerabilities fixed in a given release cycle is determined
- [x] [:link:][step_merge_fixes] **(QA)** Merge the CVE fixes in CVE identifier order
- [x] [:link:][step_patches] **(QA)** Prepare a standalone patch for the last stable release of each affected (and still maintained) product branch
- [x] [:link:][step_asn_releases] **(QA)** Prepare ASN releases (as outlined in the Release Checklist)
### At T-5
- [x] [:link:][step_send_asn] **(Support)** Send ASN to eligible customers
- [x] [:link:][step_preannouncement] **(Support)** [(BIND 9 only) Send a pre-announcement email to the *bind-announce* mailing list to alert users that the upcoming release will include security fixes](isc-private/printing-press!67)
### At T-4
- [x] [:link:][step_verify_asn] **(Support)** [Verify that all ASN-eligible customers have received the notification email](https://mattermost.isc.org/isc/pl/384wo14bmprm3yhfcbyjrmdz5r)
### At T-1
- [x] [:link:][step_check_customers] **(Support)** [Verify that any new or reinstated customers have received the notification email](https://mattermost.isc.org/isc/pl/384wo14bmprm3yhfcbyjrmdz5r)
- [x] [:link:][step_packager_emails] **(First IM)** [Send notifications to OS packagers](isc-private/printing-press!68)
### On the Day of Public Disclosure
- [x] [:link:][step_clearance] **(IM)** [Grant Support clearance to proceed with public release](https://mattermost.isc.org/isc/pl/8jq9z43tx78xbyb8g6di44dpnh)
- [x] [:link:][step_publish] **(Support)** Publish the releases (as outlined in the release checklist)
- [x] [:link:][step_matrix] **(Support)** (BIND 9 only) Add the new CVEs to the vulnerability matrix in the Knowledge Base
- [x] [:link:][step_publish_advisory] **(Support)** [Bump Document Version for the Security Advisory and publish it in the Knowledge Base](https://kb.isc.org/docs/cve-2023-4236)
- [x] [:link:][step_notifications] **(First IM)** [Send notification emails to third parties](isc-private/printing-press!70)
- [x] [:link:][step_mitre] **(First IM)** [Advise MITRE about the disclosed CVEs](https://gitlab.isc.org/isc-private/printing-press/-/merge_requests/66#note_403632)
- [x] [:link:][step_merge_advisory] **(First IM)** Merge the Security Advisory merge request
- [x] [:link:][step_embargo_end] **(IM)** Inform original reporter (if external) that the security disclosure process is complete
- [x] [:link:][step_customers] **(Support)** Inform customers a fix has been released
### After Public Disclosure
- :no_entry_sign: [:link:][step_postmortem] **(First IM)** Organize post-mortem meeting and make sure it happens
- [x] [:link:][step_tickets] **(Support)** Close support tickets
- :no_entry_sign: [:link:][step_regression] **(QA)** Merge a regression test reproducing the bug into all affected (and still maintained) branches
[step_deputy]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#pick-a-deputy-incident-manager
[step_respond]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#respond-to-the-bug-reporter
[step_etherpad]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-an-etherpad-for-post-mortem
[step_public_mrs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-there-are-no-public-merge-requests-which-inadvertently-disclose-the-issue
[step_assign_cve_id]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#assign-a-cve-identifier
[step_note_cve_info]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-this-issue-with-the-assigned-cve-identifier-and-the-cvss-score
[step_versions_affected]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-the-range-of-product-versions-affected-including-the-subscription-edition
[step_workarounds]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-whether-workarounds-for-the-problem-exist
[step_coordinate]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#if-necessary-coordinate-with-other-parties
[step_earliest]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-and-send-out-earliest-notifications
[step_advisory_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-a-merge-request-for-the-security-advisory-and-include-all-readily-available-information-in-it
[step_reproducer_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-containing-a-system-test-reproducing-the-problem
[step_notify_support]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#notify-support-when-a-reproducer-is-ready
[step_code_analysis]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-detailed-explanation-of-the-code-flow-triggering-the-problem
[step_fix_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-with-the-fix
[step_review_fix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-the-merge-request-with-the-fix-is-reviewed-and-has-no-outstanding-discussions
[step_review_docs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#review-the-documentation-changes-introduced-by-the-merge-request-with-the-fix
[step_backports]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-backports-of-the-merge-request-addressing-the-problem-for-all-affected-and-still-maintained-branches-of-a-given-product
[step_finish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#finish-preparing-the-security-advisory
[step_meta_issue]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-or-update-the-private-issue-containing-links-to-fixes-reproducers-for-all-cves-fixed-in-a-given-release-cycle
[step_changes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-reserve-a-block-of-changes-placeholders-once-the-complete-set-of-vulnerabilities-fixed-in-a-given-release-cycle-is-determined
[step_merge_fixes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-cve-fixes-in-cve-identifier-order
[step_patches]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-standalone-patch-for-the-last-stable-release-of-each-affected-and-still-maintained-product-branch
[step_asn_releases]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-asn-releases-as-outlined-in-the-release-checklist
[step_send_asn]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-asn-to-eligible-customers
[step_preannouncement]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-send-a-pre-announcement-email-to-the-bind-announce-mailing-list-to-alert-users-that-the-upcoming-release-will-include-security-fixes
[step_verify_asn]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#verify-that-all-asn-eligible-customers-have-received-the-notification-email
[step_check_customers]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#verify-that-any-new-or-reinstated-customers-have-received-the-notification-email
[step_packager_emails]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notifications-to-os-packagers
[step_clearance]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#grant-support-clearance-to-proceed-with-public-release
[step_publish]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#publish-the-releases-as-outlined-in-the-release-checklist
[step_matrix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-add-the-new-cves-to-the-vulnerability-matrix-in-the-knowledge-base
[step_publish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bump-document-version-for-the-security-advisory-and-publish-it-in-the-knowledge-base
[step_notifications]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notification-emails-to-third-parties
[step_mitre]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#advise-mitre-about-the-disclosed-cves
[step_merge_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-security-advisory-merge-request
[step_embargo_end]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#inform-original-reporter-if-external-that-the-security-disclosure-process-is-complete
[step_customers]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#inform-customers-a-fix-has-been-released
[step_postmortem]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#organize-post-mortem-meeting-and-make-sure-it-happens
[step_tickets]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#close-support-tickets
[step_regression]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-a-regression-test-reproducing-the-bug-into-all-affected-and-still-maintained-branches
---
### Summary
From the customer:
I was running dnsperf in a test environment to try and find
parameters to get the highest QPS rate. Originally tested our
custom build on CentOS 7. After it started crashing I switched to stock
named on a Fedora 37 test VM to reproduce.
dnsperf example:
```
dnsperf -d /usr/share/dnsperf/queryfile-example-current -S 1 -s 192.168.122.239 -q 15000 -c 48 -T 32 -m dot
```
This or a similar command would often run without issues. While I
was testing I'd interrupt dnsperf if QPS was lower than a previous
run to tweak parameters. This would eventually cause named to crash.
### BIND version used
<will request this>
### Steps to reproduce
See above
### What is the current *bug* behavior?
`named` dies horribly: The assert as logged is:
The assert is
```
../../../lib/isc/netmgr/netmgr.c:2921: REQUIRE(((uvreq) != ((void *)0) && ((const isc__magic_t *)(uvreq))->magic == ((('N') << 24 | ('M') << 16 | ('U') << 8 | ('R'))))) failed, back trace
```
### What is the expected *correct* behavior?
named lives forever (or at least until an operator decides to stop and restart it)
### Relevant configuration files
see internal only note
### Relevant logs and/or screenshots
This is from the backtrace from the core dump that was produced:
Stack trace of thread 35031:
#0 0x00007f40cc2afe5c __pthread_kill_implementation (libc.so.6 + 0x8ce5c)
#1 0x00007f40cc25fa76 raise (libc.so.6 + 0x3ca76)
#2 0x00007f40cc2497fc abort (libc.so.6 + 0x267fc)
#3 0x0000561a0d6fb575 assertion_failed.cold (named + 0x1c575)
#4 0x00007f40cce39a50 isc_assertion_failed (libisc-9.18.16.so + 0x39a50)
#5 0x00007f40cce296d9 isc___nm_uvreq_put (libisc-9.18.16.so + 0x296d9)
#6 0x00007f40cce29d23 isc__nm_async_sendcb (libisc-9.18.16.so + 0x29d23)
#7 0x00007f40cce2da04 process_netievent (libisc-9.18.16.so + 0x2da04)
#8 0x00007f40cce2e057 process_queue (libisc-9.18.16.so + 0x2e057)
#9 0x00007f40cce2e277 async_cb (libisc-9.18.16.so + 0x2e277)
#10 0x00007f40ccd08e23 uv__async_io.part.0 (libuv.so.1 + 0xae23)
#11 0x00007f40ccd25dcb uv__io_poll (libuv.so.1 + 0x27dcb)
#12 0x00007f40ccd0e62f uv_run (libuv.so.1 + 0x1062f)
#13 0x00007f40cce2e704 nm_thread (libisc-9.18.16.so + 0x2e704)
#14 0x00007f40cce646e9 isc__trampoline_run (libisc-9.18.16.so + 0x646e9)
#15 0x00007f40cc2ae12d start_thread (libc.so.6 + 0x8b12d)
#16 0x00007f40cc32fbc0 __clone3 (libc.so.6 + 0x10cbc0)
### Possible fixesSeptember 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)Artem BoldarievArtem Boldarievhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4240dnstap system test fails on dnstap output file2024-03-28T10:20:18ZTom Krizekdnstap system test fails on dnstap output fileSystem test `dnstap` [started](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3552551) [to](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3553512) [fail](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3554027) [very](https://gitla...System test `dnstap` [started](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3552551) [to](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3553512) [fail](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3554027) [very](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3554808) [frequently](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3557127) on ~"v9.16" in the [system:gcc:oraclelinux7:amd64](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3560021) job (specifically on `oraclelinux7`):
```
I:dnstap:checking unix socket message counts
I:dnstap:dnstap output file smaller than expected
I:dnstap:failed
I:dnstap:checking UDP message counts
I:dnstap:ns4 0 expected 4
I:dnstap:failed
I:dnstap:checking TCP message counts
I:dnstap:checking AUTH_QUERY message counts
I:dnstap:checking AUTH_RESPONSE message counts
I:dnstap:checking CLIENT_QUERY message counts
I:dnstap:ns4 0 expected 1
I:dnstap:failed
I:dnstap:checking CLIENT_RESPONSE message counts
I:dnstap:ns4 0 expected 1
I:dnstap:failed
I:dnstap:checking RESOLVER_QUERY message counts
I:dnstap:checking RESOLVER_RESPONSE message counts
I:dnstap:checking UPDATE_QUERY message counts
I:dnstap:ns4 0 expected 1
I:dnstap:failed
I:dnstap:checking UPDATE_RESPONSE message counts
I:dnstap:ns4 0 expected 1
I:dnstap:failed
I:dnstap:checking reopened unix socket message counts
I:dnstap:failed
I:dnstap:checking UDP message counts
I:dnstap:checking TCP message counts
I:dnstap:checking AUTH_QUERY message counts
I:dnstap:checking AUTH_RESPONSE message counts
I:dnstap:checking CLIENT_QUERY message counts
I:dnstap:checking CLIENT_RESPONSE message counts
I:dnstap:checking RESOLVER_QUERY message counts
I:dnstap:checking RESOLVER_RESPONSE message counts
I:dnstap:checking UPDATE_QUERY message counts
I:dnstap:checking UPDATE_RESPONSE message counts
I:dnstap:checking large packet printing
I:dnstap:checking 'rndc -roll <value>' (no versions)
I:dnstap:checking 'rndc -roll <value>' (versions)
I:dnstap:exit status: 7
```
The good news is that this isn't cause by any code change, as the test used to [pass before](https://gitlab.isc.org/isc-projects/bind9/-/pipelines/143730) on the very same commit, and then [started to fail](https://gitlab.isc.org/isc-projects/bind9/-/pipelines/143854) afterwards since 2023-07-28.
However, it is quite puzzling why the test started to fail so often - the underlying `oraclelinux-7-amd64` image was last built on 2023-07-20, and the failure wasn't occurring at first, so it can't be caused by an image rebuild / dependencies either.
I wasn't able to reproduce the issue locally on ~"v9.16" using our `oraclelinux-7-amd` container image in over a hundred runs. I also didn't manage to reproduce it in my local environment on ~"v9.19" in over a hundred runs either.
Unfortunately, [increasing the timeout](https://gitlab.isc.org/isc-projects/bind9/-/commit/af9f675bcfd758efa86571c7528692679ab5fca9) for the dnstap file to be ready to 10s up from 5s [doesn't seem to help](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3561965).September 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)https://gitlab.isc.org/isc-projects/bind9/-/issues/4238The mkeys system test can update the root zone too fast2023-08-31T15:09:17ZMark AndrewsThe mkeys system test can update the root zone too fastThe keys system test can update the root zone too fast
causing reloads to fail as the modification time of root.db
has not changed. Add sleep 1 before signing.The keys system test can update the root zone too fast
causing reloads to fail as the modification time of root.db
has not changed. Add sleep 1 before signing.September 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)https://gitlab.isc.org/isc-projects/bind9/-/issues/4234[CVE-2023-4408] Parsing large DNS messages may cause excessive CPU load2024-03-28T12:11:21ZShoham Danino[CVE-2023-4408] Parsing large DNS messages may cause excessive CPU load| Quick Links | :link: |
| ------------------------ | ------------------------------------------------------------------------------ |
| Incident Manage...| Quick Links | :link: |
| ------------------------ | ------------------------------------------------------------------------------ |
| Incident Manager: | @matthijs |
| Deputy Incident Manager: | @michal |
| Public Disclosure Date: | 2024-02-13 |
| CVSS Score: | [7.5][cvss_score] |
| Security Advisory: | isc-private/printing-press!76 |
| Mattermost Channel: | [CVE-2023-4408: O(n²) complexity in DNS message parsing logic][mattermost_url] |
| Support Ticket: | N/A |
| Release Checklist: | #4515 & #4555 |
[cvss_score]: https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H&version=3.1
[mattermost_url]: https://mattermost.isc.org/isc/channels/cve-2023-4408
:bulb: **Click [here][checklist_explanations] (internal resource) for general information about the security incident handling process.**
[checklist_explanations]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations
### Earlier Than T-5
- [x] [:link:][step_deputy] **(IM)** Pick a Deputy Incident Manager
- [x] [:link:][step_respond] **(IM)** [Respond to the bug reporter](#note_394703)
- [x] [:link:][step_public_mrs] **(SwEng)** Ensure there are no public merge requests which inadvertently disclose the issue
- [x] [:link:][step_assign_cve_id] **(IM)** [Assign a CVE identifier](#note_396170)
- [x] [:link:][step_note_cve_info] **(SwEng)** Update this issue with the assigned CVE identifier and the CVSS score
- [x] [:link:][step_versions_affected] **(SwEng)** [Determine the range of product versions affected (including the Subscription Edition)](#note_396176)
- [x] [:link:][step_workarounds] **(SwEng)** [Determine whether workarounds for the problem exist](#note_396179)
- [X] [:link:][step_coordinate] **(SwEng)** [If necessary, coordinate with other parties](#note_397738)
- [x] [:link:][step_earliest_prepare] **(Support)** Prepare "earliest" notification text and hand it off to Marketing
- [x] [:link:][step_earliest_send] **(Marketing)** Update "earliest" notification document in SF portal and send bulk email to earliest customers
- [x] [:link:][step_advisory_mr] **(Support)** [Create a merge request for the Security Advisory and include all readily available information in it](isc-private/printing-press!76)
- [x] [:link:][step_reproducer_mr] ~~**(SwEng)** [Prepare a private merge request containing a system test reproducing the problem](#note_416143)~~
- [x] [:link:][step_notify_support] **(SwEng)** [Notify Support when a reproducer is ready](https://mattermost.isc.org/isc/pl/i5jke6nbhffizdduy4opewc56w)
- [x] [:link:][step_code_analysis] **(SwEng)** [Prepare a detailed explanation of the code flow triggering the problem](#note_416180)
- [x] [:link:][step_fix_mr] **(SwEng)** [Prepare a private merge request with the fix](isc-private/bind9!560)
- [x] [:link:][step_review_fix] **(SwEng)** Ensure the merge request with the fix is reviewed and has no outstanding discussions
- [x] [:link:][step_review_docs] **(Support)** [Review the documentation changes introduced by the merge request with the fix](#note_417455)
- [x] [:link:][step_backports] **(SwEng)** Prepare backports of the merge request addressing the problem [for](isc-private/bind9!585) [all](isc-private/bind9!586) [affected](isc-private/bind9!587) (and still maintained) branches of a given product
- [x] [:link:][step_finish_advisory] **(Support)** [Finish preparing the Security Advisory](https://mattermost.isc.org/isc/pl/76qusomtej88tjkqagiu1hfroy)
- [x] [:link:][step_meta_issue] **(QA)** [Create (or update) the private issue containing links to fixes & reproducers for all CVEs fixed in a given release cycle](#4486)
- [x] [:link:][step_changes] **(QA)** (BIND 9 only) [Reserve a block of `CHANGES` placeholders once the complete set of vulnerabilities fixed in a given release cycle is determined](!8625)
- [x] [:link:][step_merge_fixes] **(QA)** Merge the CVE fixes in CVE identifier order
- [x] [:link:][step_patches] **(QA)** Prepare a standalone patch for the last stable release of each affected (and still maintained) product branch
- [x] [:link:][step_asn_releases] **(QA)** Prepare ASN releases (as outlined in the Release Checklist)
### At T-5
- [x] [:link:][step_asn_documents] **(Marketing)** Update the text on the T-5 (from the Printing Press project) and "earliest" ASN documents in the SF portal
- [x] [:link:][step_asn_links] **(Marketing)** (BIND 9 only) Update the BIND -S information document in SF with download links to the new versions
- [x] [:link:][step_asn_send] **(Marketing)** Bulk email eligible customers to check the SF portal
- [x] [:link:][step_preannouncement] **(Marketing)** (BIND 9 only) Send a pre-announcement email to the *bind-announce* mailing list to alert users that the upcoming release will include security fixes
### At T-1
- [x] [:link:][step_packager_emails] **(First IM)** Send notifications to OS packagers
### On the Day of Public Disclosure
- [x] [:link:][step_clearance] **(IM)** [Grant QA & Marketing clearance to proceed with public release](https://mattermost.isc.org/isc/pl/hg9aypmxnjnjzc7ktwfo6xcpky)
- [x] [:link:][step_publish] **(QA/Marketing)** Publish the releases (as outlined in the release checklist)
- [x] [:link:][step_matrix] **(Support)** (BIND 9 only) Add the new CVEs to the vulnerability matrix in the Knowledge Base
- [x] [:link:][step_publish_advisory] **(Support)** Bump Document Version for the Security Advisory and publish it in the Knowledge Base
- [x] [:link:][step_notifications] **(First IM)** Send notification emails to third parties
- [x] [:link:][step_mitre] **(First IM)** Advise MITRE about the disclosed CVEs
- [x] [:link:][step_merge_advisory] **(First IM)** Merge the Security Advisory merge request
- [x] [:link:][step_embargo_end] **(IM)** [Inform original reporter (if external) that the security disclosure process is complete](#note_436560)
- [x] [:link:][step_asn_clear] **(Marketing)** Update the SF portal to clear the ASN
- [x] [:link:][step_customers] **(Marketing)** Email ASN recipients that the embargo is lifted
### After Public Disclosure
- [x] [:link:][step_regression] **(QA)** ~~[Merge a regression test reproducing the bug into all affected (and still maintained) branches](#note_416143)~~
[step_deputy]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#pick-a-deputy-incident-manager
[step_respond]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#respond-to-the-bug-reporter
[step_public_mrs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-there-are-no-public-merge-requests-which-inadvertently-disclose-the-issue
[step_assign_cve_id]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#assign-a-cve-identifier
[step_note_cve_info]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-this-issue-with-the-assigned-cve-identifier-and-the-cvss-score
[step_versions_affected]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-the-range-of-product-versions-affected-including-the-subscription-edition
[step_workarounds]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-whether-workarounds-for-the-problem-exist
[step_coordinate]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#if-necessary-coordinate-with-other-parties
[step_earliest_prepare]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-earliest-notification-text-and-hand-it-off-to-marketing
[step_earliest_send]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-earliest-notification-document-in-sf-portal-and-send-bulk-email-to-earliest-customers
[step_advisory_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-a-merge-request-for-the-security-advisory-and-include-all-readily-available-information-in-it
[step_reproducer_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-containing-a-system-test-reproducing-the-problem
[step_notify_support]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#notify-support-when-a-reproducer-is-ready
[step_code_analysis]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-detailed-explanation-of-the-code-flow-triggering-the-problem
[step_fix_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-with-the-fix
[step_review_fix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-the-merge-request-with-the-fix-is-reviewed-and-has-no-outstanding-discussions
[step_review_docs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#review-the-documentation-changes-introduced-by-the-merge-request-with-the-fix
[step_backports]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-backports-of-the-merge-request-addressing-the-problem-for-all-affected-and-still-maintained-branches-of-a-given-product
[step_finish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#finish-preparing-the-security-advisory
[step_meta_issue]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-or-update-the-private-issue-containing-links-to-fixes-reproducers-for-all-cves-fixed-in-a-given-release-cycle
[step_changes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-reserve-a-block-of-changes-placeholders-once-the-complete-set-of-vulnerabilities-fixed-in-a-given-release-cycle-is-determined
[step_merge_fixes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-cve-fixes-in-cve-identifier-order
[step_patches]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-standalone-patch-for-the-last-stable-release-of-each-affected-and-still-maintained-product-branch
[step_asn_releases]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-asn-releases-as-outlined-in-the-release-checklist
[step_asn_documents]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-the-text-on-the-t-5-from-the-printing-press-project-and-earliest-asn-documents-in-the-sf-portal
[step_asn_links]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-update-the-bind-s-information-document-in-sf-with-download-links-to-the-new-versions
[step_asn_send]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bulk-email-eligible-customers-to-check-the-sf-portal
[step_preannouncement]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-send-a-pre-announcement-email-to-the-bind-announce-mailing-list-to-alert-users-that-the-upcoming-release-will-include-security-fixes
[step_packager_emails]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notifications-to-os-packagers
[step_clearance]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#grant-qa-marketing-clearance-to-proceed-with-public-release
[step_publish]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#publish-the-releases-as-outlined-in-the-release-checklist
[step_matrix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-add-the-new-cves-to-the-vulnerability-matrix-in-the-knowledge-base
[step_publish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bump-document-version-for-the-security-advisory-and-publish-it-in-the-knowledge-base
[step_notifications]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notification-emails-to-third-parties
[step_mitre]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#advise-mitre-about-the-disclosed-cves
[step_merge_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-security-advisory-merge-request
[step_embargo_end]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#inform-original-reporter-if-external-that-the-security-disclosure-process-is-complete
[step_asn_clear]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-the-sf-portal-to-clear-the-asn
[step_customers]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#email-asn-recipients-that-the-embargo-is-lifted
[step_regression]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-a-regression-test-reproducing-the-bug-into-all-affected-and-still-maintained-branches
---
Hi,
Continuing our research, regarding CVE 2023-2828 ([issue/4055](https://gitlab.isc.org/isc-projects/bind9/-/issues/4055)) we discovered yet another possible vulnerability in BIND9 resolvers where one malicious CNAME query can cause over 27,000,000 calls to the dns_name_equal function and over 1,000,000,000 CPU instructions.
In the getsection function in message.c file there is a check if a name is already present in the section before appending the new name to the section.
```
...
if (!dns_name_equal(dns_rootname, name) ||
sectionid != DNS_SECTION_ADDITIONAL ||
msg->opt != NULL)
{
DO_ERROR(DNS_R_FORMERR);
}
...
for (count = 0; count < msg->counts[sectionid]; count++) {
...
```
The function checks whether each name is already present in the section by sending all the previous names with the new name to _dns_name_equal_ function - which causes a quadratic function call.
For CNAME queries, the BIND resolver has a resolution limit of 17, so for each malicious CNAME query the resolver executes 17 queries to the authoritative name server.
I tested an answer with 1800 RRsets long CNAME chain, and as a result, the dns_name_equal function is called i-1 times for RRset i for i = n to n-17.
In the experiment, I observed 27,304,801 calls for _dns_name_equal_ function, and my calculation indeed predicts:
`∑ n^2 / 2`. n from 1800 to 1784 = 27,295,948
I used Valgrind and Kcachegrind for checking that. Here is the valgind file: [callgrind.out.cname_shoham1.shoham.fun](/uploads/09aef4edae0384e93d1a891781dcfc20/callgrind.out.cname_shoham1.shoham.fun)
You can see here the number of function calls and the CPU instructions:
![fig_callgrind.out.cname_shoham1.shoham.fun](/uploads/e878d48174b9a8ebf0a8549eb8391e04/fig_callgrind.out.cname_shoham1.shoham.fun.png)
And here is an example of the answer section I respond:
![fig_shoham1.shoham.fun_pcap](/uploads/f13a330dac95e9e63483635c97bf075d/fig_shoham1.shoham.fun_pcap.png)
How to reproduce the vulnerability and additional technical information:
For this experiment, I used my Azure environment:
I used 3 machines: a client, resolver and authoritative name server
All my machines are Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz x64 with 2 vCPUs 8 GiB RAM and Linux (Ubuntu 20.04) OS.
I used BIND 9.16.42 version with no configuration changes. Here is the config.log file: [config.log](/uploads/3e4d410a1d2ea1c4185b0faaf492f229/config.log)
and the conf.named.option file: [named.conf.options](/uploads/12e35dc45738b7b2a29b16c2092902db/named.conf.options)
In the authoritative, I have a CNAME chain with 1800 RRsets (from shoham1.shoham.fun to shoham1800.shoham.fun). Attaching the zonfile here: [shoham.fun.forward](/uploads/c4279fc7a7997435c0fedf8268195edc/shoham.fun.forward)
The client issued a single query using this command: dig shoham1.shoham.fun. @<my_resolver_ip>
To reproduce the attack:
Turn on the resolver with the Valgrind tool using the following command: valgrind --tool=callgrind named -g -c /etc/named.conf.
From the client, query shoham1.shoham.fun using this command: dig shoham1.shoham.fun. @<your_resolver_ip>
Close the Valgrind resolve (Ctrl c will work)
Open the Callgrind file using Qcachegrind: qcachegrind ./callgrind.out.<the resolver PID>
You can also create your authoritative with the zonefile attached or with any long CNAME chain, here is a script for that:
```
with open('zonfile.txt', 'w') as f:
for i in range(1, 1801):
if i % 1800 != 0:
print(f'shoham{i} 86400 IN CNAME shoham{i + 1}',file=f)
else:
print(f'shoham{i} 86400 IN A 1.1.1.1',file=f)
```
Please don't hesitate to ask any question or any additional information.
Thanks,
Shoham Danino, Anat Bremler-Barr, Yehuda Afek and Yuval ShavittJanuary 2024 (9.16.46, 9.16.46-S1, 9.18.22, 9.18.22-S1, 9.19.20) (❗RECALLED❗)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.org