BIND issueshttps://gitlab.isc.org/isc-projects/bind9/-/issues2022-12-06T10:48:19Zhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3651update danger to check for Approved2022-12-06T10:48:19ZEvan Huntupdate danger to check for ApprovedWe've started using Approve rather than LGTM to mark merge requests as ready to merge, but danger will still report:
```
elif "LGTM (Merge OK)" not in mr_labels:
warn(
"This merge request is currently in review. "
"I...We've started using Approve rather than LGTM to mark merge requests as ready to merge, but danger will still report:
```
elif "LGTM (Merge OK)" not in mr_labels:
warn(
"This merge request is currently in review. "
"It should not be merged until it is marked with the *LGTM* label."
)
```
I'd fix it myself, but I'm so unfamiliar with the gitlab API, I don't even know how Mr. Labels got set in the first place.December 2022 (9.16.36, 9.16.36-S1, 9.18.10, 9.19.8)Tom KrizekTom Krizekhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3643Shutdown crash INSIST(__v > 0 && __v < (4294967295U)) in views system test2022-11-07T10:06:21ZMichal NowakShutdown crash INSIST(__v > 0 && __v < (4294967295U)) in views system testJob [#2874705](https://gitlab.isc.org/isc-projects/bind9/-/jobs/2874705) failed for fd77c37820af3a6251ef46e1d13b8e01f0f31630 on shutdown with:
```
02-Nov-2022 07:19:46.747 zone_shutdown: zone example038.com/IN (unsigned): shutting down
0...Job [#2874705](https://gitlab.isc.org/isc-projects/bind9/-/jobs/2874705) failed for fd77c37820af3a6251ef46e1d13b8e01f0f31630 on shutdown with:
```
02-Nov-2022 07:19:46.747 zone_shutdown: zone example038.com/IN (unsigned): shutting down
02-Nov-2022 07:19:46.747 calling free_rbtdb(example038.com)
02-Nov-2022 07:19:46.747 calling free_rbtdb(example017.com)
02-Nov-2022 07:19:46.747 zone.c:5783: INSIST(__v > 0 && __v < (4294967295U)) failed, back trace
```
```
D:views:Core was generated by `/builds/isc-projects/bind9/bin/named/.libs/named -D views-ns2 -X named.lock -m'.
D:views:Program terminated with signal SIGABRT, Aborted.
D:views:#0 0x00007efe79bfdc4c in __pthread_kill_implementation () from /lib64/libc.so.6
D:views:[Current thread is 1 (Thread 0x7efe767bd640 (LWP 6646))]
D:views:#0 0x00007efe79bfdc4c in __pthread_kill_implementation () from /lib64/libc.so.6
D:views:#1 0x00007efe79bad9c6 in raise () from /lib64/libc.so.6
D:views:#2 0x00007efe79b977f4 in abort () from /lib64/libc.so.6
D:views:#3 0x00007efe7ae5fdf0 in abort () from /lib64/libtsan.so.2
D:views:#4 0x0000000000425700 in assertion_failed (file=<optimized out>, line=<optimized out>, type=<optimized out>, cond=<optimized out>) at main.c:237
D:views:#5 0x00007efe7a888345 in isc_assertion_failed (file=file@entry=0x7efe7a80a95b "zone.c", line=line@entry=5783, type=type@entry=isc_assertiontype_insist, cond=cond@entry=0x7efe7a7cf108 "__v > 0 && __v < (4294967295U)") at assertions.c:48
D:views:#6 0x00007efe7a76dd90 in dns_zone_attach (source=source@entry=0x7b7c00058600, target=target@entry=0x7b5c00054a60) at zone.c:5783
D:views:#7 0x00007efe7a77afe6 in zone_refreshkeys (zone=zone@entry=0x7b7c00058600) at zone.c:11359
D:views:#8 0x00007efe7a7a9ea0 in zone_maintenance (zone=0x7b7c00058600) at zone.c:11607
D:views:#9 zone_timer (task=<optimized out>, event=<optimized out>) at zone.c:15312
D:views:#10 0x00007efe7a8b7ad0 in task_run (task=0x7b3000000b40) at task.c:821
D:views:#11 isc_task_run (task=0x7b3000000b40) at task.c:901
D:views:#12 0x00007efe7a862e0c in isc__nm_async_task (worker=worker@entry=0x7ba400000000, ev0=ev0@entry=0x7b4800117300) at netmgr/netmgr.c:846
D:views:#13 0x00007efe7a86e11e in process_netievent (worker=worker@entry=0x7ba400000000, ievent=ievent@entry=0x7b4800117300) at netmgr/netmgr.c:918
D:views:#14 0x00007efe7a86edef in process_queue (worker=worker@entry=0x7ba400000000, type=type@entry=NETIEVENT_TASK) at netmgr/netmgr.c:1011
D:views:#15 0x00007efe7a86fb65 in process_all_queues (worker=0x7ba400000000) at netmgr/netmgr.c:765
D:views:#16 async_cb (handle=0x7ba400000360) at netmgr/netmgr.c:794
D:views:#17 0x00007efe79f6318f in uv__async_io (loop=0x7ba400000010, w=0x7ba4000001d8, events=1) at /usr/src/libuv-v1.44.1/src/unix/async.c:163
D:views:#18 0x00007efe79f7f5d7 in uv__io_poll (loop=0x7ba400000010, timeout=-1) at /usr/src/libuv-v1.44.1/src/unix/epoll.c:374
D:views:#19 0x00007efe79f63c2d in uv_run (loop=0x7ba400000010, mode=UV_RUN_DEFAULT) at /usr/src/libuv-v1.44.1/src/unix/core.c:391
D:views:#20 0x00007efe7a86f202 in nm_thread (worker0=0x7ba400000000) at netmgr/netmgr.c:696
D:views:#21 0x00007efe7a8c2fd0 in isc__trampoline_run (arg=0x7b0c00002160) at trampoline.c:189
D:views:#22 0x00007efe7ae383f0 in __tsan_thread_start_func () from /lib64/libtsan.so.2
D:views:#23 0x00007efe79bfbe2d in start_thread () from /lib64/libc.so.6
D:views:#24 0x00007efe79c80364 in clone () from /lib64/libc.so.6
```
Full back trace and `views/ns2/named.run` are in the saved job.
isc-projects/bind9#3080 is a similar but closed-fixed issue in `rndc`.November 2022 (9.16.35, 9.16.35-S1, 9.18.9, 9.19.7)Evan HuntEvan Hunthttps://gitlab.isc.org/isc-projects/bind9/-/issues/3638TLS settings for primaries not saved for catalog zone entries.2022-12-09T14:55:08ZMark AndrewsTLS settings for primaries not saved for catalog zone entries.The tls setting in the following configuration was being ignored.
```
primary default-primary port 853 {
67.195.97.218 key "example.key" tls server.example;
};
catalog-zones {
zone "example.catalog" default-masters port 853 {
...The tls setting in the following configuration was being ignored.
```
primary default-primary port 853 {
67.195.97.218 key "example.key" tls server.example;
};
catalog-zones {
zone "example.catalog" default-masters port 853 {
"default-primary";
} zone-directory "zones";};
};
```December 2022 (9.16.36, 9.16.36-S1, 9.18.10, 9.19.8)https://gitlab.isc.org/isc-projects/bind9/-/issues/3637BIND 9.18 retries same authoritative nameserver before trying another one2023-11-22T09:02:06ZDavid ZychBIND 9.18 retries same authoritative nameserver before trying another one<!--
If the bug you are reporting is potentially security-related - for example,
if it involves an assertion failure or other crash in `named` that can be
triggered repeatedly - then please do *NOT* report it here, but send an
email to [...<!--
If the bug you are reporting is potentially security-related - for example,
if it involves an assertion failure or other crash in `named` that can be
triggered repeatedly - then please do *NOT* report it here, but send an
email to [security-officer@isc.org](security-officer@isc.org).
-->
### Summary
Given an authoritative zone with multiple nameservers, some of which may currently be unavailable, how should a BIND recursive resolver react when its first query attempt times out?
Empirically, BIND 9.16 will try a _different_ authoritative nameserver next, but BIND 9.18 will try the _same_ one 3 times in a row before moving on to a different one. The BIND 9.18 behavior is significantly less robust.
### BIND version used
```
[root@dmrz-test-rdns ~]# named -V
BIND 9.18.8 (Stable Release) <id:35f5d35>
running on Linux x86_64 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020
built by make with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/opt/isc/isc-bind/root/usr' '--exec-prefix=/opt/isc/isc-bind/root/usr' '--bindir=/opt/isc/isc-bind/root/usr/bin' '--sbindir=/opt/isc/isc-bind/root/usr/sbin' '--sysconfdir=/etc/opt/isc/isc-bind' '--datadir=/opt/isc/isc-bind/root/usr/share' '--includedir=/opt/isc/isc-bind/root/usr/include' '--libdir=/opt/isc/isc-bind/root/usr/lib64' '--libexecdir=/opt/isc/isc-bind/root/usr/libexec' '--localstatedir=/var/opt/isc/isc-bind' '--sharedstatedir=/var/opt/isc/isc-bind/lib' '--mandir=/opt/isc/isc-bind/root/usr/share/man' '--infodir=/opt/isc/isc-bind/root/usr/share/info' '--disable-static' '--enable-dnstap' '--with-pic' '--with-gssapi' '--with-json-c' '--with-libxml2' '--without-lmdb' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro -L/opt/isc/isc-bind/root/usr/lib64' 'CPPFLAGS= -I/opt/isc/isc-bind/root/usr/include' 'LT_SYS_LIBRARY_PATH=/usr/lib64' 'PKG_CONFIG_PATH=:/opt/isc/isc-bind/root/usr/lib64/pkgconfig:/opt/isc/isc-bind/root/usr/share/pkgconfig' 'SPHINX_BUILD=/builddir/build/BUILD/bind-9.18.8/sphinx/bin/sphinx-build'
compiled by GCC 4.8.5 20150623 (Red Hat 4.8.5-44)
compiled with OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017
linked to OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017
compiled with libuv version: 1.44.2
linked to libuv version: 1.44.2
compiled with libnghttp2 version: 1.33.0
linked to libnghttp2 version: 1.33.0
compiled with libxml2 version: 2.9.1
linked to libxml2 version: 20901
compiled with json-c version: 0.11
linked to json-c version: 0.11
compiled with zlib version: 1.2.7
linked to zlib version: 1.2.7
compiled with protobuf-c version: 1.4.1
linked to protobuf-c version: 1.4.1
threads support is enabled
DNSSEC algorithms: RSASHA1 NSEC3RSASHA1 RSASHA256 RSASHA512 ECDSAP256SHA256 ECDSAP384SHA384
DS algorithms: SHA-1 SHA-256 SHA-384
HMAC algorithms: HMAC-MD5 HMAC-SHA1 HMAC-SHA224 HMAC-SHA256 HMAC-SHA384 HMAC-SHA512
TKEY mode 2 support (Diffie-Hellman): yes
TKEY mode 3 support (GSS-API): yes
default paths:
named configuration: /etc/opt/isc/isc-bind/named.conf
rndc configuration: /etc/opt/isc/isc-bind/rndc.conf
DNSSEC root key: /etc/opt/isc/isc-bind/bind.keys
nsupdate session key: /var/opt/isc/isc-bind/run/named/session.key
named PID file: /var/opt/isc/isc-bind/run/named/named.pid
named lock file: /var/opt/isc/isc-bind/run/named/named.lock
```
### Steps to reproduce
1. Configure an authoritative server with the following authoritative zone, listing one working nameserver IP (itself) plus two others which are "down".
```
example.com. 30 IN SOA ns1.example.com. dns-admin.illinois.edu. 1 3600 900 1209600 30
example.com. 30 IN NS ns1.example.com.
example.com. 30 IN NS ns2.example.com.
example.com. 30 IN NS ns3.example.com.
foo.example.com. 30 IN TXT "test"
ns1.example.com. 30 IN A 18.191.96.154
ns2.example.com. 30 IN A 192.168.0.20
ns3.example.com. 30 IN A 192.168.0.30
```
2. Configure a CentOS 7 instance "dmrz-test-rdns" running isc-bind-bind 9.18.8-1.1.el7 from https://copr.fedorainfracloud.org/coprs/isc/bind/ with a stub zone for example.com:
```
options {
directory "/var/opt/isc/isc-bind/named/data";
allow-query { localhost; 10.0.0.0/8; };
recursion yes;
dnssec-validation no;
# enable querylogging at startup
querylog yes;
};
logging {
channel default_debug {
file "named.run";
print-time yes;
severity dynamic;
};
# local file for query logs
channel queries_file {
file "queries.log" versions 5 size 200m;
severity info;
print-time yes; print-category yes; print-severity yes;
};
category queries { queries_file; };
};
zone "example.com" IN {
file "example.com";
type stub;
masters { 18.191.96.154; };
};
```
3. As root on dmrz-test-rdns, start a packet capture:
```
[root@dmrz-test-rdns data]# tcpdump -i eth0 -Uw /var/tmp/${HOSTNAME}.pcap
```
4. From another workstation in the allow-query ACL, send a test query:
```
$ dig @10.225.64.30 foo.example.com txt
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.10 <<>> @10.225.64.30 foo.example.com txt
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33425
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;foo.example.com. IN TXT
;; ANSWER SECTION:
foo.example.com. 30 IN TXT "test"
;; Query time: 3784 msec
;; SERVER: 10.225.64.30#53(10.225.64.30)
;; WHEN: Tue Nov 01 20:08:08 UTC 2022
;; MSG SIZE rcvd: 61
```
Note the very long query time (over 3 seconds). If you don't see a long query time at first, wait at least 30s (for cache to expire) and try again.
5. Observe named complaining about a timeout:
```
[root@dmrz-test-rdns data]# tail /var/opt/isc/isc-bind/named/data/named.run
01-Nov-2022 20:04:54.508 all zones loaded
01-Nov-2022 20:04:54.508 running
01-Nov-2022 20:08:08.556 timed out resolving 'foo.example.com/TXT/IN': 192.168.0.20#53
01-Nov-2022 20:08:51.441 timed out resolving 'foo.example.com/TXT/IN': 192.168.0.30#53
```
6. Download the packet capture file and examine it in Wireshark, filtering by `dns` protocol.
Observe that after receiving a query from client (10.224.255.16) at 20:08:04, we try 3 times in a row to query the unresponsive 192.168.0.20 before moving on to the working nameserver 18.191.96.154.
On another occasion, we try 192.168.0.30 3 times in a row.
At 20:09:32 we happen to try the working nameserver first and are able to respond to our client quickly.
```
No. Time Source Destination Protocol Length Info
3 2022-11-01 20:08:04.776005 10.224.255.16 10.225.64.30 DNS 86 Standard query 0x8291 TXT foo.example.com OPT
4 2022-11-01 20:08:04.776851 10.225.64.30 192.168.0.20 DNS 98 Standard query 0x90f6 TXT foo.example.com OPT
5 2022-11-01 20:08:05.576960 10.225.64.30 192.168.0.20 DNS 98 Standard query 0x7acc TXT foo.example.com OPT
7 2022-11-01 20:08:06.378009 10.225.64.30 192.168.0.20 DNS 98 Standard query 0x05fe TXT foo.example.com OPT
14 2022-11-01 20:08:08.557405 10.225.64.30 18.191.96.154 DNS 98 Standard query 0x7979 TXT foo.example.com OPT
15 2022-11-01 20:08:08.558188 18.191.96.154 10.225.64.30 DNS 233 Standard query response 0x7979 TXT foo.example.com TXT NS ns3.example.com NS ns2.example.com NS ns1.example.com A 18.191.96.154 A 192.168.0.20 A 192.168.0.30 OPT
16 2022-11-01 20:08:08.558432 10.225.64.30 10.224.255.16 DNS 103 Standard query response 0x8291 TXT foo.example.com TXT OPT
31 2022-11-01 20:08:47.747803 10.224.255.16 10.225.64.30 DNS 86 Standard query 0xbf02 TXT foo.example.com OPT
32 2022-11-01 20:08:47.748163 10.225.64.30 192.168.0.30 DNS 98 Standard query 0x474b TXT foo.example.com OPT
33 2022-11-01 20:08:48.549188 10.225.64.30 192.168.0.30 DNS 98 Standard query 0x2130 TXT foo.example.com OPT
34 2022-11-01 20:08:49.350213 10.225.64.30 192.168.0.30 DNS 98 Standard query 0x7d5f TXT foo.example.com OPT
37 2022-11-01 20:08:51.441890 10.225.64.30 18.191.96.154 DNS 114 Standard query 0x1ef3 TXT foo.example.com OPT
38 2022-11-01 20:08:51.442705 18.191.96.154 10.225.64.30 DNS 233 Standard query response 0x1ef3 TXT foo.example.com TXT NS ns2.example.com NS ns1.example.com NS ns3.example.com A 18.191.96.154 A 192.168.0.20 A 192.168.0.30 OPT
39 2022-11-01 20:08:51.442872 10.225.64.30 10.224.255.16 DNS 103 Standard query response 0xbf02 TXT foo.example.com TXT OPT
55 2022-11-01 20:09:32.834068 10.224.255.16 10.225.64.30 DNS 86 Standard query 0x0297 TXT foo.example.com OPT
56 2022-11-01 20:09:32.834680 10.225.64.30 18.191.96.154 DNS 114 Standard query 0x958a TXT foo.example.com OPT
57 2022-11-01 20:09:32.835082 18.191.96.154 10.225.64.30 DNS 233 Standard query response 0x958a TXT foo.example.com TXT NS ns2.example.com NS ns3.example.com NS ns1.example.com A 18.191.96.154 A 192.168.0.20 A 192.168.0.30 OPT
58 2022-11-01 20:09:32.835211 10.225.64.30 10.224.255.16 DNS 103 Standard query response 0x0297 TXT foo.example.com TXT OPT
```
7. Now retry the experiment with a different recursive resolver "dmrz-test-rdns-916" (10.225.64.119) running isc-bind-bind 9.16.34-1.1.el7 from https://copr.fedorainfracloud.org/coprs/isc/bind-esv/
BIND 9.16 handles the situation much more gracefully; at 20:42:51 we fail to get a response from 192.168.0.30, but instead of retrying the same server we try another one, which happens to be the working one, so we are able to respond successfully to the client in under 1 second.
```
No. Time Source Destination Protocol Length Info
12 2022-11-01 20:42:51.913893 10.224.255.16 10.225.64.119 DNS 86 Standard query 0x29ac TXT foo.example.com OPT
13 2022-11-01 20:42:51.915009 10.225.64.119 192.168.0.30 DNS 98 Standard query 0x19c1 TXT foo.example.com OPT
14 2022-11-01 20:42:52.713887 10.225.64.119 18.191.96.154 DNS 98 Standard query 0x2280 TXT foo.example.com OPT
15 2022-11-01 20:42:52.714303 18.191.96.154 10.225.64.119 DNS 131 Standard query response 0x2280 TXT foo.example.com TXT OPT
16 2022-11-01 20:42:52.714709 10.225.64.119 10.224.255.16 DNS 103 Standard query response 0x29ac TXT foo.example.com TXT OPT
53 2022-11-01 20:44:15.623212 10.224.255.16 10.225.64.119 DNS 86 Standard query 0xcf02 TXT foo.example.com OPT
54 2022-11-01 20:44:15.623584 10.225.64.119 18.191.96.154 DNS 114 Standard query 0xc333 TXT foo.example.com OPT
55 2022-11-01 20:44:15.624183 18.191.96.154 10.225.64.119 DNS 233 Standard query response 0xc333 TXT foo.example.com TXT NS ns1.example.com NS ns3.example.com NS ns2.example.com A 18.191.96.154 A 192.168.0.20 A 192.168.0.30 OPT
56 2022-11-01 20:44:15.624371 10.225.64.119 10.224.255.16 DNS 103 Standard query response 0xcf02 TXT foo.example.com TXT OPT
68 2022-11-01 20:44:54.321211 10.224.255.16 10.225.64.119 DNS 86 Standard query 0x6f5f TXT foo.example.com OPT
69 2022-11-01 20:44:54.321578 10.225.64.119 192.168.0.20 DNS 98 Standard query 0x1fa5 TXT foo.example.com OPT
70 2022-11-01 20:44:55.120996 10.225.64.119 18.191.96.154 DNS 114 Standard query 0xddae TXT foo.example.com OPT
71 2022-11-01 20:44:55.121400 18.191.96.154 10.225.64.119 DNS 233 Standard query response 0xddae TXT foo.example.com TXT NS ns3.example.com NS ns1.example.com NS ns2.example.com A 18.191.96.154 A 192.168.0.20 A 192.168.0.30 OPT
72 2022-11-01 20:44:55.121577 10.225.64.119 10.224.255.16 DNS 103 Standard query response 0x6f5f TXT foo.example.com TXT OPT
```
In this case, we also do not see any "timed out" log messages in named.run
### What is the current *bug* behavior?
As shown above, BIND 9.18 consistently tries multiple times in a row to reach the same unresponsive authoritative nameserver IP, and is therefore unable to answer the client in a timely fashion even though the zone has another authoritative nameserver which is working just fine.
### What is the expected *correct* behavior?
Be like BIND 9.16 and retry against a different authoritative nameserver IP when the first one doesn't respond.
aside: this may be related to #3629January 2023 (9.16.37, 9.16.37-S1, 9.18.11, 9.18.11-S1, 9.19.9)Ondřej SurýOndřej Surýhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3636UAF in TCP dispatch2023-01-03T09:47:48ZCathy AlmondUAF in TCP dispatchFrom [Support Ticket #21450](https://support.isc.org/Ticket/Display.html?id=21450)
```
This afternoon we saw an event where bind crashed due to assertion failure.
01-Nov-2022 07:01:30.212 general: critical: message.c:4692: REQUIRE(msg...From [Support Ticket #21450](https://support.isc.org/Ticket/Display.html?id=21450)
```
This afternoon we saw an event where bind crashed due to assertion failure.
01-Nov-2022 07:01:30.212 general: critical: message.c:4692: REQUIRE(msg->state == (-1)) failed, back trace
01-Nov-2022 07:01:30.212 general: critical: /usr/local/sbin/named() [0x42bd8f]
01-Nov-2022 07:01:30.213 general: critical: /usr/local/lib/libisc-9.18.7.so(isc_assertion_failed+0xa) [0x7efc8264d3ea]
01-Nov-2022 07:01:30.213 general: critical: /usr/local/lib/libdns-9.18.7.so(dns_message_setclass+0x93) [0x7efc822c1813]
01-Nov-2022 07:01:30.213 general: critical: /usr/local/lib/libdns-9.18.7.so(+0x11ea6b) [0x7efc8233ca6b]
01-Nov-2022 07:01:30.213 general: critical: /usr/local/lib/libdns-9.18.7.so(+0x5d713) [0x7efc8227b713]
01-Nov-2022 07:01:30.213 general: critical: /usr/local/lib/libisc-9.18.7.so(isc__nm_async_readcb+0x9c) [0x7efc8263b4fc]
01-Nov-2022 07:01:30.213 general: critical: /usr/local/lib/libisc-9.18.7.so(isc__nm_readcb+0xb5) [0x7efc8263b635]
01-Nov-2022 07:01:30.213 general: critical: /usr/local/lib/libisc-9.18.7.so(isc__nm_tcpdns_processbuffer+0x11c) [0x7efc826430ec]
01-Nov-2022 07:01:30.213 general: critical: /usr/local/lib/libisc-9.18.7.so(isc__nm_process_sock_buffer+0x88) [0x7efc82638438]
01-Nov-2022 07:01:30.213 general: critical: /usr/local/lib/libisc-9.18.7.so(isc__nm_tcpdns_read_cb+0xb8) [0x7efc82643288]
01-Nov-2022 07:01:30.213 general: critical: /lib64/libuv.so.1(+0x1a4a4) [0x7efc80e964a4]
01-Nov-2022 07:01:30.213 general: critical: /lib64/libuv.so.1(+0x1ae0c) [0x7efc80e96e0c]
01-Nov-2022 07:01:30.213 general: critical: /lib64/libuv.so.1(+0x22983) [0x7efc80e9e983]
01-Nov-2022 07:01:30.213 general: critical: /lib64/libuv.so.1(uv_run+0x128) [0x7efc80e8bc88]
01-Nov-2022 07:01:30.213 general: critical: /usr/local/lib/libisc-9.18.7.so(+0x24348) [0x7efc8263d348]
01-Nov-2022 07:01:30.213 general: critical: /usr/local/lib/libisc-9.18.7.so(isc__trampoline_run+0x15) [0x7efc82673855]
01-Nov-2022 07:01:30.213 general: critical: /lib64/libpthread.so.0(+0x7ea5) [0x7efc802d8ea5]
01-Nov-2022 07:01:30.213 general: critical: /lib64/libc.so.6(clone+0x6d) [0x7efc80001b0d]
01-Nov-2022 07:01:30.213 general: critical: exiting (due to assertion failure)
```
The `query` (which is `resp->arg`) has already been destroyed:
```
(gdb) print *(resquery_t *)resp->arg
$7 = {magic = 557825056, references = 139621354865616, fctx = 0x0, rmessage = 0x0, mctx = 0x2632be0, dispatchmgr = 0x7efc780008e0, dispatch = 0x0, addrinfo = 0x7efc217d78d0, start = {seconds = 1667286089, nanoseconds = 244483829}, id = 32757, dispentry = 0x0, link = {
prev = 0xffffffffffffffff, next = 0xffffffffffffffff}, buffer = {magic = 0, base = 0x0, length = 0, used = 0, current = 0, active = 0, link = {prev = 0x0, next = 0x0}, mctx = 0x0, autore = false}, tsig = 0x0, tsigkey = 0x0, dscp = -1, ednsversion = 0,
options = 35, attributes = 2, udpsize = 1232, data = "\177\365\000\020\000\001\000\000\000\000\000\001\002ns\ainetdns\002eu\000\000\001\000\001\000\000)\004\320\000\000\200", '\000' <repeats 472 times>}
```
So there seem to be a reference counting problem.January 2023 (9.16.37, 9.16.37-S1, 9.18.11, 9.18.11-S1, 9.19.9)https://gitlab.isc.org/isc-projects/bind9/-/issues/3634BIND 9.18 fails to configure on NetBSD2022-11-07T09:30:03ZHåvard EidnesBIND 9.18 fails to configure on NetBSD### Summary
BIND 9.18.8 fails to configure on NetBSD (-current or 9.3).
### BIND version used
Source code for 9.18.8.
### Steps to reproduce
`./configure --with-libxml2=yes --with-tuning=large --sysconfdir=/etc --localstatedir=/var`...### Summary
BIND 9.18.8 fails to configure on NetBSD (-current or 9.3).
### BIND version used
Source code for 9.18.8.
### Steps to reproduce
`./configure --with-libxml2=yes --with-tuning=large --sysconfdir=/etc --localstatedir=/var`
### What is the current *bug* behavior?
The above `configure` line fails with
```
checking for jemalloc/jemalloc.h... no
configure: WARNING: jemalloc not found; performance will be reduced
configure: error: You cannot compile without jemalloc; jemalloc is the system allocator on NetBSD
```
and config.log contains
```
configure:24692: result: no
configure:24692: checking jemalloc/jemalloc.h presence
configure:24692: gcc -E conftest.c
conftest.c:143:10: fatal error: jemalloc/jemalloc.h: No such file or directory
143 | #include <jemalloc/jemalloc.h>
| ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
configure:24692: $? = 1
```
and
```
configure:24692: result: no
configure:24692: checking for jemalloc/jemalloc.h
configure:24692: result: no
configure:24873: WARNING: jemalloc not found; performance will be reduced
configure:24888: error: You cannot compile without jemalloc; jemalloc is the system allocator on NetBSD
```
Trying the above `configure` line with `--with-jemalloc=yes` doesn't really change this, I still get
```
checking for jemalloc/jemalloc.h... no
configure: error: jemalloc not found
```
Trying the above `configure` line with `--with-jemalloc=no` also doesn't succeed, then I get:
```
configure: error: You cannot compile without jemalloc; jemalloc is the system allocator on NetBSD
```
### What is the expected *correct* behavior?
While some of the text above is correct, i.e. that the system malloc() implementation on NetBSD is based on `jemalloc`, NetBSD does not ship with a header accessible as `<jemalloc/jemalloc.h>`. I do see our man page for `jemalloc(3)` does (incorrectly) mention that include file location in its SYNOPSIS, that's a separate (NetBSD) documentation bug.
It does however seem that BIND's configure script still insists on the presence of this header.
Not sure what FreeBSD does, looking at `configure.ac` if it doesn't have `<jemalloc/jemalloc.h>` either, it will be similarly affected.
### Relevant configuration files
N/A
### Relevant logs and/or screenshots
See above.
### Possible fixes
So far this points towards `configure.ac`. Not sure if there is also actual code which insists on `<jemalloc/jemalloc.h>`.
One hurdle at a time...
...
It seems `lib/isc/mem.c` is the only other file which (conditionally) wants this header.November 2022 (9.16.35, 9.16.35-S1, 9.18.9, 9.19.7)https://gitlab.isc.org/isc-projects/bind9/-/issues/3631TSAN data race in zone.c2022-11-07T09:19:27ZEvan HuntTSAN data race in zone.cTSAN reports a race between `dns_zone_rekey()` and `zone_maintenance()`. Turns out `zone_maintenance()` accesses all the timer info unlocked, whoops.
```
WARNING: ThreadSanitizer: data race
Write of size 8 at 0x000000000001 by main ...TSAN reports a race between `dns_zone_rekey()` and `zone_maintenance()`. Turns out `zone_maintenance()` accesses all the timer info unlocked, whoops.
```
WARNING: ThreadSanitizer: data race
Write of size 8 at 0x000000000001 by main thread (mutexes: write M1):
#0 dns_zone_rekey lib/dns/zone.c:22097
#1 named_server_dnssec bin/named/server.c:15335
#2 named_control_docommand bin/named/control.c:206
#3 control_command bin/named/controlconf.c:390
#4 task_run lib/isc/task.c:470
#5 task__run lib/isc/task.c:287
#6 isc__job_cb lib/isc/job.c:75
#7 uv__run_idle /usr/src/libuv-v1.44.1/src/unix/loop-watcher.c:68
#8 isc_loopmgr_run lib/isc/loop.c:469
#9 main bin/named/main.c:1545
Previous read of size 4 at 0x000000000001 by thread T1:
#0 isc_time_isepoch lib/isc/time.c:81
#1 zone_maintenance lib/dns/zone.c:11559
#2 zone_timer lib/dns/zone.c:15214
#3 timer_cb lib/isc/timer.c:113
#4 uv__run_timers /usr/src/libuv-v1.44.1/src/timer.c:178
#5 isc__trampoline_run lib/isc/trampoline.c:198
Location is heap block of size 3216 at 0x000000000018 allocated by main thread:
#0 malloc <null>
#1 mallocx lib/isc/jemalloc_shim.h:57
#2 mem_get lib/isc/mem.c:343
#3 isc__mem_get lib/isc/mem.c:774
#4 dns_zone_create lib/dns/zone.c:1158
#5 dns_zonemgr_createzone lib/dns/zone.c:19035
#6 configure_zone bin/named/server.c:6791
#7 configure_view bin/named/server.c:4208
#8 load_configuration bin/named/server.c:9271
#9 run_server bin/named/server.c:10065
#10 task_run lib/isc/task.c:470
#11 task__run lib/isc/task.c:287
#12 isc__job_cb lib/isc/job.c:75
#13 uv__run_idle /usr/src/libuv-v1.44.1/src/unix/loop-watcher.c:68
#14 isc_loopmgr_run lib/isc/loop.c:469
#15 main bin/named/main.c:1545
Mutex M1 (0x000000000028) created at:
#0 pthread_mutex_init <null>
#1 dns_zone_create lib/dns/zone.c:1163
#2 dns_zonemgr_createzone lib/dns/zone.c:19035
#3 configure_zone bin/named/server.c:6791
#4 configure_view bin/named/server.c:4208
#5 load_configuration bin/named/server.c:9271
#6 run_server bin/named/server.c:10065
#7 task_run lib/isc/task.c:470
#8 task__run lib/isc/task.c:287
#9 isc__job_cb lib/isc/job.c:75
#10 uv__run_idle /usr/src/libuv-v1.44.1/src/unix/loop-watcher.c:68
#11 isc_loopmgr_run lib/isc/loop.c:469
#12 main bin/named/main.c:1545
Thread T1 'isc-loop-0001' (running) created by main thread at:
#0 pthread_create <null>
#1 isc_thread_create lib/isc/thread.c:70
#2 isc_loopmgr_run lib/isc/loop.c:463
#3 main bin/named/main.c:1545
SUMMARY: ThreadSanitizer: data race lib/dns/zone.c:22097 in dns_zone_rekey
```November 2022 (9.16.35, 9.16.35-S1, 9.18.9, 9.19.7)Evan HuntEvan Hunthttps://gitlab.isc.org/isc-projects/bind9/-/issues/3630'nextpart' is not compatible with 'set -x'2022-11-08T16:44:07ZMark Andrews'nextpart' is not compatible with 'set -x'When debugging system tests failures `nextpart` directs debugging output from `set -x` to the `.prev` file.When debugging system tests failures `nextpart` directs debugging output from `set -x` to the `.prev` file.December 2022 (9.16.36, 9.16.36-S1, 9.18.10, 9.19.8)https://gitlab.isc.org/isc-projects/bind9/-/issues/3627Inheritance bug when setting remote server port2022-11-07T09:10:57ZMatthijs Mekkingmatthijs@isc.orgInheritance bug when setting remote server portHave a secondary zone with a primary that fails the transfer over TLS but should work over plain DNS:
```
zone "dot-fallback" {
type secondary;
file "dot-fallback.db";
primaries {
10.53.0.1 tls ephemer...Have a secondary zone with a primary that fails the transfer over TLS but should work over plain DNS:
```
zone "dot-fallback" {
type secondary;
file "dot-fallback.db";
primaries {
10.53.0.1 tls ephemeral;
10.53.0.1;
};
```
This should work, but doesn't currently because the second primary (`10.53.0.1`) uses the TLS port instead of the default DNS port.November 2022 (9.16.35, 9.16.35-S1, 9.18.9, 9.19.7)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/3622[CVE-2022-3736] named configured to answer from stale cache may terminate une...2023-02-22T13:32:56ZBorja Marcos EA2EKH[CVE-2022-3736] named configured to answer from stale cache may terminate unexpectedly while processing RRSIG queries| Quick Links | :link: |
| ------------------------ | --------------------------------------------------------- |
| Incident Manager: | @michal ...| Quick Links | :link: |
| ------------------------ | --------------------------------------------------------- |
| Incident Manager: | @michal |
| Deputy Incident Manager: | @peterd |
| Public Disclosure Date: | 2023-01-25 |
| CVSS Score: | [7.5][cvss_score] |
| Security Advisory: | isc-private/printing-press!36 |
| Mattermost Channel: | [CVE-2022-3736 RRSIG query crash][mattermost_url] |
| Support Ticket: | N/A |
| Release Checklist: | #3755 |
| Post-mortem Etherpad: | [postmortem-2023-01][postmortem_url] |
[cvss_score]: https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H&version=3.1
[mattermost_url]: https://mattermost.isc.org/isc/channels/cve-2022-3736-rrsig-query-crash
[postmortem_url]: https://pad.isc.org/p/postmortem-2023-01
:bulb: **Click [here][checklist_explanations] (internal resource) for general information about the security incident handling process.**
[checklist_explanations]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations
### Earlier Than T-5
- [x] [:link:][step_deputy] **(IM)** Pick a Deputy Incident Manager
- [x] [:link:][step_respond] **(IM)** [Respond to the bug reporter](#note_331050)
- [x] [:link:][step_etherpad] **(IM)** [Create an Etherpad for post-mortem][postmortem_url]
- [x] [:link:][step_public_mrs] **(SwEng)** Ensure there are no public merge requests which inadvertently disclose the issue
- [x] [:link:][step_assign_cve_id] **(IM)** [Assign a CVE identifier](#note_325658)
- [x] [:link:][step_note_cve_info] **(SwEng)** Update this issue with the assigned CVE identifier and the CVSS score
- [x] [:link:][step_versions_affected] **(SwEng)** [Determine the range of product versions affected (including the Subscription Edition)](#note_331684)
- [x] [:link:][step_workarounds] **(SwEng)** [Determine whether workarounds for the problem exist](#note_331693)
- [x] [:link:][step_coordinate] **(SwEng)** If necessary, coordinate with other parties
- [x] [:link:][step_earliest] **(Support)** [Prepare and send out "earliest" notifications](https://mattermost.isc.org/isc/pl/8aoebzueyj8rjyyaw31omqc4ey)
- [x] [:link:][step_advisory_mr] **(Support)** [Create a merge request for the Security Advisory and include all readily available information in it](isc-private/printing-press!36)
- [x] [:link:][step_reproducer_mr] **(SwEng)** [Prepare a private merge request containing a system test reproducing the problem](isc-private/bind9!469)
- [x] [:link:][step_notify_support] **(SwEng)** [Notify Support when a reproducer is ready](https://mattermost.isc.org/isc/pl/3gtsq3trktnifd8a9xubgodo8h)
- [x] [:link:][step_code_analysis] **(SwEng)** [Prepare a detailed explanation of the code flow triggering the problem](#note_325042)
- [x] [:link:][step_fix_mr] **(SwEng)** [Prepare a private merge request with the fix](isc-private/bind9!470)
- [x] [:link:][step_review_fix] **(SwEng)** Ensure the merge request with the fix is reviewed and has no outstanding discussions
- [x] [:link:][step_review_docs] **(Support)** Review the documentation changes introduced by the merge request with the fix
- [x] [:link:][step_backports] **(SwEng)** Prepare backports of the merge request addressing the problem for all affected (and still maintained) branches of a given product
- [x] [:link:][step_finish_advisory] **(Support)** [Finish preparing the Security Advisory](https://mattermost.isc.org/isc/pl/rscwci4pp3rn5yktuoaquu14fh)
- [x] [:link:][step_meta_issue] **(QA)** [Create (or update) the private issue containing links to fixes & reproducers for all CVEs fixed in a given release cycle](isc-private/bind9#64)
- [x] [:link:][step_changes] **(QA)** [(BIND 9 only) Reserve a block of `CHANGES` placeholders once the complete set of vulnerabilities fixed in a given release cycle is determined](!7356)
- [x] [:link:][step_merge_fixes] **(QA)** Merge the CVE fixes in CVE identifier order
- [x] [:link:][step_patches] **(QA)** Prepare a standalone patch for the last stable release of each affected (and still maintained) product branch
- [x] [:link:][step_asn_releases] **(QA)** Prepare ASN releases (as outlined in the Release Checklist)
### At T-5
- [x] [:link:][step_send_asn] **(Support)** Send ASN to eligible customers
- [x] [:link:][step_preannouncement] **(Support)** (BIND 9 only) Send a pre-announcement email to the *bind-announce* mailing list to alert users that the upcoming release will include security fixes
### At T-4
- [x] [:link:][step_verify_asn] **(Support)** [Verify that all ASN-eligible customers have received the notification email](https://mattermost.isc.org/isc/pl/q1zu9qimbj8kfj1xxqn913hg6e)
### At T-1
- [x] [:link:][step_check_customers] **(Support)** [Verify that any new or reinstated customers have received the notification email](https://mattermost.isc.org/isc/pl/p6qo7p4cyi8udgj4ye9fyiqu8w)
- [x] [:link:][step_packager_emails] **(First IM)** [Send notifications to OS packagers](isc-private/printing-press!44)
### On the Day of Public Disclosure
- [x] [:link:][step_clearance] **(IM)** Grant Support clearance to proceed with public release
- [x] [:link:][step_publish] **(Support)** Publish the releases (as outlined in the release checklist)
- [x] [:link:][step_matrix] **(Support)** (BIND 9 only) Update vulnerability matrix in the Knowledge Base
- [x] [:link:][step_publish_advisory] **(Support)** [Bump Document Version for the Security Advisory and publish it in the Knowledge Base](https://kb.isc.org/docs/cve-2022-3736)
- [x] [:link:][step_notifications] **(First IM)** [Send notification emails to third parties](isc-private/printing-press!46)
- [x] [:link:][step_mitre] **(First IM)** [Advise MITRE about the disclosed CVEs](https://gitlab.isc.org/isc-private/printing-press/-/merge_requests/36#note_346749)
- [x] [:link:][step_merge_advisory] **(First IM)** Merge the Security Advisory merge request
- [x] [:link:][step_embargo_end] **(IM)** Inform original reporter (if external) that the security disclosure process is complete
- [x] [:link:][step_customers] **(Support)** Inform customers a fix has been released
### After Public Disclosure
- [x] [:link:][step_postmortem] ~~**(First IM)** [Organize post-mortem meeting and make sure it happens](https://mattermost.isc.org/isc/pl/rsebfwxjytd1ib63zjq4aunjec)~~
- [x] [:link:][step_tickets] **(Support)** Close support tickets
- [x] [:link:][step_regression] **(QA)** Merge a regression test reproducing the bug into all affected (and still maintained) branches
[step_deputy]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#pick-a-deputy-incident-manager
[step_respond]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#respond-to-the-bug-reporter
[step_etherpad]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-an-etherpad-for-post-mortem
[step_public_mrs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-there-are-no-public-merge-requests-which-inadvertently-disclose-the-issue
[step_assign_cve_id]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#assign-a-cve-identifier
[step_note_cve_info]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-this-issue-with-the-assigned-cve-identifier-and-the-cvss-score
[step_versions_affected]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-the-range-of-product-versions-affected-including-the-subscription-edition
[step_workarounds]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-whether-workarounds-for-the-problem-exist
[step_coordinate]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#if-necessary-coordinate-with-other-parties
[step_earliest]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-and-send-out-earliest-notifications
[step_advisory_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-a-merge-request-for-the-security-advisory-and-include-all-readily-available-information-in-it
[step_reproducer_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-containing-a-system-test-reproducing-the-problem
[step_notify_support]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#notify-support-when-a-reproducer-is-ready
[step_code_analysis]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-detailed-explanation-of-the-code-flow-triggering-the-problem
[step_fix_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-with-the-fix
[step_review_fix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-the-merge-request-with-the-fix-is-reviewed-and-has-no-outstanding-discussions
[step_review_docs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#review-the-documentation-changes-introduced-by-the-merge-request-with-the-fix
[step_backports]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-backports-of-the-merge-request-addressing-the-problem-for-all-affected-and-still-maintained-branches-of-a-given-product
[step_finish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#finish-preparing-the-security-advisory
[step_meta_issue]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-or-update-the-private-issue-containing-links-to-fixes-reproducers-for-all-cves-fixed-in-a-given-release-cycle
[step_changes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-reserve-a-block-of-changes-placeholders-once-the-complete-set-of-vulnerabilities-fixed-in-a-given-release-cycle-is-determined
[step_merge_fixes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-cve-fixes-in-cve-identifier-order
[step_patches]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-standalone-patch-for-the-last-stable-release-of-each-affected-and-still-maintained-product-branch
[step_asn_releases]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-asn-releases-as-outlined-in-the-release-checklist
[step_send_asn]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-asn-to-eligible-customers
[step_preannouncement]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-send-a-pre-announcement-email-to-the-bind-announce-mailing-list-to-alert-users-that-the-upcoming-release-will-include-security-fixes
[step_verify_asn]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#verify-that-all-asn-eligible-customers-have-received-the-notification-email
[step_check_customers]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#verify-that-any-new-or-reinstated-customers-have-received-the-notification-email
[step_packager_emails]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notifications-to-os-packagers
[step_clearance]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#grant-support-clearance-to-proceed-with-public-release
[step_publish]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#publish-the-releases-as-outlined-in-the-release-checklist
[step_matrix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-update-vulnerability-matrix-in-the-knowledge-base
[step_publish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bump-document-version-for-the-security-advisory-and-publish-it-in-the-knowledge-base
[step_notifications]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notification-emails-to-third-parties
[step_mitre]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#advise-mitre-about-the-disclosed-cves
[step_merge_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-security-advisory-merge-request
[step_embargo_end]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#inform-original-reporter-if-external-that-the-security-disclosure-process-is-complete
[step_customers]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#inform-customers-a-fix-has-been-released
[step_postmortem]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#organize-post-mortem-meeting-and-make-sure-it-happens
[step_tickets]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#close-support-tickets
[step_regression]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-a-regression-test-reproducing-the-bug-into-all-affected-and-still-maintained-branches
---
### Summary
BIND 9.16.33 crashes with `type != ((dns_rdatatype_t)dns_rdatatype_rrsig)`
### BIND version used
```
BIND 9.16.33 (Extended Support Version) <id:35e9c6e>
running on FreeBSD amd64 13.1-RELEASE FreeBSD 13.1-RELEASE fc952ac22 DNS13
built by make with '--disable-linux-caps' '--localstatedir=/var' '--sysconfdir=/usr/local/etc/namedb' '--with-dlopen=yes' '--without-python' '--with-libxml2' '--with-openssl=/usr' '--with-readline=-L/usr/local/lib -ledit' '--with-dlz-filesystem=yes' '--enable-dnstap' '--disable-fixed-rrset' '--disable-geoip' '--without-maxminddb' '--without-gssapi' '--with-libidn2=/usr/local' '--without-json-c' '--disable-largefile' '--with-lmdb=/usr/local' '--disable-native-pkcs11' '--disable-querytrace' '--enable-tcp-fastopen' '--enable-developer' '--enable-symtable' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/share/info/' '--build=amd64-portbld-freebsd13.1' 'build_alias=amd64-portbld-freebsd13.1' 'CC=cc' 'CFLAGS=-pipe -DLIBICONV_PLUG -g -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing ' 'LDFLAGS= -fstack-protector-strong ' 'LIBS=-L/usr/local/lib' 'CPPFLAGS=-DLIBICONV_PLUG -isystem /usr/local/include' 'CPP=cpp' 'PKG_CONFIG=pkgconf' 'PKG_CONFIG_LIBDIR=/usr/ports/dns/bind916/work/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig'
compiled by CLANG FreeBSD Clang 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
compiled with OpenSSL version: OpenSSL 1.1.1o-freebsd 3 May 2022
linked to OpenSSL version: OpenSSL 1.1.1o-freebsd 3 May 2022
compiled with libuv version: 1.44.2
linked to libuv version: 1.44.2
compiled with libxml2 version: 2.9.13
linked to libxml2 version: 20913
compiled with zlib version: 1.2.12
linked to zlib version: 1.2.12
compiled with protobuf-c version: 1.4.1
linked to protobuf-c version: 1.4.1
threads support is enabled
default paths:
named configuration: /usr/local/etc/namedb/named.conf
rndc configuration: /usr/local/etc/namedb/rndc.conf
DNSSEC root key: /usr/local/etc/namedb/bind.keys
nsupdate session key: /var/run/named/session.key
named PID file: /var/run/named/pid
named lock file: /var/run/named/named.lock
```
### Steps to reproduce
Can't reproduce but seems to be related to the stale answer cache.
The first time it happened it affected two recursive servers using the stale answer cache feature. In order to check whether it happened again I disabled stale answer cache on one of them and, surprise, it has happened again only on the server that had stale answer cache still enabled.
### What is the current *bug* behavior?
A crash due to an assert.
### What is the expected *correct* behavior?
(What you should see instead.)
### Relevant configuration files
(Paste any relevant configuration files - please use code blocks (```)
to format console output. If submitting the contents of your
configuration file in a non-confidential Issue, it is advisable to
obscure key secrets: this can be done automatically by using
`named-checkconf -px`.)
### Relevant logs and/or screenshots
```
#0 thr_kill () at thr_kill.S:4
#1 0x0000000800dd6c74 in __raise (s=s@entry=6)
at /usr/src/lib/libc/gen/raise.c:52
#2 0x0000000800e88109 in abort () at /usr/src/lib/libc/stdlib/abort.c:67
#3 0x000000000030c3d0 in assertion_failed (file=0x2a9b3e "db.c", line=524,
type=isc_assertiontype_require,
cond=0x2857d1 "type != ((dns_rdatatype_t)dns_rdatatype_rrsig)")
at ./main.c:271
#4 0x0000000000675c5d in isc_assertion_failed (file=0x2a9b3e "db.c",
line=524, type=isc_assertiontype_require,
cond=0x2857d1 "type != ((dns_rdatatype_t)dns_rdatatype_rrsig)")
at assertions.c:48
#5 0x00000000003e2b02 in dns_db_findext (db=0x8024c2400, name=0x81e6c0900,
version=0x0, type=46, options=6152, now=1666651564, nodep=0x7fffdffed120,
foundname=0x84d1f4c00, methods=0x7fffdffec770, clientinfo=0x7fffdffecbc0,
rdataset=0x84ce32f80, sigrdataset=0x0) at db.c:524
#6 0x000000000037e51d in query_lookup (qctx=0x7fffdffecc90) at query.c:5793
#7 0x000000000039c66d in query_lookup_stale (client=0x84d1f8d58)
at query.c:6034
#8 0x000000000037f61b in fetch_callback (task=0x809fcf480, event=0x82b875400)
at query.c:6076
#9 0x00000000006c6d08 in task_run (task=0x809fcf480) at task.c:851
#10 0x00000000006c68e5 in isc_task_run (task=0x809fcf480) at task.c:944
--Type <RET> for more, q to quit, c to continue without paging--
#11 0x00000000006a2e95 in isc__nm_async_task (worker=0x8016f0000,
ev0=0x83a2aba00) at netmgr.c:861
#12 0x000000000069b00b in process_netievent (worker=0x8016f0000,
ievent=0x83a2aba00) at netmgr.c:933
#13 0x00000000006a2c3a in process_queue (worker=0x8016f0000,
type=NETIEVENT_TASK) at netmgr.c:999
#14 0x00000000006a27ff in process_all_queues (worker=0x8016f0000)
at netmgr.c:780
#15 0x0000000000696f74 in async_cb (handle=0x8016f02d8) at netmgr.c:809
#16 0x0000000800cd16bf in uv__async_io (loop=0x8016f0010, w=0x8016f0188,
events=1) at src/unix/async.c:163
#17 0x0000000800cee489 in uv__io_poll (loop=0x8016f0010, timeout=-1)
at src/unix/kqueue.c:390
#18 0x0000000800cd1d5b in uv_run (loop=0x8016f0010, mode=UV_RUN_DEFAULT)
at src/unix/core.c:406
#19 0x0000000000696fdd in nm_thread (worker0=0x8016f0000) at netmgr.c:711
#20 0x00000000006ca51f in isc__trampoline_run (arg=0x80161b690)
at trampoline.c:213
#21 0x0000000800d0983a in thread_start (curthread=0x801612e00)
at /usr/src/lib/libthr/thread/thr_create.c:292
#22 0x0000000000000000 in ?? ()
```
The coredump and binaries are located on the bikeshed: /home/support/named-crash-borjam-20221025.tar.gzJanuary 2023 (9.16.37, 9.16.37-S1, 9.18.11, 9.18.11-S1, 9.19.9)Michał KępieńMichał Kępieńhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3619[CVE-2022-3924] named configured to answer from stale cache may terminate une...2023-06-01T14:25:19ZEverett Fulton[CVE-2022-3924] named configured to answer from stale cache may terminate unexpectedly at recursive-clients soft quota| Quick Links | :link: |
| ------------------------ | ---------------------------------------------------------------------------------- |
| Inciden...| Quick Links | :link: |
| ------------------------ | ---------------------------------------------------------------------------------- |
| Incident Manager: | @matthijs |
| Deputy Incident Manager: | @peterd |
| Public Disclosure Date: | 2023-01-25 |
| CVSS Score: | [7.5][cvss_score] |
| Security Advisory: | isc-private/printing-press!37 |
| Mattermost Channel: | [CVE-2022-3924: Serve-stale crash on recursive-clients soft quota][mattermost_url] |
| Support Ticket: | https://support.isc.org/Ticket/Display.html?id=21397 |
| Release Checklist: | #3755 |
| Post-mortem Etherpad: | [postmortem-2023-01][postmortem_url] |
[cvss_score]: https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H&version=3.1
[mattermost_url]: https://mattermost.isc.org/isc/channels/cve-2022-3924-serve-stale-crash-on-recursive-clients-soft-quota
[postmortem_url]: https://pad.isc.org/p/postmortem-2023-01
:bulb: **Click [here][checklist_explanations] (internal resource) for general information about the security incident handling process.**
[checklist_explanations]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations
### Earlier Than T-5
- [x] [:link:][step_deputy] **(IM)** Pick a Deputy Incident Manager
- [x] [:link:][step_respond] **(IM)** [Respond to the bug reporter](https://support.isc.org/Ticket/Display.html?id=21397#txn-826061)
- [x] [:link:][step_etherpad] **(IM)** [Create an Etherpad for post-mortem][postmortem_url]
- [x] [:link:][step_public_mrs] **(SwEng)** Ensure there are no public merge requests which inadvertently disclose the issue
- [x] [:link:][step_assign_cve_id] **(IM)** [Assign a CVE identifier](#note_328068)
- [x] [:link:][step_note_cve_info] **(SwEng)** Update this issue with the assigned CVE identifier and the CVSS score
- [x] [:link:][step_versions_affected] **(SwEng)** [Determine the range of product versions affected (including the Subscription Edition)](https://mattermost.isc.org/isc/pl/n5d9mpca4tbd8q7uje6dg7o8ga)
- [x] [:link:][step_workarounds] **(SwEng)** [Determine whether workarounds for the problem exist](#note_326320)
- [x] [:link:][step_coordinate] **(SwEng)** If necessary, coordinate with other parties
- [x] [:link:][step_earliest] **(Support)** [Prepare and send out "earliest" notifications](https://mattermost.isc.org/isc/pl/8trnd73pfjn43bnjy3fjh5m8ke)
- [x] [:link:][step_advisory_mr] **(Support)** [Create a merge request for the Security Advisory and include all readily available information in it](isc-private/printing-press!37)
- [x] [:link:][step_reproducer_mr] **(SwEng)** [Prepare a private merge request containing a system test reproducing the problem](isc-private/bind9!474)
- [x] [:link:][step_notify_support] **(SwEng)** [Notify Support when a reproducer is ready](https://mattermost.isc.org/isc/pl/ds8xu3rmhbgpxr6mjfhe5mufhh)
- [x] [:link:][step_code_analysis] **(SwEng)** [Prepare a detailed explanation of the code flow triggering the problem](#note_326320)
- [x] [:link:][step_fix_mr] **(SwEng)** [Prepare a private merge request with the fix](isc-private/bind9!475)
- [x] [:link:][step_review_fix] **(SwEng)** Ensure the merge request with the fix is reviewed and has no outstanding discussions
- [x] [:link:][step_review_docs] **(Support)** Review the documentation changes introduced by the merge request with the fix
- [x] [:link:][step_backports] **(SwEng)** Prepare backports of the merge request addressing the problem for all affected (and still maintained) branches of a given product
- [x] [:link:][step_finish_advisory] **(Support)** [Finish preparing the Security Advisory](https://mattermost.isc.org/isc/pl/be9euchbcfdk9nj488qmw5unsa)
- [x] [:link:][step_meta_issue] **(QA)** [Create (or update) the private issue containing links to fixes & reproducers for all CVEs fixed in a given release cycle](isc-private/bind9#64)
- [x] [:link:][step_changes] **(QA)** [(BIND 9 only) Reserve a block of `CHANGES` placeholders once the complete set of vulnerabilities fixed in a given release cycle is determined](!7356)
- [x] [:link:][step_merge_fixes] **(QA)** Merge the CVE fixes in CVE identifier order
- [x] [:link:][step_patches] **(QA)** Prepare a standalone patch for the last stable release of each affected (and still maintained) product branch
- [x] [:link:][step_asn_releases] **(QA)** Prepare ASN releases (as outlined in the Release Checklist)
### At T-5
- [x] [:link:][step_send_asn] **(Support)** Send ASN to eligible customers
- [x] [:link:][step_preannouncement] **(Support)** (BIND 9 only) Send a pre-announcement email to the *bind-announce* mailing list to alert users that the upcoming release will include security fixes
### At T-4
- [x] [:link:][step_verify_asn] **(Support)** [Verify that all ASN-eligible customers have received the notification email](https://mattermost.isc.org/isc/pl/q1zu9qimbj8kfj1xxqn913hg6e)
### At T-1
- [x] [:link:][step_check_customers] **(Support)** [Verify that any new or reinstated customers have received the notification email](https://mattermost.isc.org/isc/pl/p6qo7p4cyi8udgj4ye9fyiqu8w)
- [x] [:link:][step_packager_emails] **(First IM)** [Send notifications to OS packagers](isc-private/printing-press!44)
### On the Day of Public Disclosure
- [x] [:link:][step_clearance] **(IM)** Grant Support clearance to proceed with public release
- [x] [:link:][step_publish] **(Support)** Publish the releases (as outlined in the release checklist)
- [x] [:link:][step_matrix] **(Support)** (BIND 9 only) Update vulnerability matrix in the Knowledge Base
- [x] [:link:][step_publish_advisory] **(Support)** [Bump Document Version for the Security Advisory and publish it in the Knowledge Base](https://kb.isc.org/docs/cve-2022-3924)
- [x] [:link:][step_notifications] **(First IM)** [Send notification emails to third parties](isc-private/printing-press!46)
- [x] [:link:][step_mitre] **(First IM)** [Advise MITRE about the disclosed CVEs](https://gitlab.isc.org/isc-private/printing-press/-/merge_requests/37#note_346753)
- [x] [:link:][step_merge_advisory] **(First IM)** Merge the Security Advisory merge request
- [x] [:link:][step_embargo_end] **(IM)** Inform original reporter (if external) that the security disclosure process is complete
- [x] [:link:][step_customers] **(Support)** Inform customers a fix has been released
### After Public Disclosure
- [x] [:link:][step_postmortem] ~~**(First IM)** [Organize post-mortem meeting and make sure it happens](https://mattermost.isc.org/isc/pl/rsebfwxjytd1ib63zjq4aunjec)~~
- [x] [:link:][step_tickets] **(Support)** Close support tickets
- [x] [:link:][step_regression] **(QA)** Merge a regression test reproducing the bug into all affected (and still maintained) branches
[step_deputy]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#pick-a-deputy-incident-manager
[step_respond]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#respond-to-the-bug-reporter
[step_etherpad]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-an-etherpad-for-post-mortem
[step_public_mrs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-there-are-no-public-merge-requests-which-inadvertently-disclose-the-issue
[step_assign_cve_id]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#assign-a-cve-identifier
[step_note_cve_info]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-this-issue-with-the-assigned-cve-identifier-and-the-cvss-score
[step_versions_affected]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-the-range-of-product-versions-affected-including-the-subscription-edition
[step_workarounds]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-whether-workarounds-for-the-problem-exist
[step_coordinate]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#if-necessary-coordinate-with-other-parties
[step_earliest]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-and-send-out-earliest-notifications
[step_advisory_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-a-merge-request-for-the-security-advisory-and-include-all-readily-available-information-in-it
[step_reproducer_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-containing-a-system-test-reproducing-the-problem
[step_notify_support]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#notify-support-when-a-reproducer-is-ready
[step_code_analysis]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-detailed-explanation-of-the-code-flow-triggering-the-problem
[step_fix_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-with-the-fix
[step_review_fix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-the-merge-request-with-the-fix-is-reviewed-and-has-no-outstanding-discussions
[step_review_docs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#review-the-documentation-changes-introduced-by-the-merge-request-with-the-fix
[step_backports]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-backports-of-the-merge-request-addressing-the-problem-for-all-affected-and-still-maintained-branches-of-a-given-product
[step_finish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#finish-preparing-the-security-advisory
[step_meta_issue]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-or-update-the-private-issue-containing-links-to-fixes-reproducers-for-all-cves-fixed-in-a-given-release-cycle
[step_changes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-reserve-a-block-of-changes-placeholders-once-the-complete-set-of-vulnerabilities-fixed-in-a-given-release-cycle-is-determined
[step_merge_fixes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-cve-fixes-in-cve-identifier-order
[step_patches]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-standalone-patch-for-the-last-stable-release-of-each-affected-and-still-maintained-product-branch
[step_asn_releases]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-asn-releases-as-outlined-in-the-release-checklist
[step_send_asn]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-asn-to-eligible-customers
[step_preannouncement]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-send-a-pre-announcement-email-to-the-bind-announce-mailing-list-to-alert-users-that-the-upcoming-release-will-include-security-fixes
[step_verify_asn]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#verify-that-all-asn-eligible-customers-have-received-the-notification-email
[step_check_customers]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#verify-that-any-new-or-reinstated-customers-have-received-the-notification-email
[step_packager_emails]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notifications-to-os-packagers
[step_clearance]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#grant-support-clearance-to-proceed-with-public-release
[step_publish]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#publish-the-releases-as-outlined-in-the-release-checklist
[step_matrix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-update-vulnerability-matrix-in-the-knowledge-base
[step_publish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bump-document-version-for-the-security-advisory-and-publish-it-in-the-knowledge-base
[step_notifications]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notification-emails-to-third-parties
[step_mitre]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#advise-mitre-about-the-disclosed-cves
[step_merge_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-security-advisory-merge-request
[step_embargo_end]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#inform-original-reporter-if-external-that-the-security-disclosure-process-is-complete
[step_customers]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#inform-customers-a-fix-has-been-released
[step_postmortem]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#organize-post-mortem-meeting-and-make-sure-it-happens
[step_tickets]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#close-support-tickets
[step_regression]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-a-regression-test-reproducing-the-bug-into-all-affected-and-still-maintained-branches
---
# Report: Resolver crash when stale cache entry matches RPZ entry
**Note: The report mentions RPZ, but it is actually a crash caused by the `recursive-clients` soft quota being hit. The crash only happens when `stale-answer-client-timeout` is enabled and has a non-zero value.**
A customer has reported a repeatable crash on 9.16.34 (on Amazon Linux 2012) and 9.18.8 (on Ubuntu 22.04.1 LTS), when 'stale-answer-enable yes;' is configured, and a query matches both a stale cache entry (auth nameservers blocked) and an RPZ entry.
## Reproducer:
```
# named -V
BIND 9.18.1-1ubuntu1.2-Ubuntu (Stable Release) <id:>
running on Linux x86_64 5.15.0-1021-aws #25-Ubuntu SMP Fri Sep 23 12:20:42 UTC 2022
built by make with '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--runstatedir=/run' '--disable-maintainer-mode' '--disable-dependency-tracking' '--libdir=/usr/lib/x86_64-linux-gnu' '--sysconfdir=/etc/bind' '--with-python=python3' '--localstatedir=/' '--enable-threads' '--enable-largefile' '--with-libtool' '--enable-shared' '--disable-static' '--with-gost=no' '--with-openssl=/usr' '--with-gssapi=yes' '--with-libidn2' '--with-json-c' '--with-lmdb=/usr' '--with-gnu-ld' '--with-maxminddb' '--with-atf=no' '--enable-ipv6' '--enable-rrl' '--enable-filter-aaaa' '--disable-native-pkcs11' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -ffile-prefix-map=/build/bind9-2lYtkE/bind9-9.18.1=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -fno-strict-aliasing -fno-delete-null-pointer-checks -DNO_VERSION_DATE -DDIG_SIGCHASE' 'LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'
compiled by GCC 11.2.0
compiled with OpenSSL version: OpenSSL 3.0.2 15 Mar 2022
linked to OpenSSL version: OpenSSL 3.0.2 15 Mar 2022
compiled with libuv version: 1.43.0
linked to libuv version: 1.43.0
compiled with libnghttp2 version: 1.43.0
linked to libnghttp2 version: 1.43.0
compiled with libxml2 version: 2.9.13
linked to libxml2 version: 20913
compiled with json-c version: 0.15
linked to json-c version: 0.15
compiled with zlib version: 1.2.11
linked to zlib version: 1.2.11
linked to maxminddb version: 1.5.2
threads support is enabled
default paths:
named configuration: /etc/bind/named.conf
rndc configuration: /etc/bind/rndc.conf
DNSSEC root key: /etc/bind/bind.keys
nsupdate session key: //run/named/session.key
named PID file: //run/named/named.pid
named lock file: //run/named/named.lock
geoip-directory: /usr/share/GeoIP
```
```
# named-checkconf -p
options {
directory "/var/cache/bind";
listen-on-v6 {
"any";
};
dnssec-validation no;
response-policy {
zone "test.rpz.local" max-policy-ttl 86400;
} break-dnssec yes qname-wait-recurse no;
stale-answer-enable yes;
stale-answer-client-timeout 1800;
stale-cache-enable yes;
};
zone "." {
type hint;
file "/usr/share/dns/root.hints";
};
zone "localhost" {
type master;
file "/etc/bind/db.local";
};
zone "127.in-addr.arpa" {
type master;
file "/etc/bind/db.127";
};
zone "0.in-addr.arpa" {
type master;
file "/etc/bind/db.0";
};
zone "255.in-addr.arpa" {
type master;
file "/etc/bind/db.255";
};
zone "test.rpz.local" in {
type master;
file "/etc/bind/db.rpz.local";
allow-query {
"localhost";
};
allow-transfer {
"localhost";
};
forwarders {
};
};
zone "myctl.com" in {
type master;
file "/etc/bind/myctl.com.local";
allow-query {
"localhost";
};
allow-transfer {
"localhost";
};
forwarders {
};
};
```
```
# cat db.rpz.local
$TTL 900
@ IN SOA localhost. root.localhost. (
1 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
86400 ) ; Negative Cache TTL
IN NS localhost.
$TTL 293
test.myctl.com CNAME test-cname-a.myctl.com.
```
```
# cat myctl.com.local
$ORIGIN .
$TTL 86400
myctl.com IN SOA localhost. root.localhost. (
1 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
86400 ) ; Negative Cache TTL
IN NS ns-canada.topdns.com.
IN NS ns-usa.topdns.com.
IN NS ns-uk.topdns.com.
$ORIGIN myctl.com
test-cname-a NS ns-canada.topdns.com.
NS ns-usa.topdns.com.
NS ns-uk.topdns.com.
test NS ns-canada.topdns.com.
NS ns-usa.topdns.com.
NS ns-uk.topdns.com.
$ORIGIN test.myctl.com
$TTL 300
* CNAME test.myctl.com.
```
---
Regular queries work fine, with the RPZ taking effect:
```
# dig @127.0.0.1 999.test.myctl.com
; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> @127.0.0.1 999.test.myctl.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50674
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 2
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 66f39e5345dbbe20010000006352b399a385cb42e1947df1 (good)
;; QUESTION SECTION:
;999.test.myctl.com. IN A
;; ANSWER SECTION:
999.test.myctl.com. 300 IN CNAME test.myctl.com.
test.myctl.com. 293 IN CNAME test-cname-a.myctl.com.
test-cname-a.myctl.com. 60 IN A 127.0.0.1
;; ADDITIONAL SECTION:
test.rpz.local. 1 IN SOA localhost. root.localhost. 1 604800 86400 2419200 86400
;; Query time: 143 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Fri Oct 21 14:58:33 UTC 2022
;; MSG SIZE rcvd: 205
```
Command to run test is
`queryperf -d ./test.pattern.shuffled -s 127.0.0.1 -l 900 -T 10000 -v`
The input file was created with:
```
for m in $(seq 1 200); do for i in $(seq 1 1000); do echo $i.test.myctl.com A >> test.pattern; done; done
shuf test.pattern > test.pattern.shuffled
```
After one minute, we block upstream NSs with iptables:
```
iptables -I INPUT -s 77.247.183.137/32,108.61.150.91/32,109.201.142.225/32,46.166.189.99/32,108.61.12.163/32 -j DROP
```
As soon as TTL for all CNAMEs are expired (checking with rndc dumpdb), we get this set of error messages in log file:
9.16:
```
21-Oct-2022 18:16:07.996 client @0x7f261d905230 127.0.0.1#47018 (61.test.myctl.com): rpz QNAME Local-Data rewrite test.myctl.com/A/IN via test.myctl.com.test.rpz.local
21-Oct-2022 18:16:07.996 test-cname-a.myctl.com query within stale refresh time, stale answer used
21-Oct-2022 18:16:07.996 client @0x7f261d783450 127.0.0.1#47018 (42.test.myctl.com): rpz QNAME Local-Data rewrite test.myctl.com/A/IN via test.myctl.com.test.rpz.local
21-Oct-2022 18:16:07.996 test-cname-a.myctl.com query within stale refresh time, stale answer used
21-Oct-2022 18:16:08.000 client @0x7f261d783450 127.0.0.1#47018 (14.test.myctl.com): recursive-clients soft limit exceeded (901/900/1000), aborting oldest query
21-Oct-2022 18:16:08.000 client @0x7f261d891dc0 127.0.0.1#47018 (429.test.myctl.com): rpz QNAME Local-Data rewrite test.myctl.com/A/IN via test.myctl.com.test.rpz.local
21-Oct-2022 18:16:08.000 client @0x7f261ce109b0 127.0.0.1#47018 (907.test.myctl.com): rpz QNAME Local-Data rewrite test.myctl.com/A/IN via test.myctl.com.test.rpz.local
21-Oct-2022 18:16:08.000 resolver.c:11282: fatal error:
21-Oct-2022 18:16:08.000 RUNTIME_CHECK(event->fetch != fetch) failed
21-Oct-2022 18:16:08.000 exiting (due to fatal error in library)
```
9.18:
```
Oct 21 18:25:03 ip-172-31-41-183 named[13856]: client @0x7fd9082306a8 127.0.0.1#55429 (875.test.myctl.com): rpz QNAME Local-Data rewrite test.myctl.com/A/IN via test.myctl.com.test.rpz.local
Oct 21 18:25:03 ip-172-31-41-183 named[13856]: client @0x7fd9082306a8 127.0.0.1#55429 (459.test.myctl.com): rpz QNAME Local-Data rewrite test.myctl.com/A/IN via test.myctl.com.test.rpz.local
Oct 21 18:25:03 ip-172-31-41-183 named[13856]: client @0x7fd9082306a8 127.0.0.1#55429 (82.test.myctl.com): rpz QNAME Local-Data rewrite test.myctl.com/A/IN via test.myctl.com.test.rpz.local
Oct 21 18:26:10 ip-172-31-41-183 named[13856]: client @0x7fd909aa06f8 127.0.0.1#55429 (256.test.myctl.com): recursive-clients soft limit exceeded (901/900/1000), aborting oldest query
Oct 21 18:26:10 ip-172-31-41-183 rsyslogd: imjournal: 4812 messages lost due to rate-limiting
Oct 21 18:26:11 ip-172-31-41-183 named[13856]: client @0x7fd908dcdf58 127.0.0.1#55429 (75.test.myctl.com): recursive-clients soft limit exceeded (901/900/1000), aborting oldest query
Oct 21 18:26:12 ip-172-31-41-183 named[13856]: client @0x7fd9088ef9e8 127.0.0.1#55429 (510.test.myctl.com): recursive-clients soft limit exceeded (901/900/1000), aborting oldest query
Oct 21 18:26:12 ip-172-31-41-183 named[13856]: resolver.c:11000: fatal error:
Oct 21 18:26:12 ip-172-31-41-183 named[13856]: RUNTIME_CHECK(event->fetch != fetch) failed
Oct 21 18:26:12 ip-172-31-41-183 named[13856]: exiting (due to fatal error in library)
```January 2023 (9.16.37, 9.16.37-S1, 9.18.11, 9.18.11-S1, 9.19.9)Arаm SаrgsyаnArаm Sаrgsyаnhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3617rbtdb.c:1368: REQUIRE(rbtdb->future_version == ((void *)0)) failed, back trace2022-11-07T10:23:55ZMichal Nowakrbtdb.c:1368: REQUIRE(rbtdb->future_version == ((void *)0)) failed, back trace`mkeys` system test failed locally on `mnowak/v9_16_sub-post-October` branch (`v9_16_sub` rebased on top of 9.16.34-S1):
```
rbtdb.c:1368: REQUIRE(rbtdb->future_version == ((void *)0)) failed, back trace
#0 0x43cffd in assertion_failed()...`mkeys` system test failed locally on `mnowak/v9_16_sub-post-October` branch (`v9_16_sub` rebased on top of 9.16.34-S1):
```
rbtdb.c:1368: REQUIRE(rbtdb->future_version == ((void *)0)) failed, back trace
#0 0x43cffd in assertion_failed()+0x4d
#1 0x6445aa in isc_assertion_failed()+0xa
#2 0x531aba in newversion()+0x27a
#3 0x5d80f8 in sync_keyzone()+0x168
#4 0x5d8791 in dns_zone_synckeyzone()+0x71
#5 0x460edf in named_server_mkeys()+0x4ef
#6 0x43800e in named_control_docommand()+0x8ee
#7 0x439fd1 in control_recvmessage()+0x6a1
#8 0x6734ac in isc_task_run()+0x12c
#9 0x65dabe in process_netievent()+0x4ce
#10 0x65dced in process_queue()+0xbd
#11 0x65e693 in async_cb()+0x23
#12 0x7f9a0b250c9d in _fini()+0x7f9a0abca6d1
#13 0x7f9a0b26c7dd in _fini()+0x7f9a0abe6211
#14 0x7f9a0b25672a in _fini()+0x7f9a0abd015e
#15 0x65dfb9 in nm_thread()+0x59
#16 0x675755 in isc__trampoline_run()+0x15
#17 0x7f9a0aa8ce2d in _fini()+0x7f9a0a406861
#18 0x7f9a0ab121b0 in _fini()+0x7f9a0a48bbe4
exiting (due to assertion failure)
```
[named.run](/uploads/b085ac8511ec9e6b75f00b461180d468/named.run)
[core.named.zst](/uploads/46fe2445f94444265ed191f9bdba386f/core.named.1000.b92b34b7d3184d539c1237b79885c7d0.1109907.1666349864000000.zst)
Similar previous issues:
- https://bugs.isc.org/Ticket/Display.html?id=46468
- https://gitlab.isc.org/isc-projects/bind9/-/issues/1205
Also see the MM discussion here: https://mattermost.isc.org/isc/pl/du8hawy8tinn7khhortqwo6wnc.November 2022 (9.16.35, 9.16.35-S1, 9.18.9, 9.19.7)https://gitlab.isc.org/isc-projects/bind9/-/issues/3613TTL issue with resolver's cached and expired results for qtype ANY queries2023-04-03T08:44:39ZArаm SаrgsyаnTTL issue with resolver's cached and expired results for qtype ANY queriesWhen the resolver retrieves cached `rdataset`s from a database node in `lib/ns/query.c:query_respond_any()` for a qtype ANY query, and some of them are expired, those expired RRs are included in the answer with big (UNIX timestamp big) T...When the resolver retrieves cached `rdataset`s from a database node in `lib/ns/query.c:query_respond_any()` for a qtype ANY query, and some of them are expired, those expired RRs are included in the answer with big (UNIX timestamp big) TTLs. Here's how to reproduce the issue.
Run a resolver locally with the following configuration:
```
options {
directory ".";
pid-file "named.pid";
listen-on-v6 { none; };
listen-on port 9053 { 127.0.0.1; };
dnssec-validation auto;
recursion yes;
allow-recursion { 127.0.0.1; };
};
```
Query for github.com ANY:
```
$ dig @127.0.0.1 -p9053 github.com ANY
; <<>> DiG 9.18.7 <<>> @127.0.0.1 -p9053 github.com ANY
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42497
;; flags: qr rd ra; QUERY: 1, ANSWER: 28, AUTHORITY: 0, ADDITIONAL: 2
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 3e7b2b8f73eb454a01000000634fe23fe9aa5ebb8af2d349 (good)
;; QUESTION SECTION:
;github.com. IN ANY
;; ANSWER SECTION:
github.com. 3600 IN CAA 0 issue "globalsign.com"
github.com. 3600 IN CAA 0 issuewild "digicert.com"
github.com. 3600 IN CAA 0 issue "digicert.com"
github.com. 3600 IN TXT "adobe-idp-site-verification=b92c9e999aef825edc36e0a3d847d2dbad5b2fc0e05c79ddd7a16139b48ecf4b"
github.com. 3600 IN TXT "google-site-verification=UTM-3akMgubp6tQtgEuAkYNYLyYAvpTnnSrDMWoDR3o"
github.com. 3600 IN TXT "MS=6BF03E6AF5CB689E315FB6199603BABF2C88D805"
github.com. 3600 IN TXT "apple-domain-verification=RyQhdzTl6Z6x8ZP4"
github.com. 3600 IN TXT "MS=ms58704441"
github.com. 3600 IN TXT "MS=ms44452932"
github.com. 3600 IN TXT "docusign=087098e3-3d46-47b7-9b4e-8a23028154cd"
github.com. 3600 IN TXT "atlassian-domain-verification=jjgw98AKv2aeoYFxiL/VFaoyPkn3undEssTRuMg6C/3Fp/iqhkV4HVV7WjYlVeF8"
github.com. 3600 IN TXT "v=spf1 ip4:192.30.252.0/22 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com include:spf.protection.outlook.com include:mail.zendesk.com include:_spf.salesforce.com include:servers.mcsv.net ip4:166.78.69.169 ip4:1" "66.78.69.170 ip4:166.78.71.131 ip4:167.89.101.2 ip4:167.89.101.192/28 ip4:192.254.112.60 ip4:192.254.112.98/31 ip4:192.254.113.10 ip4:192.254.113.101 ip4:192.254.114.176 ip4:62.253.227.114 ~all"
github.com. 3600 IN TXT "stripe-verification=f88ef17321660a01bab1660454192e014defa29ba7b8de9633c69d6b4912217f"
github.com. 3600 IN MX 10 alt4.aspmx.l.google.com.
github.com. 3600 IN MX 1 aspmx.l.google.com.
github.com. 3600 IN MX 5 alt1.aspmx.l.google.com.
github.com. 3600 IN MX 5 alt2.aspmx.l.google.com.
github.com. 3600 IN MX 10 alt3.aspmx.l.google.com.
github.com. 900 IN SOA ns-1707.awsdns-21.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
github.com. 60 IN A 140.82.114.3
github.com. 900 IN NS ns-520.awsdns-01.net.
github.com. 900 IN NS ns-421.awsdns-52.com.
github.com. 900 IN NS dns4.p08.nsone.net.
github.com. 900 IN NS ns-1707.awsdns-21.co.uk.
github.com. 900 IN NS dns1.p08.nsone.net.
github.com. 900 IN NS ns-1283.awsdns-32.org.
github.com. 900 IN NS dns3.p08.nsone.net.
github.com. 900 IN NS dns2.p08.nsone.net.
;; ADDITIONAL SECTION:
ns-421.awsdns-52.com. 172800 IN A 205.251.193.165
;; Query time: 63 msec
;; SERVER: 127.0.0.1#9053(127.0.0.1) (TCP)
;; WHEN: Wed Oct 19 11:40:47 UTC 2022
;; MSG SIZE rcvd: 1670
```
As you can see, the A record has a TTL value of 60. Wait 60 seconds for it to expire, then repeat the query:
```
$ dig @127.0.0.1 -p9053 github.com ANY
; <<>> DiG 9.18.7 <<>> @127.0.0.1 -p9053 github.com ANY
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27528
;; flags: qr rd ra; QUERY: 1, ANSWER: 28, AUTHORITY: 0, ADDITIONAL: 2
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: fdb7a33b7c5d8e0601000000634fe28a0d3f8db7e69dc8f7 (good)
;; QUESTION SECTION:
;github.com. IN ANY
;; ANSWER SECTION:
github.com. 3525 IN CAA 0 issue "digicert.com"
github.com. 3525 IN CAA 0 issuewild "digicert.com"
github.com. 3525 IN CAA 0 issue "globalsign.com"
github.com. 3525 IN TXT "adobe-idp-site-verification=b92c9e999aef825edc36e0a3d847d2dbad5b2fc0e05c79ddd7a16139b48ecf4b"
github.com. 3525 IN TXT "atlassian-domain-verification=jjgw98AKv2aeoYFxiL/VFaoyPkn3undEssTRuMg6C/3Fp/iqhkV4HVV7WjYlVeF8"
github.com. 3525 IN TXT "v=spf1 ip4:192.30.252.0/22 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com include:spf.protection.outlook.com include:mail.zendesk.com include:_spf.salesforce.com include:servers.mcsv.net ip4:166.78.69.169 ip4:1" "66.78.69.170 ip4:166.78.71.131 ip4:167.89.101.2 ip4:167.89.101.192/28 ip4:192.254.112.60 ip4:192.254.112.98/31 ip4:192.254.113.10 ip4:192.254.113.101 ip4:192.254.114.176 ip4:62.253.227.114 ~all"
github.com. 3525 IN TXT "MS=ms44452932"
github.com. 3525 IN TXT "google-site-verification=UTM-3akMgubp6tQtgEuAkYNYLyYAvpTnnSrDMWoDR3o"
github.com. 3525 IN TXT "MS=6BF03E6AF5CB689E315FB6199603BABF2C88D805"
github.com. 3525 IN TXT "apple-domain-verification=RyQhdzTl6Z6x8ZP4"
github.com. 3525 IN TXT "MS=ms58704441"
github.com. 3525 IN TXT "stripe-verification=f88ef17321660a01bab1660454192e014defa29ba7b8de9633c69d6b4912217f"
github.com. 3525 IN TXT "docusign=087098e3-3d46-47b7-9b4e-8a23028154cd"
github.com. 3525 IN MX 1 aspmx.l.google.com.
github.com. 3525 IN MX 10 alt4.aspmx.l.google.com.
github.com. 3525 IN MX 10 alt3.aspmx.l.google.com.
github.com. 3525 IN MX 5 alt1.aspmx.l.google.com.
github.com. 3525 IN MX 5 alt2.aspmx.l.google.com.
github.com. 825 IN SOA ns-1707.awsdns-21.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
github.com. 1666179707 IN A 140.82.114.3
github.com. 825 IN NS ns-1283.awsdns-32.org.
github.com. 825 IN NS dns1.p08.nsone.net.
github.com. 825 IN NS dns2.p08.nsone.net.
github.com. 825 IN NS dns3.p08.nsone.net.
github.com. 825 IN NS dns4.p08.nsone.net.
github.com. 825 IN NS ns-421.awsdns-52.com.
github.com. 825 IN NS ns-520.awsdns-01.net.
github.com. 825 IN NS ns-1707.awsdns-21.co.uk.
;; ADDITIONAL SECTION:
ns-421.awsdns-52.com. 172725 IN A 205.251.193.165
;; Query time: 0 msec
;; SERVER: 127.0.0.1#9053(127.0.0.1) (TCP)
;; WHEN: Wed Oct 19 11:42:02 UTC 2022
;; MSG SIZE rcvd: 1670
```
As you can see, now the A record has a TTL value of 1666179707.December 2022 (9.16.36, 9.16.36-S1, 9.18.10, 9.19.8)Mark AndrewsMark Andrewshttps://gitlab.isc.org/isc-projects/bind9/-/issues/3607apex in name_external may be invalid when using dual stack servers2022-11-17T11:45:22ZMark Andrewsapex in name_external may be invalid when using dual stack serversWhen using `dual-stack-servers` and `-4` or `-6` is in use, `apex` in resolver.c:name_external may be set to a `fctx->fwdname` instance which has not been populated leading to an INSIST failure in dns_name_fullcompare.
```
Thread 0 Cra...When using `dual-stack-servers` and `-4` or `-6` is in use, `apex` in resolver.c:name_external may be set to a `fctx->fwdname` instance which has not been populated leading to an INSIST failure in dns_name_fullcompare.
```
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x0000000192e00e28 __pthread_kill + 8
1 libsystem_pthread.dylib 0x0000000192e3343c pthread_kill + 292
2 libsystem_c.dylib 0x0000000192d7b454 abort + 124
3 named 0x0000000104b9a80c assertion_failed + 552 (main.c:238)
4 libisc-9.19.7-dev.dylib 0x00000001050af878 isc_assertion_failed + 56 (assertions.c:49)
5 libdns-9.19.7-dev.dylib 0x0000000104cc8128 dns_name_fullcompare + 316 (name.c:464)
6 libdns-9.19.7-dev.dylib 0x0000000104da1598 name_external + 160 (resolver.c:6824)
7 libdns-9.19.7-dev.dylib 0x0000000104da07b0 rctx_answer_scan + 332 (resolver.c:8641)
8 libdns-9.19.7-dev.dylib 0x0000000104da01e4 rctx_answer_positive + 132 (resolver.c:8514)
9 libdns-9.19.7-dev.dylib 0x0000000104d9d56c rctx_answer + 128 (resolver.c:8407)
10 libdns-9.19.7-dev.dylib 0x0000000104d99fc4 resquery_response + 3456 (resolver.c:7919)
11 libdns-9.19.7-dev.dylib 0x0000000104c6eb20 udp_recv + 1200 (dispatch.c:605)
12 libisc-9.19.7-dev.dylib 0x000000010509aed0 isc__nm_async_readcb + 408 (netmgr.c:2188)
13 libisc-9.19.7-dev.dylib 0x0000000105099070 isc__nm_readcb + 332 (netmgr.c:2161)
14 libisc-9.19.7-dev.dylib 0x00000001050add5c isc__nm_udp_read_cb + 884 (udp.c:620)
15 libuv.1.dylib 0x0000000105dfa614 uv__udp_io + 288
16 libuv.1.dylib 0x0000000105dfd7c0 uv__io_poll + 1744
17 libuv.1.dylib 0x0000000105dedd00 uv_run + 252
18 libisc-9.19.7-dev.dylib 0x00000001050c9c34 loop_run + 460 (loop.c:266)
19 libisc-9.19.7-dev.dylib 0x00000001050c8388 loop_thread + 44 (loop.c:293)
20 libisc-9.19.7-dev.dylib 0x00000001050c824c isc_loopmgr_run + 428 (loop.c:473)
21 named 0x0000000104b9a510 main + 424 (main.c:1548)
22 libdyld.dylib 0x0000000192e51430 start + 4
```
```
options {
dual-stack-servers { 2001:4860:4860::8888; 8.8.8.8; };
};
```December 2022 (9.16.36, 9.16.36-S1, 9.18.10, 9.19.8)Mark AndrewsMark Andrewshttps://gitlab.isc.org/isc-projects/bind9/-/issues/3606Ensure atomicity/synchronisation in isc_nmsocket_shutdown() for multi-layer t...2022-10-26T07:54:37ZArtem BoldarievEnsure atomicity/synchronisation in isc_nmsocket_shutdown() for multi-layer transports (HTTP/2 and Generic TLS code).The current implementation of `isc_nmsocket_shutdown()` for HTTP/2 and generic TLS transport does not explicitly guarantee that clean shutdown is performed and no new associated connections will be accepted after the call of `isc_nmsocke...The current implementation of `isc_nmsocket_shutdown()` for HTTP/2 and generic TLS transport does not explicitly guarantee that clean shutdown is performed and no new associated connections will be accepted after the call of `isc_nmsocket_shutdown()` has been completed, and its associated data is safe to destroy. That could lead to potential issues on a BIND shutdown as well as unit tests for "multilayer" transports.
To fix that, we can do the following:
1. Mark the listening socket as being shutting down so that the associated connections being accepted at the moment might notice that we are wrapping up and handle the situation accordingly;
2. Broadcast a "stop" message on all worker threads and wait for them to be processed. Then, the underlying listening socket can be safely shot down too. Something very similar is being done in other transports implemented directly on top of libUV - but there is a separate "child" socket object associated with any worker. For "multilayer" transports, we can do simpler than that, as there is no direct need to maintain a per-worker object.November 2022 (9.16.35, 9.16.35-S1, 9.18.9, 9.19.7)Artem BoldarievArtem Boldarievhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3603Resolver prefetch eligibility edge case bug2022-11-16T15:22:32ZArаm SаrgsyаnResolver prefetch eligibility edge case bugAccording to ARM, the prefetch "eligibility" parameter is the smallest original TTL value that is accepted for a record to be eligible for prefetching, but in fact, when the original TTL value is equal to the eligibility value, the recor...According to ARM, the prefetch "eligibility" parameter is the smallest original TTL value that is accepted for a record to be eligible for prefetching, but in fact, when the original TTL value is equal to the eligibility value, the record is not treated as eligible for prefetching.
For example, with `prefetch 4 10;` configuration in `tests/system/resolver/ns5` instance, prefetch doesn't work for the following records in `tests/system/resolver/ns4`:
```
fetchall 10 TXT A short ttl
fetchall 10 A 1.2.3.4
fetchall 10 AAAA ::1
```
Although the test passes because it fails to check that prefetch occurred.November 2022 (9.16.35, 9.16.35-S1, 9.18.9, 9.19.7)Arаm SаrgsyаnArаm Sаrgsyаnhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3598ADB quota might not be decremented2022-12-21T17:11:49ZPetr Špačekpspacek@isc.orgADB quota might not be decrementedAffected version: v9_18_4, v9_18_7
Reproducer: unknown at the moment, but happens in production regularly. Special circumstances: Forwarding to OpenDNS which currently blocks HTTPS queries - supplies fake answers for HTTPS queries.
Bas...Affected version: v9_18_4, v9_18_7
Reproducer: unknown at the moment, but happens in production regularly. Special circumstances: Forwarding to OpenDNS which currently blocks HTTPS queries - supplies fake answers for HTTPS queries.
Based on a coredump provided by an user, I think there is a corner case when ADB quota for forwarders is not decremented.
Affected config has 4 forwarders (2x IPv4 + 2xIPv6) together with `forward only` and fetch quotas:
```
forward only;
forwarders { 208.67.222.222; 2620:119:35::35; 208.67.220.220; 2620:119:53::53; };
fetches-per-server 200;
fetch-quota-params 100 0.1 0.3 0.7;
fetches-per-zone 200;
```
All four entries in ADB have quota == active == 200. All queries going outside SERVFAIL immediately.
Log at debug level 99 shows only this:
```
client @0x7fb13e18b568 10.1.23.4#23785: UDP request
client @0x7fb13e18b568 10.1.23.4#23785: view external: using view 'external'
client @0x7fb13e18b568 10.1.23.4#23785 (example.com): view external: rrl=(nil), HAVECOOKIE=0, result=DNS_R_DELEGATION, fname=0x7fb13e34e000(1), is_zone=0, RECURSIONOK=1, query.rpz_st=0x7fb13e1dd000(0), RRL_CHECKED=0
fetch: example.com/A
log_ns_ttl: fctx 0x7fb11f09c800: fctx_create: example.com (in '.'?): 1 498995
QNAME minimization - minimized, qmintype 1 qminname _.com
findaddrinfo: found entry 0x7fb13dc49000
findaddrinfo: found entry 0x7fb13dc49140
findaddrinfo: found entry 0x7fb13dc49280
findaddrinfo: found entry 0x7fb13dc493c0
client @0x7fb13e18b568 10.1.23.4#23785 (example.com): view external: rrl=(nil), HAVECOOKIE=0, result=DNS_R_SERVFAIL, fname=0x7fb13e34e000(0), is_zone=0, RECURSIONOK=1, query.rpz_st=0x7fb13e1dd000(0), RRL_CHECKED=0
client @0x7fb13e18b568 10.1.23.4#23785 (example.com): view external: rpz QNAME rewrite example.com stop on unrecognized qresult in rpz_rewrite() failed: SERVFAIL
client @0x7fb13e18b568 10.1.23.4#23785 (example.com): view external: query failed (SERVFAIL) for example.com/IN/A at query.c:7722
fetch completed at resolver.c:4139 for example.com/A in 0.000000: SERVFAIL/success [domain:.,referral:0,restart:1,qrysent:0,timeout:0,lame:0,quota:0,neterr:0,badresp:0,adberr:0,findfail:0,valfail:0]
```November 2022 (9.16.35, 9.16.35-S1, 9.18.9, 9.19.7)https://gitlab.isc.org/isc-projects/bind9/-/issues/3596"doth" system test fails intermittently with "The single DIG instance has sto...2022-12-12T15:49:55ZMichał Kępień"doth" system test fails intermittently with "The single DIG instance has stopped prematurely"Sample failure: https://gitlab.isc.org/isc-private/bind9/-/jobs/2819987
```
I:doth:checking server quotas for both encrypted and unencrypted HTTP (170)
Traceback (most recent call last):
File "/builds/isc-private/bind9/bin/tests/syste...Sample failure: https://gitlab.isc.org/isc-private/bind9/-/jobs/2819987
```
I:doth:checking server quotas for both encrypted and unencrypted HTTP (170)
Traceback (most recent call last):
File "/builds/isc-private/bind9/bin/tests/system/doth/stress_http_quota.py", line 252, in <module>
sys.exit(main())
File "/builds/isc-private/bind9/bin/tests/system/doth/stress_http_quota.py", line 244, in main
run_test(http_secure=True)
File "/builds/isc-private/bind9/bin/tests/system/doth/stress_http_quota.py", line 225, in run_test
assert (
AssertionError: The single DIG instance has stopped prematurely
I:doth:failed
```
Just in the public BIND 9 project in the past 6 days, this failure mode
has been triggered **over 50 times**.
Except for [two][1] [outliers][2], it always happened in system test jobs
with either ASAN or TSAN enabled and, AFAICT, only for MRs targeting the
`main` branch, so this might be related to !6040 (see also #3595).
[1]: https://gitlab.isc.org/isc-projects/bind9/-/jobs/2811437
[2]: https://gitlab.isc.org/isc-projects/bind9/-/jobs/2813094November 2022 (9.16.35, 9.16.35-S1, 9.18.9, 9.19.7)Artem BoldarievArtem Boldarievhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3592bin/tests/system/start.pl improperly detects server startup after all servers...2022-10-26T09:10:30ZMichał Kępieńbin/tests/system/start.pl improperly detects server startup after all servers are restarted`bin/tests/system/start.pl` waits until a `running` message is logged by
a given name server instance before attempting to send a
`version.bind/CH/TXT` query to it. The idea behind this was to make the
script wait until `named` loads al...`bin/tests/system/start.pl` waits until a `running` message is logged by
a given name server instance before attempting to send a
`version.bind/CH/TXT` query to it. The idea behind this was to make the
script wait until `named` loads all the zones it is configured to serve
before telling the system test framework that a given server is ready to
use; this prevents the need to add boilerplate code that waits for a
specific zone to be loaded to each test expecting that.
However, when looking for `running` messages,
`bin/tests/system/start.pl` assumes that the existence of *any* such
message in the `named.run` file means that a given instance has finished
loading all zones. Meanwhile, some system tests restart all the server
instances throughout their lifetime (some even do that a few times), for
example to run Python-based tests. `bin/tests/system/start.pl` handles
such a scenario incorrectly - as soon as it finds *any* `running`
message in the `named.run` file it inspects and it gets a response to a
`version.bind/CH/TXT` query, it tells the system test framework that a
given server is ready to use, which might not be true - it is possible
that only the `version.bind` zone has been loaded at that point and the
`running` message was logged by a previously-shutdown `named` instance.
This triggers intermittent failures for Python-based tests, e.g.:
https://gitlab.isc.org/isc-private/bind9/-/jobs/2819525
```
D:doth: # Check whether a response was received and whether it is sane.
D:doth: assert response
D:doth: assert query.id == response.id
D:doth:> assert len(response.answer) == 1
D:doth:E assert 0 == 1
D:doth:E +0
D:doth:E -1
D:doth:
D:doth:tests_gnutls.py:102: AssertionError
```
In this case, the response received was a SERVFAIL response (while a
NOERROR response was expected):
```
10-Oct-2022 08:08:10.578 TLS server session created for 10.53.0.1#36440 on 10.53.0.1#9042
10-Oct-2022 08:08:10.578 clientmgr @0x7b2000014100 attach: 2
10-Oct-2022 08:08:10.578 query client=0x7b7c00030368 thread=0x7f6305f7b640(<unknown-query>): query_reset
10-Oct-2022 08:08:10.578 client @0x7b7c00030368 (no-peer): allocate new client
10-Oct-2022 08:08:10.578 client @0x7b7c00030368 10.53.0.1#36440: TCP request
10-Oct-2022 08:08:10.578 client @0x7b7c00030368 10.53.0.1#36440: using view '_default'
10-Oct-2022 08:08:10.578 client @0x7b7c00030368 10.53.0.1#36440: request is not signed
10-Oct-2022 08:08:10.578 client @0x7b7c00030368 10.53.0.1#36440: recursion not available (recursion not enabled for view)
10-Oct-2022 08:08:10.578 query client=0x7b7c00030368 thread=0x7f6305f7b640(<unknown-query>): ns_query_start
10-Oct-2022 08:08:10.578 query client=0x7b7c00030368 thread=0x7f6305f7b640(example/SOA): qctx_init
10-Oct-2022 08:08:10.578 query client=0x7b7c00030368 thread=0x7f6305f7b640(example/SOA): client attr:0x20001, query attr:0x700, restarts:0, origqname:example, timer:0, authdb:0, referral:0
10-Oct-2022 08:08:10.578 query client=0x7b7c00030368 thread=0x7f6305f7b640(example/SOA): ns__query_start
10-Oct-2022 08:08:10.578 query client=0x7b7c00030368 thread=0x7f6305f7b640(example/SOA): ns__query_start: query_getdb failed
10-Oct-2022 08:08:10.578 query client=0x7b7c00030368 thread=0x7f6305f7b640(example/SOA): ns_query_done
10-Oct-2022 08:08:10.578 client @0x7b7c00030368 10.53.0.1#36440 (example): query failed (zone not loaded) for example/IN/SOA at query.c:5644
10-Oct-2022 08:08:10.578 client @0x7b7c00030368 10.53.0.1#36440 (example): reset client
10-Oct-2022 08:08:10.578 query client=0x7b7c00030368 thread=0x7f6305f7b640(example/SOA): query_reset
10-Oct-2022 08:08:10.578 client @0x7b7c00030368 10.53.0.1#36440: freeing client
10-Oct-2022 08:08:10.578 query client=0x7b7c00030368 thread=0x7f6305f7b640(<unknown-query>): query_reset
...
10-Oct-2022 08:08:10.726 zone example/IN: loaded serial 1397051952
```November 2022 (9.16.35, 9.16.35-S1, 9.18.9, 9.19.7)Michał KępieńMichał Kępieńhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3591rbtdb.c:6762: REQUIRE(((rbtnode->nsec == DNS_RBT_NSEC_NSEC3 && (rdataset->typ...2022-11-07T10:37:57ZAsk Bjørn Hansenrbtdb.c:6762: REQUIRE(((rbtnode->nsec == DNS_RBT_NSEC_NSEC3 && (rdataset->type == ((dns_rdatatype_t)dns_rdatatype_nsec3) ...### Summary
Bind crashes after being upgraded.
### BIND version used
```
BIND 9.18.7 (Stable Release) <id:85a6eb1>
running on Linux x86_64 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022
built by make with '--build=x8...### Summary
Bind crashes after being upgraded.
### BIND version used
```
BIND 9.18.7 (Stable Release) <id:85a6eb1>
running on Linux x86_64 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022
built by make with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/opt/isc/isc-bind/root/usr' '--exec-prefix=/opt/isc/isc-bind/root/usr' '--bindir=/opt/isc/isc-bind/root/usr/bin' '--sbindir=/opt/isc/isc-bind/root/usr/sbin' '--sysconfdir=/etc/opt/isc/isc-bind' '--datadir=/opt/isc/isc-bind/root/usr/share' '--includedir=/opt/isc/isc-bind/root/usr/include' '--libdir=/opt/isc/isc-bind/root/usr/lib64' '--libexecdir=/opt/isc/isc-bind/root/usr/libexec' '--localstatedir=/var/opt/isc/isc-bind' '--sharedstatedir=/var/opt/isc/isc-bind/lib' '--mandir=/opt/isc/isc-bind/root/usr/share/man' '--infodir=/opt/isc/isc-bind/root/usr/share/info' '--disable-static' '--enable-dnstap' '--with-pic' '--with-gssapi' '--with-json-c' '--with-libxml2' '--without-lmdb' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro -L/opt/isc/isc-bind/root/usr/lib64' 'CPPFLAGS= -I/opt/isc/isc-bind/root/usr/include' 'LT_SYS_LIBRARY_PATH=/usr/lib64' 'PKG_CONFIG_PATH=:/opt/isc/isc-bind/root/usr/lib64/pkgconfig:/opt/isc/isc-bind/root/usr/share/pkgconfig' 'SPHINX_BUILD=/builddir/build/BUILD/bind-9.18.7/sphinx/bin/sphinx-build'
compiled by GCC 4.8.5 20150623 (Red Hat 4.8.5-44)
compiled with OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017
linked to OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017
compiled with libuv version: 1.41.0
linked to libuv version: 1.44.2
compiled with libnghttp2 version: 1.33.0
linked to libnghttp2 version: 1.33.0
compiled with libxml2 version: 2.9.1
linked to libxml2 version: 20901
compiled with json-c version: 0.11
linked to json-c version: 0.11
compiled with zlib version: 1.2.7
linked to zlib version: 1.2.7
compiled with protobuf-c version: 1.3.3
linked to protobuf-c version: 1.4.1
threads support is enabled
default paths:
named configuration: /etc/opt/isc/isc-bind/named.conf
rndc configuration: /etc/opt/isc/isc-bind/rndc.conf
DNSSEC root key: /etc/opt/isc/isc-bind/bind.keys
nsupdate session key: /var/opt/isc/isc-bind/run/named/session.key
named PID file: /var/opt/isc/isc-bind/run/named/named.pid
named lock file: /var/opt/isc/isc-bind/run/named/named.lock
```
### Steps to reproduce
After upgrading and adding an explicit `inline-signing: yes;` to zones with a `dnssec-policy` bind crashes immediately after startup.
### What is the current *bug* behavior?
```
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: all zones loaded
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: running
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: network unreachable resolving './DNSKEY/IN': 2001:500:12::d0d#53
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: network unreachable resolving './NS/IN': 2001:500:12::d0d#53
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: rbtdb.c:6762: REQUIRE(((rbtnode->nsec == DNS_RBT_NSEC_NSEC3 && (rdataset->type == ((dns_rdatatype_t)dns_rdatatype_nsec3) || rdataset->covers == ((dns_rdatatype_t)dns_rdatatype_nsec3))) || (rbtnode->nsec != DNS_RBT_NSEC_NSEC3 && rdataset->type != ((dns_rdatatype_t)dns_rdatatype_nsec3) && rdataset->covers != ((dns_rdatatype_t)dns_rdatatype_nsec3)))) failed, back trace
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /opt/isc/isc-bind/root/usr/sbin/named() [0x42db32]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /opt/isc/isc-bind/root/usr/lib64/libisc-9.18.7.so(isc_assertion_failed+0xa) [0x7f4887cfa97a]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /opt/isc/isc-bind/root/usr/lib64/libdns-9.18.7.so(+0xd15b3) [0x7f48879845b3]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /opt/isc/isc-bind/root/usr/lib64/libdns-9.18.7.so(+0x1680b0) [0x7f4887a1b0b0]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /opt/isc/isc-bind/root/usr/lib64/libdns-9.18.7.so(+0x19210f) [0x7f4887a4510f]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /opt/isc/isc-bind/root/usr/lib64/libisc-9.18.7.so(isc_task_run+0x174) [0x7f4887d1a244]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /opt/isc/isc-bind/root/usr/lib64/libisc-9.18.7.so(+0x1f43d) [0x7f4887ce143d]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /opt/isc/isc-bind/root/usr/lib64/libisc-9.18.7.so(+0x26df8) [0x7f4887ce8df8]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /opt/isc/isc-bind/root/usr/lib64/libisc-9.18.7.so(+0x2763b) [0x7f4887ce963b]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /opt/isc/isc-bind/root/usr/lib64/libisc-9.18.7.so(+0x28003) [0x7f4887cea003]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /opt/isc/isc-bind/root/usr/lib64/libuv.so.1(+0xf503) [0x7f4886109503]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /opt/isc/isc-bind/root/usr/lib64/libuv.so.1(+0x228f3) [0x7f488611c8f3]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /opt/isc/isc-bind/root/usr/lib64/libuv.so.1(uv_run+0xbd) [0x7f4886109d3d]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /opt/isc/isc-bind/root/usr/lib64/libisc-9.18.7.so(+0x27928) [0x7f4887ce9928]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /opt/isc/isc-bind/root/usr/lib64/libisc-9.18.7.so(isc__trampoline_run+0x15) [0x7f4887d22925]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /lib64/libpthread.so.0(+0x7ea5) [0x7f488532fea5]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: /lib64/libc.so.6(clone+0x6d) [0x7f4885058b0d]
Oct 09 20:38:19 dns2.ewr1.develooper.com named[15953]: exiting (due to assertion failure)
```
### What is the expected *correct* behavior?
Not crashing. :-)
### Relevant configuration files
I'm happy to share privately if it'll help.November 2022 (9.16.35, 9.16.35-S1, 9.18.9, 9.19.7)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.org