BIND issueshttps://gitlab.isc.org/isc-projects/bind9/-/issues2023-05-22T09:58:02Zhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4070Data race in pthread_mutex_destroy (xferquota)2023-05-22T09:58:02ZMichal NowakData race in pthread_mutex_destroy (xferquota)Job [#3387972](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3387972) failed for 15eaf9d3f2d9a3ab8e3ca9dc8ff63f07deed48b7.
The `xferquota` system test produces TSAN error in `system:clang:tsan` CI job.
```
WARNING: ThreadSanitizer: ...Job [#3387972](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3387972) failed for 15eaf9d3f2d9a3ab8e3ca9dc8ff63f07deed48b7.
The `xferquota` system test produces TSAN error in `system:clang:tsan` CI job.
```
WARNING: ThreadSanitizer: data race
Write of size 1 at 0x000000000001 by thread T1:
#0 pthread_mutex_destroy <null> (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#1 qpmulti_destroy_cb lib/dns/qp.c:1442:2 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#2 <null> <null> (BuildId: 317457052cc2fe394e6cea9241fff1d4303f7d0a)
Previous atomic read of size 1 at 0x000000000001 by thread T2 (mutexes: write M1):
#0 pthread_mutex_unlock <null> (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#1 reclaim_chunks_cb lib/dns/qp.c:668:2 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#2 <null> <null> (BuildId: 317457052cc2fe394e6cea9241fff1d4303f7d0a)
Location is heap block of size 168 at 0x000000000007 allocated by main thread:
#0 malloc <null> (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#1 mallocx lib/isc/./jemalloc_shim.h:65:14 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#2 mem_get lib/isc/mem.c:305:8
#3 isc__mem_get lib/isc/mem.c:674:8 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#4 dns_qpmulti_create lib/dns/qp.c:1375:25 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#5 dns_zt_create lib/dns/zt.c:104:2 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#6 dns_view_create lib/dns/view.c:137:2 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#7 create_view bin/named/server.c:6440:11 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#8 load_configuration bin/named/server.c:9140:12 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#9 run_server bin/named/server.c:9982:2 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#10 isc__async_cb lib/isc/async.c:112:3 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#11 uv__async_io /usr/src/libuv-v1.44.1/src/unix/async.c:163:5 (BuildId: 120c450d14885aa5308bc95c4ea77de2c2b1cc36)
#12 thread_body lib/isc/thread.c:87:8 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#13 isc_thread_main lib/isc/thread.c:118:2
#14 isc_loopmgr_run lib/isc/loop.c:452:2 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#15 main bin/named/main.c:1532:2 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
Mutex M1 (0x000000000001) created at:
#0 pthread_mutex_init <null> (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#1 dns_qpmulti_create lib/dns/qp.c:1380:2 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#2 dns_zt_create lib/dns/zt.c:104:2 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#3 dns_view_create lib/dns/view.c:137:2 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#4 create_view bin/named/server.c:6440:11 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#5 load_configuration bin/named/server.c:9140:12 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#6 run_server bin/named/server.c:9982:2 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#7 isc__async_cb lib/isc/async.c:112:3 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#8 uv__async_io /usr/src/libuv-v1.44.1/src/unix/async.c:163:5 (BuildId: 120c450d14885aa5308bc95c4ea77de2c2b1cc36)
#9 thread_body lib/isc/thread.c:87:8 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#10 isc_thread_main lib/isc/thread.c:118:2
#11 isc_loopmgr_run lib/isc/loop.c:452:2 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#12 main bin/named/main.c:1532:2 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
Thread T1 (running) created by main thread at:
#0 pthread_create <null> (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#1 <null> <null> (BuildId: 317457052cc2fe394e6cea9241fff1d4303f7d0a)
#2 main bin/named/main.c:1532:2 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
Thread T2 (running) created by main thread at:
#0 pthread_create <null> (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#1 <null> <null> (BuildId: 317457052cc2fe394e6cea9241fff1d4303f7d0a)
#2 main bin/named/main.c:1532:2 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
SUMMARY: ThreadSanitizer: data race (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098) in pthread_mutex_destroy
```
In daily CI pipelines, it started on Saturday when the `fanf-urcu-qp` branch was merged, and happens with 75 % reliability since.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Tony FinchTony Finchhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4069upforwd test failure2023-05-17T14:21:58ZEvan Huntupforwd test failureWhen CI is running low, the upforwd test fails. Previously this could cause an assertion failure, which has now been fixed by #4064, but we still need to address the failure.
The failure is described at https://gitlab.isc.org/isc-projec...When CI is running low, the upforwd test fails. Previously this could cause an assertion failure, which has now been fixed by #4064, but we still need to address the failure.
The failure is described at https://gitlab.isc.org/isc-projects/bind9/-/issues/4064#note_372499. It usually looks like this:
```
I:upforwd:checking DNSTAP logging of UPDATE forwarded update replies (26)
I:upforwd:UQ=4 UR=1
I:upforwd:failed
...
I:upforwd:checking DNSTAP logging of UPDATE forwarded update replies (28)
I:upforwd:UQ=3 UR=1
I:upforwd:failed
```
As I said above, I suspect this is because the test platform is slow: some update forwarding requests are timing out before replies are sent, and so we count more requests (UQ) than replies (UR) in the DNSTAP stats.
I'd love to find out why these CI jobs are so draggy. It shouldn't be taking 30-40 minutes for a system test job to run. It's possible there's some environmental issue, and if we fixed it the test failures would go away. That said, however, I think we can fix the test so it's more resilient.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Tony FinchTony Finchhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4068CID 455002: Error handling issues in lib/isc/loop.c2023-05-15T20:49:24ZMichal NowakCID 455002: Error handling issues in lib/isc/loop.cI think this Coverity Scan issue was triggered by adding two `uv_async_send()` calls in 7b1d985de2f3be760ea5fedd191482fb943ed934 that check return values, so Coverity thinks we might want to check `uv_async_send()`s return value elsewher...I think this Coverity Scan issue was triggered by adding two `uv_async_send()` calls in 7b1d985de2f3be760ea5fedd191482fb943ed934 that check return values, so Coverity thinks we might want to check `uv_async_send()`s return value elsewhere as well:
```
/lib/isc/loop.c: 483 in isc_loopmgr_pause()
477
478 /* Skip current loop */
479 if (i == isc_tid()) {
480 continue;
481 }
482
>>> CID 455002: Error handling issues (CHECKED_RETURN)
>>> Calling "uv_async_send" without checking return value (as is done elsewhere 5 out of 6 times).
483 uv_async_send(&loop->pause_trigger);
484 }
485
486 RUNTIME_CHECK(atomic_compare_exchange_strong(&loopmgr->paused,
487 &(bool){ false }, true));
488 pause_loop(CURRENT_LOOP(loopmgr));
```
For your consideration @fanf.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Tony FinchTony Finchhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4067Configuring with --with-liburcu=qsbr won't build2023-05-15T20:49:47ZMichal NowakConfiguring with --with-liburcu=qsbr won't buildThe [`pairwise`](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3385043) test fails with
```
In file included from ../../../lib/isc/include/isc/job.h:31,
from ../../../lib/isc/include/isc/async.h:28,
...The [`pairwise`](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3385043) test fails with
```
In file included from ../../../lib/isc/include/isc/job.h:31,
from ../../../lib/isc/include/isc/async.h:28,
from ../../../lib/dns/rbtdb.c:22:
../../../lib/dns/rbtdb.c: In function 'rdataset_addglue':
../../../lib/isc/include/isc/urcu.h:113:3: error: expected expression before 'if'
113 | if (!urcu_qsbr_read_ongoing()) { \
| ^~
../../../lib/isc/include/isc/urcu.h:120:30: note: in expansion of macro 'isc_qsbr_rcu_dereference'
120 | #define rcu_dereference(ptr) isc_qsbr_rcu_dereference(ptr)
| ^~~~~~~~~~~~~~~~~~~~~~~~
../../../lib/dns/rbtdb.c:9857:23: note: in expansion of macro 'rcu_dereference'
9857 | rbtdb_glue_t *glue = rcu_dereference(header->glue_list);
| ^~~~~~~~~~~~~~~
```
for the following configuration:
```
--disable-developer --enable-warn-error --with-liburcu=qsbr --disable-kqueue --enable-epoll --disable-devpoll --disable-geoip --with-locktype=adaptive --enable-doh --with-libnghttp2=auto --enable-pthread-rwlock --with-gssapi=auto --with-lmdb=auto --without-libxml2 --with-json-c=detect --with-zlib=auto --without-libsystemd --enable-tcp-fastopen --with-readline=yes --enable-chroot --enable-fixed-rrset --disable-dnstap --without-libidn2 --with-cmocka=yes --without-jemalloc --disable-leak-detection --enable-singletrace --enable-querytrace --disable-auto-validation --enable-dnsrps --enable-dnsrps-dl --disable-full-report
```
`--with-liburcu=qsbr` is the culprit here.
Starts with !7895.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Tony FinchTony Finchhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4066resolv.conf parsing eats lines if more than 3 nameservers set2023-05-17T22:53:09ZRobert Bridgeresolv.conf parsing eats lines if more than 3 nameservers set<!--
If the bug you are reporting is potentially security-related - for example,
if it involves an assertion failure or other crash in `named` that can be
triggered repeatedly - then please make sure that you make the new issue
confident...<!--
If the bug you are reporting is potentially security-related - for example,
if it involves an assertion failure or other crash in `named` that can be
triggered repeatedly - then please make sure that you make the new issue
confidential!
-->
### Summary
The resolv.conf parsing used in nslookup eats the lines of resolv.conf if there are more than 3 nameservers defined in resolv.conf. This means that if there is an even number of nameservers defined, the first line following the nameservers gets silently eaten and ignored.
### BIND version used
Identified on CentOS 9 stream, confirmed from git on gentoo:
```
BIND 9.19.14-dev (Development Release) <id:562697e>
running on Linux x86_64 6.2.9-gentoo #2 SMP PREEMPT_DYNAMIC Mon May 1 09:27:12 BST 2023
built by make with default
compiled by GCC 13.1.0
compiled with OpenSSL version: OpenSSL 3.0.8 7 Feb 2023
linked to OpenSSL version: OpenSSL 3.0.8 7 Feb 2023
compiled with libuv version: 1.44.2
linked to libuv version: 1.44.2
compiled with liburcu version: 0.14.0
compiled with libnghttp2 version: 1.52.0
linked to libnghttp2 version: 1.52.0
compiled with libxml2 version: 2.11.3
linked to libxml2 version: 21103
compiled with json-c version: 0.16
linked to json-c version: 0.16
compiled with zlib version: 1.2.13
linked to zlib version: 1.2.13
threads support is enabled
DNSSEC algorithms: RSASHA1 NSEC3RSASHA1 RSASHA256 RSASHA512 ECDSAP256SHA256 ECDSAP384SHA384 ED25519 ED448
DS algorithms: SHA-1 SHA-256 SHA-384
HMAC algorithms: HMAC-MD5 HMAC-SHA1 HMAC-SHA224 HMAC-SHA256 HMAC-SHA384 HMAC-SHA512
TKEY mode 2 support (Diffie-Hellman): no
TKEY mode 3 support (GSS-API): no
default paths:
named configuration: /usr/local/etc/named.conf
rndc configuration: /usr/local/etc/rndc.conf
nsupdate session key: /usr/local/var/run/named/session.key
named PID file: /usr/local/var/run/named/named.pid
named lock file: /usr/local/var/run/named/named.lock
```
### Steps to reproduce
Create resolv.conf with 4/6/8 nameserver entries, and a search line immediately after the last nameserver entry, e.g.:
```
nameserver 8.8.8.8
nameserver 8.8.8.8
nameserver 8.8.8.8
nameserver 8.8.8.8
search google.com
```
Run nslookup for a name that relies on the search line.
```
nslookup www
```
### What is the current *bug* behavior?
nslookup returns NXDOMAIN (typically)
```
# nslookup www
Server: 8.8.8.8
Address: 8.8.8.8#53
** server can't find www: NXDOMAIN
```
### What is the expected *correct* behavior?
nslookup should search the domains and find the relevant record
```
$ ./bin/dig/nslookup www
Server: 8.8.8.8
Address: 8.8.8.8#53
Non-authoritative answer:
Name: www.google.com
Address: 172.217.16.228
Name: www.google.com
Address: 2a00:1450:4009:820::2004
```
### Relevant configuration files
/etc/resolv.conf posted above
### Relevant logs and/or screenshots
### Possible fixesJune 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Mark AndrewsMark Andrewshttps://gitlab.isc.org/isc-projects/bind9/-/issues/4064upforwd test causes assertion failure in streamdns.c2023-05-17T13:49:54ZTony Finchupforwd test causes assertion failure in streamdns.cI modified the `upforwd` test to add a delay:
```
--- bin/tests/system/upforwd/tests.sh
+++ bin/tests/system/upforwd/tests.sh
@@ -23,6 +23,8 @@ RNDCCMD="$RNDC -p ${CONTROLPORT} -c ../common/rndc.conf"
status=0
n=1
capture_dnstap() {
+...I modified the `upforwd` test to add a delay:
```
--- bin/tests/system/upforwd/tests.sh
+++ bin/tests/system/upforwd/tests.sh
@@ -23,6 +23,8 @@ RNDCCMD="$RNDC -p ${CONTROLPORT} -c ../common/rndc.conf"
status=0
n=1
capture_dnstap() {
+ echo_i "sleeping 15 seconds"
+ sleep 15
retry_quiet 20 test -f ns3/dnstap.out && mv ns3/dnstap.out dnstap.out.$n
$RNDCCMD -s 10.53.0.3 dnstap -reopen
}
```
With this change, I get the following failure fairly repeatably:
```
12-May-2023 18:59:21.745 dispatch 0x7b5000050000: TCP read:timed out:requests 20
12-May-2023 18:59:21.745 dispatch 0x7b5000050000: TCP response 0x7b5000050200: reading from 0x7b4c00040180
12-May-2023 18:59:21.745 netmgr/streamdns.c:834: REQUIRE(sock->recv_handle == ((void *)0)) failed
```
The failures also happen with revision 88cf7e7e9a5c70201d3f981e44b372785d19fb85 (main yesterday before RCU changes), when compiled with thread sanitizer and without.
I am running BIND on my Debian Bullseye box.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)https://gitlab.isc.org/isc-projects/bind9/-/issues/4055[CVE-2023-2828] named's configured cache size limit can be significantly exce...2023-11-15T08:47:01ZShoham Danino[CVE-2023-2828] named's configured cache size limit can be significantly exceeded| Quick Links | :link: |
| ------------------------ | ------------------------------------------------------------------------------|
| Incident Manager:...| Quick Links | :link: |
| ------------------------ | ------------------------------------------------------------------------------|
| Incident Manager: | @michal |
| Deputy Incident Manager: | @aram |
| Public Disclosure Date: | 2023-06-21 |
| CVSS Score: | [7.5][cvss_score] |
| Security Advisory: | isc-private/printing-press!54 |
| Mattermost Channel: | [CVE-2023-2828: max-cache-size can be significantly exceeded][mattermost_url] |
| Support Ticket: | N/A |
| Release Checklist: | https://gitlab.isc.org/isc-projects/bind9/-/issues/4123 |
| Post-mortem Etherpad: | [postmortem-2023-06][postmortem_url] |
[cvss_score]: https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H&version=3.1
[mattermost_url]: https://mattermost.isc.org/isc/channels/cve-2023-2828-max-cache-size
[postmortem_url]: https://pad.isc.org/p/postmortem-2023-06
:bulb: **Click [here][checklist_explanations] (internal resource) for general information about the security incident handling process.**
[checklist_explanations]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations
### Earlier Than T-5
- [x] [:link:][step_deputy] **(IM)** Pick a Deputy Incident Manager
- [x] [:link:][step_respond] **(IM)** [Respond to the bug reporter](#note_373642)
- [x] [:link:][step_etherpad] **(IM)** [Create an Etherpad for post-mortem][postmortem_url]
- [x] [:link:][step_public_mrs] **(SwEng)** Ensure there are no public merge requests which inadvertently disclose the issue
- [x] [:link:][step_assign_cve_id] **(IM)** [Assign a CVE identifier](#note_374026)
- [x] [:link:][step_note_cve_info] **(SwEng)** Update this issue with the assigned CVE identifier and the CVSS score
- [x] [:link:][step_versions_affected] **(SwEng)** [Determine the range of product versions affected (including the Subscription Edition)](#note_374064)
- [x] [:link:][step_workarounds] **(SwEng)** [Determine whether workarounds for the problem exist](#note_374750)
- [x] [:link:][step_coordinate] **(SwEng)** ~~If necessary, coordinate with other parties~~
- [x] [:link:][step_earliest] **(Support)** Prepare and send out "earliest" notifications
- [x] [:link:][step_advisory_mr] **(Support)** [Create a merge request for the Security Advisory and include all readily available information in it](isc-private/printing-press!54)
- [x] [:link:][step_reproducer_mr] **(SwEng)** [Prepare a private merge request containing a system test reproducing the problem](isc-private/bind9!519)
- [x] [:link:][step_notify_support] **(SwEng)** [Notify Support when a reproducer is ready](https://mattermost.isc.org/isc/pl/bfjbhijg1jnqzxoix8q6uhsp7e)
- [x] [:link:][step_code_analysis] **(SwEng)** [Prepare a detailed explanation of the code flow triggering the problem](#note_372558)
- [x] [:link:][step_fix_mr] **(SwEng)** [Prepare a private merge request with the fix](isc-private/bind9!520)
- [x] [:link:][step_review_fix] **(SwEng)** [Ensure the merge request with the fix is reviewed and has no outstanding discussions](https://gitlab.isc.org/isc-private/bind9/-/merge_requests/520#note_378278)
- [x] [:link:][step_review_docs] **(Support)** [Review the documentation changes introduced by the merge request with the fix](https://mattermost.isc.org/isc/pl/ys9ngz8hpffitrdefochzi1ayh)
- [x] [:link:][step_backports] **(SwEng)** Prepare backports of the merge request addressing the problem for all affected ([and](https://gitlab.isc.org/isc-private/bind9/-/merge_requests/527) [still](https://gitlab.isc.org/isc-private/bind9/-/merge_requests/528) [maintained](https://gitlab.isc.org/isc-private/bind9/-/merge_requests/529)) branches of a given product
- [x] [:link:][step_finish_advisory] **(Support)** [Finish preparing the Security Advisory](https://mattermost.isc.org/isc/pl/nt6beetinpnwzg8678sb5fi79r)
- [x] [:link:][step_meta_issue] **(QA)** [Create (or update) the private issue containing links to fixes & reproducers for all CVEs fixed in a given release cycle](https://gitlab.isc.org/isc-private/bind9/-/issues/68)
- [x] [:link:][step_changes] **(QA)** (BIND 9 only) [Reserve a block of `CHANGES` placeholders once the complete set of vulnerabilities fixed in a given release cycle is determined](isc-projects/bind9!8010)
- [x] [:link:][step_merge_fixes] **(QA)** Merge the CVE fixes in CVE identifier order
- [x] [:link:][step_patches] **(QA)** [Prepare a standalone patch for the last stable release of each affected (and still maintained) product branch](https://mattermost.isc.org/isc/pl/apq5r8ir7ffnb8br3bdpkqbg5a)
- [x] [:link:][step_asn_releases] **(QA)** [Prepare ASN releases (as outlined in the Release Checklist)](https://mattermost.isc.org/isc/pl/gbe5fz3bypfixqt67b85tapruc)
### At T-5
- [x] [:link:][step_send_asn] **(Support)** [Send ASN to eligible customers](https://mattermost.isc.org/isc/pl/43d9p7bou7g79e1zbnp91fooxr)
- [x] [:link:][step_preannouncement] **(Support)** [(BIND 9 only) Send a pre-announcement email to the *bind-announce* mailing list to alert users that the upcoming release will include security fixes](isc-private/printing-press!57)
### At T-4
- [x] [:link:][step_verify_asn] **(Support)** Verify that all ASN-eligible customers have received the notification email
### At T-1
- [x] [:link:][step_check_customers] **(Support)** Verify that any new or reinstated customers have received the notification email
- [x] [:link:][step_packager_emails] **(First IM)** [Send notifications to OS packagers](https://gitlab.isc.org/isc-private/printing-press/-/merge_requests/58#note_382746)
### On the Day of Public Disclosure
- [x] [:link:][step_clearance] **(IM)** [Grant Support clearance to proceed with public release](https://mattermost.isc.org/isc/pl/kx7cyamrwiysdf3wqgq7qwo4me)
- [x] [:link:][step_publish] **(Support)** Publish the releases (as outlined in the release checklist)
- [x] [:link:][step_matrix] **(Support)** (BIND 9 only) Update vulnerability matrix in the Knowledge Base
- [x] [:link:][step_publish_advisory] **(Support)** Bump Document Version for the Security Advisory and publish it in the Knowledge Base
- [x] [:link:][step_notifications] **(First IM)** [Send notification emails to third parties](https://gitlab.isc.org/isc-private/printing-press/-/merge_requests/61#note_383415)
- [x] [:link:][step_mitre] **(First IM)** [Advise MITRE about the disclosed CVEs](https://gitlab.isc.org/isc-private/printing-press/-/merge_requests/54#note_383433)
- [x] [:link:][step_merge_advisory] **(First IM)** Merge the Security Advisory merge request
- [x] [:link:][step_embargo_end] **(IM)** [Inform original reporter (if external) that the security disclosure process is complete](#note_383448)
- [x] [:link:][step_customers] **(Support)** Inform customers a fix has been released
### After Public Disclosure
- [x] [:link:][step_postmortem] **(First IM)** [Organize post-mortem meeting and make sure it happens](https://mattermost.isc.org/isc/pl/8bh5bx4betgbxx67zoxfa3zioc)
- [x] [:link:][step_tickets] **(Support)** Close support tickets
- [x] [:link:][step_regression] **(QA)** Merge a regression test reproducing the bug into all affected (and still maintained) branches
[step_deputy]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#pick-a-deputy-incident-manager
[step_respond]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#respond-to-the-bug-reporter
[step_etherpad]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-an-etherpad-for-post-mortem
[step_public_mrs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-there-are-no-public-merge-requests-which-inadvertently-disclose-the-issue
[step_assign_cve_id]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#assign-a-cve-identifier
[step_note_cve_info]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-this-issue-with-the-assigned-cve-identifier-and-the-cvss-score
[step_versions_affected]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-the-range-of-product-versions-affected-including-the-subscription-edition
[step_workarounds]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-whether-workarounds-for-the-problem-exist
[step_coordinate]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#if-necessary-coordinate-with-other-parties
[step_earliest]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-and-send-out-earliest-notifications
[step_advisory_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-a-merge-request-for-the-security-advisory-and-include-all-readily-available-information-in-it
[step_reproducer_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-containing-a-system-test-reproducing-the-problem
[step_notify_support]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#notify-support-when-a-reproducer-is-ready
[step_code_analysis]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-detailed-explanation-of-the-code-flow-triggering-the-problem
[step_fix_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-with-the-fix
[step_review_fix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-the-merge-request-with-the-fix-is-reviewed-and-has-no-outstanding-discussions
[step_review_docs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#review-the-documentation-changes-introduced-by-the-merge-request-with-the-fix
[step_backports]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-backports-of-the-merge-request-addressing-the-problem-for-all-affected-and-still-maintained-branches-of-a-given-product
[step_finish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#finish-preparing-the-security-advisory
[step_meta_issue]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-or-update-the-private-issue-containing-links-to-fixes-reproducers-for-all-cves-fixed-in-a-given-release-cycle
[step_changes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-reserve-a-block-of-changes-placeholders-once-the-complete-set-of-vulnerabilities-fixed-in-a-given-release-cycle-is-determined
[step_merge_fixes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-cve-fixes-in-cve-identifier-order
[step_patches]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-standalone-patch-for-the-last-stable-release-of-each-affected-and-still-maintained-product-branch
[step_asn_releases]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-asn-releases-as-outlined-in-the-release-checklist
[step_send_asn]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-asn-to-eligible-customers
[step_preannouncement]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-send-a-pre-announcement-email-to-the-bind-announce-mailing-list-to-alert-users-that-the-upcoming-release-will-include-security-fixes
[step_verify_asn]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#verify-that-all-asn-eligible-customers-have-received-the-notification-email
[step_check_customers]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#verify-that-any-new-or-reinstated-customers-have-received-the-notification-email
[step_packager_emails]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notifications-to-os-packagers
[step_clearance]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#grant-support-clearance-to-proceed-with-public-release
[step_publish]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#publish-the-releases-as-outlined-in-the-release-checklist
[step_matrix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-update-vulnerability-matrix-in-the-knowledge-base
[step_publish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bump-document-version-for-the-security-advisory-and-publish-it-in-the-knowledge-base
[step_notifications]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notification-emails-to-third-parties
[step_mitre]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#advise-mitre-about-the-disclosed-cves
[step_merge_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-security-advisory-merge-request
[step_embargo_end]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#inform-original-reporter-if-external-that-the-security-disclosure-process-is-complete
[step_customers]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#inform-customers-a-fix-has-been-released
[step_postmortem]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#organize-post-mortem-meeting-and-make-sure-it-happens
[step_tickets]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#close-support-tickets
[step_regression]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-a-regression-test-reproducing-the-bug-into-all-affected-and-still-maintained-branches
---
### Summary
This vulnerability results in high memory cache usage for a DNS resolver, even larger than the maximum cache size configured. This happens when the resolver gets around 20,000 requests in several minutes or hours.
For example, with a 250 QPS rate, 1000MB RAM is used after 80 seconds when cache max size is configured to 32MB (the results example is attached to this message).
### BIND version used
BIND 9.16.40 (Extended Support Version) <id:113a865>
### Steps to reproduce
We reproduce NRDelegationAttack with some changes, for more details: https://www.usenix.org/system/files/sec23fall-prepub-309-afek.pdf
1. set the maximum cache size to 32MB:
in named.conf.option (attached example:[named.conf.options](/uploads/bd5dadb4674f120dbe96fd6ca5d060ba/named.conf.options)):
```
options {
...
max-cache-size 32m;
...
}
```
2. Run the resolver: `named -g -c /etc/named.conf`
3. Run psrecord for testing the RAM usage of the resolver: `psrecord NAMED_PID --interval 1 --plot OUTPUT_FILE.png`
4. Option a:
> Request my domain (shoham-shani.online) up to 50,000 dns queries (my authoritative ip address is 74.234.116.29):
`dig shoham{count}.shoham-shani.online. @resolver_ip (count is from 0 to 49,999)` (you can use dnsperf: `dnsperf -d queries.txt -s resolver_ip -v -Q 250`, an example to queries.txt is attached: [queries.txt](/uploads/3a710d8c8416c83ce0469dc7d6b7c92a/queries.txt)
> You can provide us with a test resolver that you want us to attack, and we will perform the attack from our client side at the time we will agree on.
5. option b:
> Create a zone file (example is attached) that has 1500 referrals per one request, you can use this script for that:
with open('zonfile.txt', 'w') as f:
for i in range(1, 50000):
for j in range(0, 1500):
print(f'shoham{i} 8600 IN NS attack{j}.auth{j}.shoham.store.',file=f)
[shoham-shani.online_zonefile_example.txt](/uploads/1cce5b5b5a7118205bde4f199e1c3404/shoham-shani.online_zonefile_example.txt)
> Create another zonefile that answers all shoham{i}.shoham.store. request:
For example: `* IN A 127.0.0.1`
[shoham.store_zonefile_example_copy.txt](/uploads/1fbda03a3a24fdffe95df466d3b031e6/shoham.store_zonefile_example_copy.txt)
> Request 50,000 dns queries as I described in option a.
6. Take a dump of the cache and examine its size: `rndc dumpdb -cache`
7. Stop the resolver service and download the OUTPUT_FILE.png image, examining RAM usage.
note: we checked the bug with 32MB, 64MB and 1GB max-cache-size and with rate of 1,5,10,13,100,250 QPS (all the results are attached)
For 1 QPS I got 440MB RAM used after 8,000 seconds for max-cache-size 32MB
![attack_1_qps](/uploads/4e918e5100db74bae11edf5df8a485bd/attack_1_qps.png)
For 5 QPS I got 840MB RAM used after 4,000 seconds for max-cache-size 32MB
![attack_5_qps](/uploads/c55932c784c402d868af2d306efb6795/attack_5_qps.png)
For 10 QPS I got 840MB RAM used after 2,000 seconds for max-cache-size 32MB
![attack_10_qps](/uploads/d4293561ddc6ba22d8ee203fb2244249/attack_10_qps.png)
For 100 QPS I got 840MB RAM used after 200 seconds for max-cache-size 32MB
![attack_100_qps](/uploads/a18150c0b47480e8ff9d5df5d68afce4/attack_100_qps.png)
For 250 QPS I got 1000MB RAM used after 80 seconds for max-cache-size 32MB
![attack_250_qps](/uploads/116feeb2217d76b84aa3427a0a52db32/attack_250_qps.png)
For 13 QPS I got 1550MB RAM used after 1150 seconds for max-cache-size 1000MB
![attack_1GB_cache](/uploads/bc7a472599212d5b1feed1563ee014bb/attack_1GB_cache.jpeg)
### What is the current *bug* behavior?
1. The cache size expands beyond the limit resulting in an increasing amount of memory being allocated. In addition, if there is no memory available on the machine, the resolver service will crash.
2. A free memory action is not performed for the referral list buffer, which results in an increase in memory allocation for buffers (dns_rdataslab_fromrdataset function in line 272 of the rdataslab.c file).
3. It seems the cache size calculation does not consider authoritative nameservers' refferal answers, although they are stored in the cache.
4. High and low watermarks are set incorrectly to 0, which means the resolver is unaware that the memory usage exceeds the maximum level and does not reduce it.
### What is the expected *correct* behavior?
1. The cache size should be the maximum size we configured.
2. There should be a free memory action to the buffer (rawbuf in rdataslab.c file)
3. In order to calculate cache size, it is necessary to take into account referral list and perform a deletion when the cache exceeds the configured limit.
### Relevant configuration files
configuration file:
> named.conf.options
zonefiles:
> shoham-shani.online_zonefile_example.txt
> shoham.store_zonefile_example copy.txt
### Relevant logs and/or screenshots
Tests are attached
### Possible fixes
1. Free the buffer:
```
free_rawbuf:
isc_mem_put(mctx, rawbuf, buflen);
```
please add the following to this issue:
Yehuda Afek, yehuda.afek@gmail.com /cc @afek
Anat Bremler-Barr, anat.bremlerbarr@gmail.com /cc @anat_bremler_barr
Yuval Shavitt, shavitt@eng.tau.ac.ilJune 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Tom KrizekTom Krizekhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4054Reference counting failure "REQUIRE(ptr != ((void *)0))" in db.c2023-05-17T13:16:28ZMichal NowakReference counting failure "REQUIRE(ptr != ((void *)0))" in db.cJob [#3368111](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3368111) failed for 9b28225611f34dac71bd3885e15ad554ccb60a97.
`ns3` of the `inline` test hit a reference counting issue `REQUIRE(ptr != ((void *)0))` in `db.c`.
```
D:/bui...Job [#3368111](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3368111) failed for 9b28225611f34dac71bd3885e15ad554ccb60a97.
`ns3` of the `inline` test hit a reference counting issue `REQUIRE(ptr != ((void *)0))` in `db.c`.
```
D:/builds/isc-projects/bind9/bin/tests/system/inline:Core was generated by `/builds/isc-projects/bind9/bin/named/.libs/named -D inline-ns3 -X named.lock -m'.
D:/builds/isc-projects/bind9/bin/tests/system/inline:Program terminated with signal SIGABRT, Aborted.
D:/builds/isc-projects/bind9/bin/tests/system/inline:#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
D:/builds/isc-projects/bind9/bin/tests/system/inline:[Current thread is 1 (Thread 0x7f6719e53680 (LWP 1823))]
D:/builds/isc-projects/bind9/bin/tests/system/inline:#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
D:/builds/isc-projects/bind9/bin/tests/system/inline:#1 0x00007f67205f5d2f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
D:/builds/isc-projects/bind9/bin/tests/system/inline:#2 0x00007f67205a6ef2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
D:/builds/isc-projects/bind9/bin/tests/system/inline:#3 0x00007f6720591472 in __GI_abort () at ./stdlib/abort.c:79
D:/builds/isc-projects/bind9/bin/tests/system/inline:#4 0x000055a4e5ebb96a in assertion_failed (file=<optimized out>, line=150, type=isc_assertiontype_require, cond=0x7f67211a7b8d "ptr != ((void *)0)") at main.c:225
D:/builds/isc-projects/bind9/bin/tests/system/inline:#5 0x00007f672121f29a in isc_assertion_failed (file=file@entry=0x7f672118d510 "db.c", line=line@entry=150, type=type@entry=isc_assertiontype_require, cond=cond@entry=0x7f67211a7b8d "ptr != ((void *)0)") at assertions.c:48
D:/builds/isc-projects/bind9/bin/tests/system/inline:#6 0x00007f6721011801 in dns_db_ref (ptr=<optimized out>) at db.c:150
D:/builds/isc-projects/bind9/bin/tests/system/inline:#7 0x00007f67210118ad in dns_db_attach (ptr=0x0, ptrp=ptrp@entry=0x7f6719e51f90) at db.c:150
D:/builds/isc-projects/bind9/bin/tests/system/inline:#8 0x00007f6721169559 in zone_resigninc (zone=zone@entry=0x55a4e64d2710) at zone.c:6828
D:/builds/isc-projects/bind9/bin/tests/system/inline:#9 0x00007f672117214f in zone_maintenance (zone=<optimized out>) at zone.c:11100
D:/builds/isc-projects/bind9/bin/tests/system/inline:#10 0x00007f67211750e6 in zone_timer (arg=<optimized out>) at zone.c:14634
D:/builds/isc-projects/bind9/bin/tests/system/inline:#11 0x00007f672124776e in timer_cb (handle=<optimized out>) at timer.c:111
D:/builds/isc-projects/bind9/bin/tests/system/inline:#12 0x00007f6720a35cbe in ?? () from /lib/x86_64-linux-gnu/libuv.so.1
D:/builds/isc-projects/bind9/bin/tests/system/inline:#13 0x00007f6720a3999f in uv_run () from /lib/x86_64-linux-gnu/libuv.so.1
D:/builds/isc-projects/bind9/bin/tests/system/inline:#14 0x00007f6721234748 in loop_thread (arg=arg@entry=0x55a4e613b4b8) at loop.c:311
D:/builds/isc-projects/bind9/bin/tests/system/inline:#15 0x00007f6721246012 in thread_body (wrap=0x55a4e61782f0) at thread.c:89
D:/builds/isc-projects/bind9/bin/tests/system/inline:#16 thread_run (wrap=0x55a4e61782f0) at thread.c:104
D:/builds/isc-projects/bind9/bin/tests/system/inline:#17 0x00007f67205f3fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
D:/builds/isc-projects/bind9/bin/tests/system/inline:#18 0x00007f6720673820 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
```
It seems that `named` crashed shortly before the `testing that inline signing works with inactive ZSK and active KSK` test.
CI artifacts were saved in the job.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Arаm SаrgsyаnArаm Sаrgsyаnhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4053CID 453470: Use after free in lib/ns/client.c2023-06-14T10:19:31ZMichal NowakCID 453470: Use after free in lib/ns/client.cCoverity Scan claims use after free in `lib/ns/client.c`.
```c
1. Switch case value ns_cookiealg_aes.
1146 switch (client->manager->sctx->cookiealg) {
1147 case ns_cookiealg_siphash24: {
1148 unsigned ch...Coverity Scan claims use after free in `lib/ns/client.c`.
```c
1. Switch case value ns_cookiealg_aes.
1146 switch (client->manager->sctx->cookiealg) {
1147 case ns_cookiealg_siphash24: {
1148 unsigned char input[16 + 16] ISC_NONSTRING = { 0 };
1149 size_t inputlen = 0;
1150 isc_netaddr_t netaddr;
1151 unsigned char *cp;
1152
1153 cp = isc_buffer_used(buf);
1154 isc_buffer_putmem(buf, client->cookie, 8);
1155 isc_buffer_putuint8(buf, NS_COOKIE_VERSION_1);
1156 isc_buffer_putuint8(buf, 0); /* Reserved */
1157 isc_buffer_putuint16(buf, 0); /* Reserved */
1158 isc_buffer_putuint32(buf, when);
1159
CID 453470 (2) (#1-3 of 4): Use after free (USE_AFTER_FREE) [select issue]
1160 memmove(input, cp, 16);
1161
1162 isc_netaddr_fromsockaddr(&netaddr, &client->peeraddr);
1163 switch (netaddr.family) {
1164 case AF_INET:
1165 cp = (unsigned char *)&netaddr.type.in;
1166 memmove(input + 16, cp, 4);
1167 inputlen = 20;
1168 break;
1169 case AF_INET6:
1170 cp = (unsigned char *)&netaddr.type.in6;
1171 memmove(input + 16, cp, 16);
1172 inputlen = 32;
1173 break;
1174 default:
1175 UNREACHABLE();
1176 }
1177
1178 isc_siphash24(secret, input, inputlen, true, digest);
1179 isc_buffer_putmem(buf, digest, 8);
1180 break;
1181 }
1182 case ns_cookiealg_aes: {
1183 unsigned char input[4 + 4 + 16] ISC_NONSTRING = { 0 };
1184 isc_netaddr_t netaddr;
1185 unsigned char *cp;
1186 unsigned int i;
1187
2. assign: Assigning: cp = (void *)((unsigned char *)buf->base + buf->used).
1188 cp = isc_buffer_used(buf);
1189 isc_buffer_putmem(buf, client->cookie, 8);
1190 isc_buffer_putuint32(buf, nonce);
3. freed_arg: isc_buffer_putuint32 frees buf->base. [show details]
1191 isc_buffer_putuint32(buf, when);
CID 453470 (#2-4 of 4): Use after free (USE_AFTER_FREE)
deref_arg: Calling memmove dereferences freed pointer cp. [Note: The source code implementation of the function has been overridden by a builtin model.]
1192 memmove(input, cp, 16);
```
Note that it might not be a new issue, but something the new Coverity Scan 2022.12 detected.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)https://gitlab.isc.org/isc-projects/bind9/-/issues/4050Add option to not generate CDNSKEY record2023-06-06T10:20:58ZMatthijs Mekkingmatthijs@isc.orgAdd option to not generate CDNSKEY recordWe already have the ability to not generate `CDS` records (by configuring an empty `cds-digest-types {};`), but we do still always generate the `CDNSKEY`.
We could solve this either with:
1. Add new option `cdnskey yes:no`
1. Allow spe...We already have the ability to not generate `CDS` records (by configuring an empty `cds-digest-types {};`), but we do still always generate the `CDNSKEY`.
We could solve this either with:
1. Add new option `cdnskey yes:no`
1. Allow special value `cdnskey` in `cds-digest-types` list.
Despite it being yet another configuration option, I think the first one is preferred, rather than overloading the `cds-digest-types` option.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/4049Detect FORMERR with an echoed DNS COOKIE client cookie and retry without DNS ...2023-07-05T10:39:40ZMark AndrewsDetect FORMERR with an echoed DNS COOKIE client cookie and retry without DNS COOKIEServers that send this sort of response to request with DNS COOKIE are broken as they don't comply with RFC 6891 but they are still at around .5% of servers and are declining.Servers that send this sort of response to request with DNS COOKIE are broken as they don't comply with RFC 6891 but they are still at around .5% of servers and are declining.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)https://gitlab.isc.org/isc-projects/bind9/-/issues/4046[ISC-support #22037] rndc times out in 30 seconds when using BIND 9.18.112023-05-17T13:23:06ZEverett Fulton[ISC-support #22037] rndc times out in 30 seconds when using BIND 9.18.11A Support customer is reporting that 'rndc' is timing out in 30 seconds, where the timeout was previously 60 seconds. Their complex configuration and multiple catalog zones creates a requirement for more time for a reconfig.
https://su...A Support customer is reporting that 'rndc' is timing out in 30 seconds, where the timeout was previously 60 seconds. Their complex configuration and multiple catalog zones creates a requirement for more time for a reconfig.
https://support.isc.org/Ticket/Display.html?id=22037
As mentioned in the BIND/Support meeting on 3 May, it may be possible to backport the 60s timeout from 9.19.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Evan HuntEvan Hunthttps://gitlab.isc.org/isc-projects/bind9/-/issues/4045glue-cache scales very poorly on multi-CPU systems2023-06-14T09:43:01ZPetr Špačekpspacek@isc.orgglue-cache scales very poorly on multi-CPU systems### Summary
Glue cache scales very poorly and suffers from lock contention. It eventually improves max QPS by 1/3 on 16-thread system with delegation-heavy workload. Maxing out QPS on 16-thread system takes over 300 seconds of query loa...### Summary
Glue cache scales very poorly and suffers from lock contention. It eventually improves max QPS by 1/3 on 16-thread system with delegation-heavy workload. Maxing out QPS on 16-thread system takes over 300 seconds of query load.
### BIND version used
* ~"Affects v9.18": v9.18.14
* Other versions were not tested, but are assumed to be affected
### Steps to reproduce
* configure delegation-heavy zone, e.g. SE from https://zonedata.iis.se/
* issue queries which hit delegations, preferably unique: [querydb.xz](/uploads/b3b8bf5b546acb28148133b9f854a738/querydb.xz)
### What is the current *bug* behavior?
Initially QPS is very low, and adding CPUs is not improving performance. Gradually BIND builds glue-cache and overall QPS improves.
### What is the expected *correct* behavior?
* Initial QPS should not be that low.
* Adding CPUs should improve performance, also initially.
### Workaround
```
options {
glue-cache no;
};
```
This provides more predictable performance but incurs ~ 1/3 performance hit (in terms of max QPS).
### Benchmarks
* 16-thread machine in AWS, type c5n.4xlarge
* BIND v9.18.4 with glue-cache on / off
* SE zone serial 2021122008
* client kxdpgun: `kxdpgun -t 5 -Q $QPS -i querydb 10.10.126.46 -p 5300`
* 5-second tests, QPS in the table below is average
* Individual lines in table are successive tests
* Each step starts with the same query set (so successive tests repeat some of the queries)
* QPS step is +50k QPS
* QPS is incremented only if reponse rate was >= 99 %
You can see that `glue-cache yes;` requires significant warm-up time and eventually provides up to 1/3 higher **max** QPS than configuration with `glue-cache no;`. Problem is the ridiculously long warm-up phase.
`glue-cache no` config hits max QPS right away without any warm-up.
Raw data - each line is one 5-second benchmark:
| glue-cache yes | | | glue-cache no | |
|----------------|---------------|--|---------------|---------------|
| QPS | Response rate | | QPS | Response rate |
| 50000 | 77 % | | 300000 | 99 % |
| 50000 | 90 % | | 350000 | 97 % |
| 50000 | 79 % | | 350000 | 96 % |
| 50000 | 99 % | | 350000 | 96 % |
| 100000 | 69 % | | 350000 | 97 % |
| 100000 | 80 % | | 350000 | max reached |
| 100000 | 99 % | | | |
| 150000 | 74 % | | | |
| 150000 | 77 % | | | |
| 150000 | 83 % | | | |
| 150000 | 96 % | | | |
| 150000 | 99 % | | | |
| 200000 | 79 % | | | |
| 200000 | 80 % | | | |
| 200000 | 82 % | | | |
| 200000 | 82 % | | | |
| 200000 | 17 % | | | |
| 200000 | 22 % | | | |
| 200000 | 28 % | | | |
| 200000 | 39 % | | | |
| 200000 | 62 % | | | |
| 200000 | 99 % | | | |
| 250000 | 82 % | | | |
| 250000 | 83 % | | | |
| 250000 | 84 % | | | |
| 250000 | 85 % | | | |
| 250000 | 87 % | | | |
| 250000 | 90 % | | | |
| 250000 | 95 % | | | |
| 250000 | 99 % | | | |
| 300000 | 85 % | | | |
| 300000 | 85 % | | | |
| 300000 | 86 % | | | |
| 300000 | 86 % | | | |
| 300000 | 87 % | | | |
| 300000 | 88 % | | | |
| 300000 | 90 % | | | |
| 300000 | 93 % | | | |
| 300000 | 98 % | | | |
| 300000 | 99 % | | | |
| 350000 | 86 % | | | |
| 350000 | 87 % | | | |
| 350000 | 87 % | | | |
| 350000 | 87 % | | | |
| 350000 | 88 % | | | |
| 350000 | 88 % | | | |
| 350000 | 89 % | | | |
| 350000 | 90 % | | | |
| 350000 | 92 % | | | |
| 350000 | 94 % | | | |
| 350000 | 98 % | | | |
| 350000 | 99 % | | | |
| 400000 | 88 % | | | |
| 400000 | 88 % | | | |
| 400000 | 88 % | | | |
| 400000 | 88 % | | | |
| 400000 | 89 % | | | |
| 400000 | 89 % | | | |
| 400000 | 90 % | | | |
| 400000 | 90 % | | | |
| 400000 | 91 % | | | |
| 400000 | 92 % | | | |
| 400000 | 93 % | | | |
| 400000 | 95 % | | | |
| 400000 | 98 % | | | |
| 400000 | 99 % | | | |
| 450000 | 82 % | | | |
| 450000 | 82 % | | | |
| 450000 | 84 % | | | |
| 450000 | 83 % | | | |
| 450000 | 83 % | | | |
| 450000 | 83 % | | | |
| 450000 | 84 % | | | |
| 450000 | 84 % | | | |
| 450000 | 85 % | | | |
| 450000 | 85 % | | | |
| 450000 | 86 % | | | |
| 450000 | 85 % | | | |
| 450000 | 90 % | | | |
| 450000 | 88 % | | | |
| 450000 | 91 % | | | |
| 450000 | 92 % | | | |
| 450000 | max reached | | | |
Flame chart with sleeper + waker threads generated by [offwaketime.py](https://github.com/iovisor/bcc/blob/d27fd5a7bc8a37679cd3bc0bbdb63f713b0be36f/tools/offwaketime.py):
![offcpu.svg](/uploads/096d197619ed8064a45d330828f78b23/offcpu.svg)
(Sorry for missing stack frames, but you get the point.)June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)https://gitlab.isc.org/isc-projects/bind9/-/issues/4039bind 9.18.14 with error: task.c:805: INSIST((task->events).head != (event)) f...2023-05-10T11:41:24Zcarlos-paniagobind 9.18.14 with error: task.c:805: INSIST((task->events).head != (event)) failed, back trace<pre>
# named -V
BIND 9.18.14 (Extended Support Version) <id:>
running on FreeBSD amd64 12.4-STABLE FreeBSD 12.4-STABLE SOL
built by make with '--disable-linux-caps' '--localstatedir=/var' '--sysconfdir=/usr/local/etc/namedb' '--with-d...<pre>
# named -V
BIND 9.18.14 (Extended Support Version) <id:>
running on FreeBSD amd64 12.4-STABLE FreeBSD 12.4-STABLE SOL
built by make with '--disable-linux-caps' '--localstatedir=/var' '--sysconfdir=/usr/local/etc/namedb' '--with-dlopen=yes' '--with-libxml2' '--with-openssl=/usr' '--enable-dnsrps' '--with-readline=libedit' '--enable-dnstap' '--disable-fixed-rrset' '--disable-geoip' '--without-maxminddb' '--without-gssapi' '--with-libidn2=/usr/local' '--with-json-c' '--disable-largefile' '--with-lmdb=/usr/local' '--disable-querytrace' '--enable-tcp-fastopen' '--prefix=/usr/local' '--mandir=/usr/local/man' '--disable-silent-rules' '--infodir=/usr/local/share/info/' '--build=amd64-portbld-freebsd12.4' 'build_alias=amd64-portbld-freebsd12.4' 'CC=cc' 'CFLAGS=-O2 -pipe -DLIBICONV_PLUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing ' 'LDFLAGS= -L/usr/local/lib -ljson-c -fstack-protector-strong ' 'LIBS=-L/usr/local/lib' 'CPPFLAGS=-DLIBICONV_PLUG -isystem /usr/local/include' 'CPP=cpp' 'PKG_CONFIG=pkgconf' 'PKG_CONFIG_LIBDIR=/usr/ports/dns/bind918/work/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig' 'PYTHON=/usr/local/bin/python3.9' 'READLINE_CFLAGS=-L/usr/local/lib'
compiled by CLANG FreeBSD Clang 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
compiled with OpenSSL version: OpenSSL 1.1.1t-freebsd 7 Feb 2023
linked to OpenSSL version: OpenSSL 1.1.1t-freebsd 7 Feb 2023
compiled with libuv version: 1.44.2
linked to libuv version: 1.44.2
compiled with libnghttp2 version: 1.52.0
linked to libnghttp2 version: 1.52.0
compiled with libxml2 version: 2.10.3
linked to libxml2 version: 21003
compiled with json-c version: 0.16
linked to json-c version: 0.16
compiled with zlib version: 1.2.13
linked to zlib version: 1.2.13
compiled with protobuf-c version: 1.4.1
linked to protobuf-c version: 1.4.1
threads support is enabled
DNSSEC algorithms: RSASHA1 NSEC3RSASHA1 RSASHA256 RSASHA512 ECDSAP256SHA256 ECDSAP384SHA384 ED25519 ED448
DS algorithms: SHA-1 SHA-256 SHA-384
HMAC algorithms: HMAC-MD5 HMAC-SHA1 HMAC-SHA224 HMAC-SHA256 HMAC-SHA384 HMAC-SHA512
TKEY mode 2 support (Diffie-Hellman): yes
TKEY mode 3 support (GSS-API): no
default paths:
named configuration: /usr/local/etc/namedb/named.conf
rndc configuration: /usr/local/etc/namedb/rndc.conf
DNSSEC root key: /usr/local/etc/namedb/bind.keys
nsupdate session key: /var/run/named/session.key
named PID file: /var/run/named/pid
named lock file: /var/run/named/named.lock
-------------------------------------------
Commands
# nslookup
> set q=any
> server ns1.rnp.br
Default server: ns1.rnp.br
Address: 200.133.59.172#53
> 0-63.254.129.200.in-addr.arpa
;; Connection to 200.133.59.172#53(200.133.59.172) for 0-63.254.129.200.in-addr.arpa failed: timed out.
;; Connection to 200.133.59.172#53(200.133.59.172) for 0-63.254.129.200.in-addr.arpa failed: connection refused.
;; Connection to 200.133.59.172#53(200.133.59.172) for 0-63.254.129.200.in-addr.arpa failed: connection refused.
task.c:805: INSIST((task->events).head != (event)) failed, back trace
0x8002be98e <isc_assertion_setcallback+0x5e> at /usr/local/lib/bind-tools/libisc-9.18.14.so
0x8002be92a <isc_assertion_failed+0xa> at /usr/local/lib/bind-tools/libisc-9.18.14.so
0x8002de2de <isc_task_run+0x43e> at /usr/local/lib/bind-tools/libisc-9.18.14.so
0x8002adc92 <isc__nmsocket_log_tls_session_reuse+0x3c2> at /usr/local/lib/bind-tools/libisc-9.18.14.so
0x8002a8223 <isc__nm_maybe_enqueue_ievent+0xa3> at /usr/local/lib/bind-tools/libisc-9.18.14.so
0x8002adb16 <isc__nmsocket_log_tls_session_reuse+0x246> at /usr/local/lib/bind-tools/libisc-9.18.14.so
0x8002a41ed <isc__netmgr_create+0x6dd> at /usr/local/lib/bind-tools/libisc-9.18.14.so
0x800b4fd1a <uv_async_send+0x3da> at /usr/local/lib/libuv.so.1
0x800b610d5 <uv_cpu_info+0xd95> at /usr/local/lib/libuv.so.1
0x800b502c8 <uv_run+0x1e8> at /usr/local/lib/libuv.so.1
0x8002a42db <isc__netmgr_create+0x7cb> at /usr/local/lib/bind-tools/libisc-9.18.14.so
0x8002e6f76 <isc__trampoline_run+0x16> at /usr/local/lib/bind-tools/libisc-9.18.14.so
Abort
</pre>June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Arаm SаrgsyаnArаm Sаrgsyаnhttps://gitlab.isc.org/isc-projects/bind9/-/issues/40379.18.13 crashed after run a few days2023-05-05T08:55:02Zshipei.xu9.18.13 crashed after run a few days
### Summary
After two days of running, my 9.18.13 version of named received a SIGSEGV signal.
### BIND version used
```
BIND 9.18.13 (Extended Support Version) <id:>
[root@ip-X-X-X-X named]# /data/named/sbin/named -V
BIND 9.18.13 (Ext...
### Summary
After two days of running, my 9.18.13 version of named received a SIGSEGV signal.
### BIND version used
```
BIND 9.18.13 (Extended Support Version) <id:>
[root@ip-X-X-X-X named]# /data/named/sbin/named -V
BIND 9.18.13 (Extended Support Version) <id:>
running on Linux x86_64 5.10.130-118.517.amzn2.x86_64 #1 SMP Wed Jul 13 16:51:52 UTC 2022
built by make with '--enable-dnstap' '--enable-epoll' '--with-json-c' '--with-libnghttp2' '--enable-doh' '--prefix=/data/named/' 'PKG_CONFIG_PATH=:/usr/local/lib/pkgconfig'
compiled by GCC 7.3.1 20180712 (Red Hat 7.3.1-15)
compiled with OpenSSL version: OpenSSL 1.1.1g FIPS 21 Apr 2020
linked to OpenSSL version: OpenSSL 1.1.1g FIPS 21 Apr 2020
compiled with libuv version: 1.39.0
linked to libuv version: 1.39.0
compiled with libnghttp2 version: 1.41.0
linked to libnghttp2 version: 1.41.0
compiled with json-c version: 0.11
linked to json-c version: 0.11
compiled with zlib version: 1.2.7
linked to zlib version: 1.2.7
compiled with protobuf-c version: 1.0.2
linked to protobuf-c version: 1.0.2
threads support is enabled
DNSSEC algorithms: RSASHA1 NSEC3RSASHA1 RSASHA256 RSASHA512 ECDSAP256SHA256 ECDSAP384SHA384 ED25519 ED448
DS algorithms: SHA-1 SHA-256 SHA-384
HMAC algorithms: HMAC-MD5 HMAC-SHA1 HMAC-SHA224 HMAC-SHA256 HMAC-SHA384 HMAC-SHA512
TKEY mode 2 support (Diffie-Hellman): yes
TKEY mode 3 support (GSS-API): yes
default paths:
named configuration: /data/named/etc/named.conf
rndc configuration: /data/named/etc/rndc.conf
DNSSEC root key: /data/named/etc/bind.keys
nsupdate session key: /data/named/var/run/named/session.key
named PID file: /data/named/var/run/named/named.pid
named lock file: /data/named/var/run/named/named.lock
```
### Steps to reproduce
This is my build environment, I didn't do anything and it just crashed.
### What is the current *bug* behavior?
Named received a SIGSEGV signal and crashed.
### What is the expected *correct* behavior?
Runs normally.
### Relevant configuration files
```
tls test-tls {
key-file "/ssl_cert/star.key";
cert-file "/ssl_cert/star.pem";
dhparam-file "/ssl_cert/dhparam.pem";
ciphers "HIGH:!kRSA:!aNULL:!eNULL:!RC4:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!SHA1:!SHA256:!SHA384";
prefer-server-ciphers yes;
session-tickets no;
};
http local {
endpoints { "/dns-query"; };
};
options {
listen-on port 53 { any; };
listen-on tls test-tls { any; };
listen-on tls test-tls http local { any; };
listen-on-v6 { none; };
directory "/var/named/";
dump-file "/var/named/data/cache_dump.db";
session-keyfile "/var/named/run/session.key";
bindkeys-file "/etc/bind.keys";
key-directory "/etc";
version none;
notify no;
servfail-ttl 30;
allow-query { any; };
allow-query-cache { any; };
forward first;
hostname none;
reuseport yes;
max-cache-size 6g;
recursion yes;
querylog no;
http-streams-per-connection 100000;
http-listener-clients 100000;
recursive-clients 65535;
clients-per-query 100000;
max-clients-per-query 150000;
tcp-clients 80000;
tcp-initial-timeout 30;
tcp-idle-timeout 50;
tcp-keepalive-timeout 50;
minimal-responses no-auth;
minimal-any yes;
dnstap {
client query;
client response;
resolver query;
resolver response;
};
dnstap-output file "/var/log/dns.tap";
dnstap-identity none;
dnstap-version none;
allow-new-zones yes;
new-zones-directory "/dns-root/";
dnssec-validation no;
};
view "any" {
match-clients { any; };
allow-query-cache { any; };
max-cache-size 256m;
prefetch 10;
max-ncache-ttl 300;
forwarders { *.*.*.* port 5533; };
dlz "file system zone" {
database "dlopen /lib/dlz_filesystem_dynamic.so /dns-root/ .dns .xfr 0 ~";
};
};
```
### Relevant logs and/or screenshots
```
[root@ip-172-16-7-111 named]# gdb /data/named/sbin/named ./core.22530
GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-36.amzn2.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /data/named/sbin/named...done.
[New LWP 22532]
[New LWP 22530]
[New LWP 22531]
[New LWP 22533]
[New LWP 22534]
[New LWP 22535]
[New LWP 22541]
warning: Could not load shared library symbols for /lib/dlz_filesystem_dynamic.so.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/data/named/sbin/named -u named -c /etc/named.conf -t /data/named/chroot/'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f3c3f30f062 in SSL_SESSION_list_remove.isra.0 () from /lib64/libssl.so.1.1
[Current thread is 1 (Thread 0x7f3c3a5ff700 (LWP 22532))]
Missing separate debuginfos, use: debuginfo-install glibc-2.26-60.amzn2.x86_64 json-c-0.11-4.amzn2.0.4.x86_64 keyutils-libs-1.5.8-3.amzn2.0.2.x86_64 krb5-libs-1.15.1-37.amzn2.2.4.x86_64 libcap-2.54-1.amzn2.0.1.x86_64 libcom_err-1.42.9-19.amzn2.x86_64 libgcc-7.3.1-15.amzn2.x86_64 libnghttp2-1.41.0-1.amzn2.x86_64 libselinux-2.5-12.amzn2.0.2.x86_64 libstdc++-7.3.1-15.amzn2.x86_64 libuv-1.39.0-1.amzn2.x86_64 openssl11-libs-1.1.1g-12.amzn2.0.9.x86_64 pcre-8.32-17.amzn2.0.2.x86_64 protobuf-c-1.0.2-3.amzn2.0.1.x86_64 sssd-client-1.16.5-10.amzn2.10.x86_64 zlib-1.2.7-19.amzn2.0.1.x86_64
(gdb) bt
#0 0x00007f3c3f30f062 in SSL_SESSION_list_remove.isra.0 () from /lib64/libssl.so.1.1
#1 0x00007f3c3f30fc4b in timeout_cb () from /lib64/libssl.so.1.1
#2 0x00007f3c3ef78165 in OPENSSL_LH_doall_arg () from /lib64/libcrypto.so.1.1
#3 0x00007f3c3f310977 in SSL_CTX_flush_sessions () from /lib64/libssl.so.1.1
#4 0x00007f3c3f3297dc in tls_construct_new_session_ticket () from /lib64/libssl.so.1.1
#5 0x00007f3c3f31bbba in state_machine () from /lib64/libssl.so.1.1
#6 0x00007f3c3f309000 in SSL_do_handshake () from /lib64/libssl.so.1.1
#7 0x00007f3c410570a3 in tls_try_handshake (sock=sock@entry=0x7f3c2463fe00, presult=presult@entry=0x7f3c3a5ea070) at netmgr/tlsstream.c:340
#8 0x00007f3c41057d50 in tls_do_bio (sock=0x7f3c2463fe00, received_data=0x7f3c3a5fa0c0, send_data=0x0, finish=false) at netmgr/tlsstream.c:457
#9 0x00007f3c410174bc in isc__nm_async_readcb (worker=worker@entry=0x0, ev0=ev0@entry=0x7f3c3a5fa0f0) at netmgr/netmgr.c:2890
#10 0x00007f3c41017609 in isc__nm_readcb (sock=sock@entry=0x7f3c2478dc00, uvreq=<optimized out>, eresult=eresult@entry=ISC_R_SUCCESS) at netmgr/netmgr.c:2863
#11 0x00007f3c4101c22a in isc__nm_tcp_read_cb (stream=<optimized out>, nread=80, buf=0x7f3c3a5fa1c0) at netmgr/tcp.c:904
#12 0x00007f3c3e5c8aff in uv.read () from /lib64/libuv.so.1
#13 0x00007f3c3e5c9736 in uv.stream_io () from /lib64/libuv.so.1
#14 0x00007f3c3e5ced96 in uv.io_poll () from /lib64/libuv.so.1
#15 0x00007f3c3e5bf1a3 in uv_run () from /lib64/libuv.so.1
#16 0x00007f3c41019563 in nm_thread (worker0=0x7f3c3c0c15b8) at netmgr/netmgr.c:698
#17 0x00007f3c4104e765 in isc__trampoline_run (arg=0x7f3c3c032330) at trampoline.c:189
#18 0x00007f3c3db4c44b in start_thread () from /lib64/libpthread.so.0
#19 0x00007f3c3d88756f in clone () from /lib64/libc.so.6
(gdb)
```
### Possible fixesJune 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)https://gitlab.isc.org/isc-projects/bind9/-/issues/4035fuzz: mem.c:871: fatal error: RUNTIME_CHECK(((pthread_mutex_lock(((&contextsl...2023-05-18T04:15:20ZMichal Nowakfuzz: mem.c:871: fatal error: RUNTIME_CHECK(((pthread_mutex_lock(((&contextslock))) == 0) ? 0 : 34) == 0) failed`dns_rdata_fromwire_text` and `dns_name_fromtext_target` fail on `bind-9.16` on Fedora 38 dev system.
<details><summary>mem.c:871: fatal error: RUNTIME_CHECK(((pthread_mutex_lock(((&contextslock))) == 0) ? 0 : 34) == 0) failed</summary>...`dns_rdata_fromwire_text` and `dns_name_fromtext_target` fail on `bind-9.16` on Fedora 38 dev system.
<details><summary>mem.c:871: fatal error: RUNTIME_CHECK(((pthread_mutex_lock(((&contextslock))) == 0) ? 0 : 34) == 0) failed</summary>
```
testing 12 bytes from /home/newman/isc/ws/bind9/fuzz/dns_name_fromtext_target.in/example.com
mem.c:871: fatal error: RUNTIME_CHECK(((pthread_mutex_lock(((&contextslock))) == 0) ? 0 : 34) == 0) failed
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-140
testing 48 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-93
testing 8 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-24
testing 9 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-7
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-71
testing 22 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-39
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-69
testing 50 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-137
testing 71 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-110
testing 115 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-118
testing 11 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-33
testing 57 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-114
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-40
testing 5 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-2
testing 5 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-6
testing 18 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-50
testing 10 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-36
testing 57 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-115
testing 36 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-88
testing 22 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-19
testing 63 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-89
testing 44 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-103
testing 23 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-18
testing 52 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-9
testing 48 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-127
testing 15 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-83
testing 50 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-126
testing 26 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-102
testing 50 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-133
testing 9 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-99
testing 42 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-91
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-106
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-68
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-122
testing 38 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-58
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-142
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-75
testing 15 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-0
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-76
testing 12 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-51
testing 83 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-54
testing 12 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-35
testing 12 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-23
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-117
testing 38 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-17
testing 51 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-139
testing 72 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-3
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-87
testing 74 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-90
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-72
testing 66 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-37
testing 50 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-134
testing 15 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-25
testing 74 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-92
testing 68 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-109
testing 39 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/svcb
testing 7 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-1
testing 110 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-121
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-59
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-66
testing 12 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-82
testing 31 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-107
testing 50 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-136
testing 23 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-63
testing 12 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-78
testing 25 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-123
testing 33 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-30
testing 50 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-132
testing 9 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-32
testing 5 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-16
testing 65 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-124
testing 75 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-21
testing 38 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-45
testing 77 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-10
testing 5 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-5
testing 50 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-60
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-65
testing 50 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-98
testing 60 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-130
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-44
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-104
testing 30 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-13
testing 8 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-12
testing 54 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-48
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-41
testing 51 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-131
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-116
testing 30 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-43
testing 8 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-77
testing 9 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-84
testing 56 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-128
testing 11 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-81
testing 25 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-101
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-49
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-108
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-95
testing 93 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-113
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-11
testing 126 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-120
testing 7 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-34
testing 50 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-135
testing 37 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-31
testing 6 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/smimea
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-86
testing 9 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-79
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-47
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-125
testing 5 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-4
testing 7 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-27
testing 19 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-22
testing 84 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-111
testing 7 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-53
testing 7 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-15
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-67
testing 6 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/sshfp
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-55
testing 151 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-20
testing 11 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-80
testing 66 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-38
testing 22 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-105
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-141
testing 6 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-52
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-74
testing 38 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-14
testing 29 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-61
testing 69 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-112
testing 11 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-96
testing 12 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-100
testing 49 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-57
testing 28 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-62
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-64
testing 7 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-143
testing 50 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-138
testing 8 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-28
testing 27 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-70
testing 20 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/cdnskey
testing 17 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-97
testing 67 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-8
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-26
testing 26 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-129
testing 23 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-94
testing 23 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-46
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-56
testing 57 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-119
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-42
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-85
testing 8 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-29
testing 21 bytes from /home/newman/isc/ws/bind9/fuzz/dns_rdata_fromwire_text.in/input-73
mem.c:871: fatal error: RUNTIME_CHECK(((pthread_mutex_lock(((&contextslock))) == 0) ? 0 : 34) == 0) failed
make: *** [Makefile:475: check] Error 134
```
</details>
```
Core was generated by `./dns_name_fromtext_target'.
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1 0x00007f98efd3bc03 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2 0x00007f98efceaaee in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3 0x00007f98efcd387f in __GI_abort () at abort.c:79
#4 0x0000000000464cb8 in isc_error_fatal (file=file@entry=0x48919d "mem.c", line=line@entry=871, format=format@entry=0x487bdc "RUNTIME_CHECK(%s) failed") at error.c:70
#5 0x0000000000464cd5 in isc_error_runtimecheck (file=file@entry=0x48919d "mem.c", line=line@entry=871, expression=expression@entry=0x489950 "((pthread_mutex_lock(((&contextslock))) == 0) ? 0 : 34) == 0")
at error.c:75
#6 0x000000000046c6c8 in destroy (ctx=0x9002c0) at mem.c:871
#7 0x000000000046ce97 in isc_mem_destroy (ctxp=0x4a4570 <mctx>) at mem.c:1043
#8 0x00007f98f06d30f2 in _dl_call_fini (closure_map=closure_map@entry=0x7f98f07062c0) at dl-call_fini.c:43
#9 0x00007f98f06d6fbe in _dl_fini () at dl-fini.c:114
#10 0x00007f98efced176 in __run_exit_handlers (status=0, listp=0x7f98efe82840 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#11 0x00007f98efced2be in __GI_exit (status=<optimized out>) at exit.c:138
#12 0x00007f98efcd4b51 in __libc_start_call_main (main=main@entry=0x40e090 <main>, argc=argc@entry=1, argv=argv@entry=0x7ffdc50702d8) at ../sysdeps/nptl/libc_start_call_main.h:74
#13 0x00007f98efcd4c0b in __libc_start_main_impl (main=0x40e090 <main>, argc=1, argv=0x7ffdc50702d8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffdc50702c8)
at ../csu/libc-start.c:360
#14 0x000000000040e3a5 in _start ()
```June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)https://gitlab.isc.org/isc-projects/bind9/-/issues/4034ThreadSanitizer: heap-use-after-free lib/dns/resolver.c:1967 in process_sende...2023-06-21T08:15:57ZMichal NowakThreadSanitizer: heap-use-after-free lib/dns/resolver.c:1967 in process_sendeventJob [#3348568](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3348568) failed for 9b57b81ed053404e95a674023e94f74c6755fb2f.
```
WARNING: ThreadSanitizer: heap-use-after-free
Read of size 4 at 0x000000000001 by thread T1:
#0 pro...Job [#3348568](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3348568) failed for 9b57b81ed053404e95a674023e94f74c6755fb2f.
```
WARNING: ThreadSanitizer: heap-use-after-free
Read of size 4 at 0x000000000001 by thread T1:
#0 process_sendevent lib/dns/resolver.c:1967
#1 resquery_senddone lib/dns/resolver.c:2029
#2 task_run lib/isc/task.c:859
#3 isc_task_run lib/isc/task.c:953
#4 isc__nm_async_task lib/isc/netmgr/netmgr.c:871
#5 process_netievent lib/isc/netmgr/netmgr.c:943
#6 process_queue lib/isc/netmgr/netmgr.c:1009
#7 process_all_queues lib/isc/netmgr/netmgr.c:790
#8 async_cb lib/isc/netmgr/netmgr.c:819
#9 uv__async_io /usr/src/libuv-v1.44.1/src/unix/async.c:163
#10 isc__trampoline_run lib/isc/trampoline.c:213
Previous write of size 8 at 0x000000000001 by thread T1 (mutexes: write M1):
#0 free ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:707
#1 default_memfree lib/isc/mem.c:744
#2 mem_put lib/isc/mem.c:656
#3 isc___mem_put lib/isc/mem.c:1132
#4 isc__mem_put lib/isc/mem.c:2391
#5 resquery_destroy lib/dns/resolver.c:1255
#6 fctx_cancelquery lib/dns/resolver.c:1516
#7 process_sendevent lib/dns/resolver.c:1962
#8 resquery_senddone lib/dns/resolver.c:2029
#9 task_run lib/isc/task.c:859
#10 isc_task_run lib/isc/task.c:953
#11 isc__nm_async_task lib/isc/netmgr/netmgr.c:871
#12 process_netievent lib/isc/netmgr/netmgr.c:943
#13 process_queue lib/isc/netmgr/netmgr.c:1009
#14 process_all_queues lib/isc/netmgr/netmgr.c:790
#15 async_cb lib/isc/netmgr/netmgr.c:819
#16 uv__async_io /usr/src/libuv-v1.44.1/src/unix/async.c:163
#17 isc__trampoline_run lib/isc/trampoline.c:213
Mutex M1 is already destroyed.
Thread T1 (running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962
#1 isc_thread_create lib/isc/pthreads/thread.c:81
#2 isc__netmgr_create lib/isc/netmgr/netmgr.c:355
#3 isc_managers_create lib/isc/managers.c:28
#4 create_managers main.c:1065
#5 setup main.c:1397
#6 main main.c:1711
SUMMARY: ThreadSanitizer: heap-use-after-free lib/dns/resolver.c:1967 in process_sendevent
```
Could it be isc-projects/bind9!7868? (The only thing merged since yesterday's scheduled run on ~"v9.16".)June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Michal NowakMichal Nowakhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4023LeakSanitizer detected memory leak in nsupdate2023-05-11T14:05:33ZMichal NowakLeakSanitizer detected memory leak in nsupdateJob [#3336065](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3336065) failed in `system:clang:asan` which has Clang 16.
```
$NSUPDATE -D -S -A CA/CA-other.pem -K CA/certs/srv01.client01.example.nil.key -E CA/certs/client01.crt01.examp...Job [#3336065](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3336065) failed in `system:clang:asan` which has Clang 16.
```
$NSUPDATE -D -S -A CA/CA-other.pem -K CA/certs/srv01.client01.example.nil.key -E CA/certs/client01.crt01.example.nil.pem -k ns1/ddns.key <<END > nsupdate.out.test$n 2>&1
server 10.53.0.1 ${EXTRAPORT2}
update add dot-fsmt-auth-bad.example.nil. 600 A 10.10.10.3
send
END
```
```
setup_system()
Creating key...
Creating key...
namefromtext
keycreate
reset_system()
user_interaction()
do_next_command()
do_next_command()
evaluate_update()
update_addordelete()
do_next_command()
start_update()
dns_request_create: TLS error
=================================================================
==26697==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 16 byte(s) in 1 object(s) allocated from:
#0 0x55ad0ab4e3de in malloc (/builds/isc-projects/bind9/bin/nsupdate/.libs/nsupdate+0xc93de) (BuildId: cb6f56478f03cc1cdba7bb8d05bbc4bef756b573)
#1 0x7f43ee89a115 in mallocx /builds/isc-projects/bind9/lib/isc/./jemalloc_shim.h:65:14
#2 0x7f43ee89a115 in mem_get /builds/isc-projects/bind9/lib/isc/mem.c:304:8
#3 0x7f43ee899e31 in isc__mem_get /builds/isc-projects/bind9/lib/isc/mem.c:667:8
#4 0x55ad0ab9e371 in sendrequest /builds/isc-projects/bind9/bin/nsupdate/nsupdate.c:2947:12
#5 0x55ad0ab8f770 in start_update /builds/isc-projects/bind9/bin/nsupdate/nsupdate.c:3405:2
#6 0x55ad0ab8f770 in getinput /builds/isc-projects/bind9/bin/nsupdate/nsupdate.c:3499:2
#7 0x7f43ee891721 in setup_jobs_cb /builds/isc-projects/bind9/lib/isc/loop.c:255:3
#8 0x7f43ee80a13f in isc__async_cb /builds/isc-projects/bind9/lib/isc/async.c:84:3
#9 0x7f43ecd08e92 in uv__async_io /usr/src/libuv-v1.44.1/src/unix/async.c:163:5
#10 0x7f43ecd24bc2 in uv__io_poll /usr/src/libuv-v1.44.1/src/unix/epoll.c:374:11
#11 0x7f43ecd098ad in uv_run /usr/src/libuv-v1.44.1/src/unix/core.c:391:5
#12 0x7f43ee88bc6a in loop_run /builds/isc-projects/bind9/lib/isc/loop.c:272:6
#13 0x7f43ee88bc6a in loop_thread /builds/isc-projects/bind9/lib/isc/loop.c:299:2
#14 0x7f43ee88b8a7 in isc_loopmgr_run /builds/isc-projects/bind9/lib/isc/loop.c:473:2
#15 0x55ad0ab8afcb in main /builds/isc-projects/bind9/bin/nsupdate/nsupdate.c:3536:2
#16 0x7f43ecd74d09 in __libc_start_main csu/../csu/libc-start.c:308:16
SUMMARY: AddressSanitizer: 16 byte(s) leaked in 1 allocation(s).
```June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Michal NowakMichal Nowakhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4022Three "dnssec" failures on Fedora 382023-05-18T13:54:06ZMichal NowakThree "dnssec" failures on Fedora 38The `dnssec` system test [fails](https://gitlab.isc.org/isc-projects/bind9/-/pipelines/134795) in three tests on Fedora 38. Supposedly, there were [no crypto changes in Fedora 38](https://fedoraproject.org/wiki/Releases/38/ChangeSet).
#...The `dnssec` system test [fails](https://gitlab.isc.org/isc-projects/bind9/-/pipelines/134795) in three tests on Fedora 38. Supposedly, there were [no crypto changes in Fedora 38](https://fedoraproject.org/wiki/Releases/38/ChangeSet).
# 1
```
I:dnssec: check that 'dnssec-signzone -F' failed with disallowed algorithm (110)
I:dnssec:failed
```
The test expects `fatal: dnskey 'example.com/RSASHA1/19857' failed to sign data` but gets `dnssec-signzone: fatal: No signing keys specified or found.` although `dnssec/signer/general/Kexample.com.+005+19857.{key,private}` files are present.
# 2
```
I:dnssec:check that 'dnssec-keygen -F' disables rsasha1 (232)
I:dnssec:failed
```
The test expects `unsupported algorithm: RSASHA1` but gets `dnssec-keygen: fatal: unsupported algorithm: rsasha1`.
# 3
```
I:dnssec:check that 'dnssec-keygen -F' disables nsec3rsasha1 (233)
I:dnssec:failed
```
The test expects `unsupported algorithm: NSEC3RSASHA1` but gets `dnssec-keygen: fatal: unsupported algorithm: nsec3rsasha1`.
For issues **2** and **3**, matching case-insentively with `-i` is enough.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)https://gitlab.isc.org/isc-projects/bind9/-/issues/4019CID 452208: locking issue in tests/bench/qpmulti.c2023-05-12T20:43:31ZMichal NowakCID 452208: locking issue in tests/bench/qpmulti.cCoverity Scan claims a locking issue in a test code rooted in !7582.
```
/tests/bench/qpmulti.c: 296 in read_transactions()
290 if (result == ISC_R_SUCCESS) {
291 args->present++;
292 } else {
293 args->abs...Coverity Scan claims a locking issue in a test code rooted in !7582.
```
/tests/bench/qpmulti.c: 296 in read_transactions()
290 if (result == ISC_R_SUCCESS) {
291 args->present++;
292 } else {
293 args->absent++;
294 }
295 }
>>> CID 452208: API usage errors (LOCK)
>>> "dns_qpread_destroy" unlocks "args->multi->mutex" while it is unlocked.
296 dns_qpread_destroy(args->multi, &qp);
297 }
298 next_loop(args, start);
299 }
300
301 static void
```June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Tony FinchTony Finch