BIND issueshttps://gitlab.isc.org/isc-projects/bind9/-/issues2023-06-14T10:19:31Zhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4053CID 453470: Use after free in lib/ns/client.c2023-06-14T10:19:31ZMichal NowakCID 453470: Use after free in lib/ns/client.cCoverity Scan claims use after free in `lib/ns/client.c`.
```c
1. Switch case value ns_cookiealg_aes.
1146 switch (client->manager->sctx->cookiealg) {
1147 case ns_cookiealg_siphash24: {
1148 unsigned ch...Coverity Scan claims use after free in `lib/ns/client.c`.
```c
1. Switch case value ns_cookiealg_aes.
1146 switch (client->manager->sctx->cookiealg) {
1147 case ns_cookiealg_siphash24: {
1148 unsigned char input[16 + 16] ISC_NONSTRING = { 0 };
1149 size_t inputlen = 0;
1150 isc_netaddr_t netaddr;
1151 unsigned char *cp;
1152
1153 cp = isc_buffer_used(buf);
1154 isc_buffer_putmem(buf, client->cookie, 8);
1155 isc_buffer_putuint8(buf, NS_COOKIE_VERSION_1);
1156 isc_buffer_putuint8(buf, 0); /* Reserved */
1157 isc_buffer_putuint16(buf, 0); /* Reserved */
1158 isc_buffer_putuint32(buf, when);
1159
CID 453470 (2) (#1-3 of 4): Use after free (USE_AFTER_FREE) [select issue]
1160 memmove(input, cp, 16);
1161
1162 isc_netaddr_fromsockaddr(&netaddr, &client->peeraddr);
1163 switch (netaddr.family) {
1164 case AF_INET:
1165 cp = (unsigned char *)&netaddr.type.in;
1166 memmove(input + 16, cp, 4);
1167 inputlen = 20;
1168 break;
1169 case AF_INET6:
1170 cp = (unsigned char *)&netaddr.type.in6;
1171 memmove(input + 16, cp, 16);
1172 inputlen = 32;
1173 break;
1174 default:
1175 UNREACHABLE();
1176 }
1177
1178 isc_siphash24(secret, input, inputlen, true, digest);
1179 isc_buffer_putmem(buf, digest, 8);
1180 break;
1181 }
1182 case ns_cookiealg_aes: {
1183 unsigned char input[4 + 4 + 16] ISC_NONSTRING = { 0 };
1184 isc_netaddr_t netaddr;
1185 unsigned char *cp;
1186 unsigned int i;
1187
2. assign: Assigning: cp = (void *)((unsigned char *)buf->base + buf->used).
1188 cp = isc_buffer_used(buf);
1189 isc_buffer_putmem(buf, client->cookie, 8);
1190 isc_buffer_putuint32(buf, nonce);
3. freed_arg: isc_buffer_putuint32 frees buf->base. [show details]
1191 isc_buffer_putuint32(buf, when);
CID 453470 (#2-4 of 4): Use after free (USE_AFTER_FREE)
deref_arg: Calling memmove dereferences freed pointer cp. [Note: The source code implementation of the function has been overridden by a builtin model.]
1192 memmove(input, cp, 16);
```
Note that it might not be a new issue, but something the new Coverity Scan 2022.12 detected.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)https://gitlab.isc.org/isc-projects/bind9/-/issues/4050Add option to not generate CDNSKEY record2023-06-06T10:20:58ZMatthijs Mekkingmatthijs@isc.orgAdd option to not generate CDNSKEY recordWe already have the ability to not generate `CDS` records (by configuring an empty `cds-digest-types {};`), but we do still always generate the `CDNSKEY`.
We could solve this either with:
1. Add new option `cdnskey yes:no`
1. Allow spe...We already have the ability to not generate `CDS` records (by configuring an empty `cds-digest-types {};`), but we do still always generate the `CDNSKEY`.
We could solve this either with:
1. Add new option `cdnskey yes:no`
1. Allow special value `cdnskey` in `cds-digest-types` list.
Despite it being yet another configuration option, I think the first one is preferred, rather than overloading the `cds-digest-types` option.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/4045glue-cache scales very poorly on multi-CPU systems2023-06-14T09:43:01ZPetr Špačekpspacek@isc.orgglue-cache scales very poorly on multi-CPU systems### Summary
Glue cache scales very poorly and suffers from lock contention. It eventually improves max QPS by 1/3 on 16-thread system with delegation-heavy workload. Maxing out QPS on 16-thread system takes over 300 seconds of query loa...### Summary
Glue cache scales very poorly and suffers from lock contention. It eventually improves max QPS by 1/3 on 16-thread system with delegation-heavy workload. Maxing out QPS on 16-thread system takes over 300 seconds of query load.
### BIND version used
* ~"Affects v9.18": v9.18.14
* Other versions were not tested, but are assumed to be affected
### Steps to reproduce
* configure delegation-heavy zone, e.g. SE from https://zonedata.iis.se/
* issue queries which hit delegations, preferably unique: [querydb.xz](/uploads/b3b8bf5b546acb28148133b9f854a738/querydb.xz)
### What is the current *bug* behavior?
Initially QPS is very low, and adding CPUs is not improving performance. Gradually BIND builds glue-cache and overall QPS improves.
### What is the expected *correct* behavior?
* Initial QPS should not be that low.
* Adding CPUs should improve performance, also initially.
### Workaround
```
options {
glue-cache no;
};
```
This provides more predictable performance but incurs ~ 1/3 performance hit (in terms of max QPS).
### Benchmarks
* 16-thread machine in AWS, type c5n.4xlarge
* BIND v9.18.4 with glue-cache on / off
* SE zone serial 2021122008
* client kxdpgun: `kxdpgun -t 5 -Q $QPS -i querydb 10.10.126.46 -p 5300`
* 5-second tests, QPS in the table below is average
* Individual lines in table are successive tests
* Each step starts with the same query set (so successive tests repeat some of the queries)
* QPS step is +50k QPS
* QPS is incremented only if reponse rate was >= 99 %
You can see that `glue-cache yes;` requires significant warm-up time and eventually provides up to 1/3 higher **max** QPS than configuration with `glue-cache no;`. Problem is the ridiculously long warm-up phase.
`glue-cache no` config hits max QPS right away without any warm-up.
Raw data - each line is one 5-second benchmark:
| glue-cache yes | | | glue-cache no | |
|----------------|---------------|--|---------------|---------------|
| QPS | Response rate | | QPS | Response rate |
| 50000 | 77 % | | 300000 | 99 % |
| 50000 | 90 % | | 350000 | 97 % |
| 50000 | 79 % | | 350000 | 96 % |
| 50000 | 99 % | | 350000 | 96 % |
| 100000 | 69 % | | 350000 | 97 % |
| 100000 | 80 % | | 350000 | max reached |
| 100000 | 99 % | | | |
| 150000 | 74 % | | | |
| 150000 | 77 % | | | |
| 150000 | 83 % | | | |
| 150000 | 96 % | | | |
| 150000 | 99 % | | | |
| 200000 | 79 % | | | |
| 200000 | 80 % | | | |
| 200000 | 82 % | | | |
| 200000 | 82 % | | | |
| 200000 | 17 % | | | |
| 200000 | 22 % | | | |
| 200000 | 28 % | | | |
| 200000 | 39 % | | | |
| 200000 | 62 % | | | |
| 200000 | 99 % | | | |
| 250000 | 82 % | | | |
| 250000 | 83 % | | | |
| 250000 | 84 % | | | |
| 250000 | 85 % | | | |
| 250000 | 87 % | | | |
| 250000 | 90 % | | | |
| 250000 | 95 % | | | |
| 250000 | 99 % | | | |
| 300000 | 85 % | | | |
| 300000 | 85 % | | | |
| 300000 | 86 % | | | |
| 300000 | 86 % | | | |
| 300000 | 87 % | | | |
| 300000 | 88 % | | | |
| 300000 | 90 % | | | |
| 300000 | 93 % | | | |
| 300000 | 98 % | | | |
| 300000 | 99 % | | | |
| 350000 | 86 % | | | |
| 350000 | 87 % | | | |
| 350000 | 87 % | | | |
| 350000 | 87 % | | | |
| 350000 | 88 % | | | |
| 350000 | 88 % | | | |
| 350000 | 89 % | | | |
| 350000 | 90 % | | | |
| 350000 | 92 % | | | |
| 350000 | 94 % | | | |
| 350000 | 98 % | | | |
| 350000 | 99 % | | | |
| 400000 | 88 % | | | |
| 400000 | 88 % | | | |
| 400000 | 88 % | | | |
| 400000 | 88 % | | | |
| 400000 | 89 % | | | |
| 400000 | 89 % | | | |
| 400000 | 90 % | | | |
| 400000 | 90 % | | | |
| 400000 | 91 % | | | |
| 400000 | 92 % | | | |
| 400000 | 93 % | | | |
| 400000 | 95 % | | | |
| 400000 | 98 % | | | |
| 400000 | 99 % | | | |
| 450000 | 82 % | | | |
| 450000 | 82 % | | | |
| 450000 | 84 % | | | |
| 450000 | 83 % | | | |
| 450000 | 83 % | | | |
| 450000 | 83 % | | | |
| 450000 | 84 % | | | |
| 450000 | 84 % | | | |
| 450000 | 85 % | | | |
| 450000 | 85 % | | | |
| 450000 | 86 % | | | |
| 450000 | 85 % | | | |
| 450000 | 90 % | | | |
| 450000 | 88 % | | | |
| 450000 | 91 % | | | |
| 450000 | 92 % | | | |
| 450000 | max reached | | | |
Flame chart with sleeper + waker threads generated by [offwaketime.py](https://github.com/iovisor/bcc/blob/d27fd5a7bc8a37679cd3bc0bbdb63f713b0be36f/tools/offwaketime.py):
![offcpu.svg](/uploads/096d197619ed8064a45d330828f78b23/offcpu.svg)
(Sorry for missing stack frames, but you get the point.)June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)https://gitlab.isc.org/isc-projects/bind9/-/issues/2710Allow for arbitrary DNSKEY/CDS/CDNSKEY records to be published2023-07-20T13:48:38ZMatthijs Mekkingmatthijs@isc.orgAllow for arbitrary DNSKEY/CDS/CDNSKEY records to be publishedTo support the multi-signer model (2), we want to allow arbitrary CDS/CDNSKEY records to be published in the zone. Currently this is not possible, because `zone_cdscheck` will error if there are CDS/CDNSKEY records in the zone that do no...To support the multi-signer model (2), we want to allow arbitrary CDS/CDNSKEY records to be published in the zone. Currently this is not possible, because `zone_cdscheck` will error if there are CDS/CDNSKEY records in the zone that do not have a matching DNSKEY record.
The multi-signer model (2) ensures for a safe transition from one provider to another provider without going insecure. In this model, both providers have their own KSK. To rollover to the other provider, the DS records of both KSKs need to be published at some point, and if the double DS RRset is known to the world, the old DS record can be removed and the transition to the new provider is complete.
If the parent supports DNSSEC Child-Parent synchronization, it may query for the child zone servers for CDS/CDNSKEY records in order to update their DS RRset. In the case of a provider transition, both providers should publish the CDS/CDNSKEY RRset that contain two entries, one corresponding to the KSK of one provider, one corresponding to the KSK of the other provider.
It should be possible that such CDS/CDNSKEY record is added to the zone file, or it may be added with a Dynamic Update.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/4116Building with --with-liburcu=qsbr fails2023-06-05T23:46:49ZMichal NowakBuilding with --with-liburcu=qsbr failsJob [#3440349](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3440349) failed for c26d66604bd5bebb93b365b4c2f1fb3fc94bcf92.
Building with `--with-liburcu=qsbr` fails with:
```
In file included from view.c:33:
view.c: In function 'dns...Job [#3440349](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3440349) failed for c26d66604bd5bebb93b365b4c2f1fb3fc94bcf92.
Building with `--with-liburcu=qsbr` fails with:
```
In file included from view.c:33:
view.c: In function 'dns_view_detach':
../../lib/isc/include/isc/urcu.h:106:27: error: implicit declaration of function 'isc_qsbr_syncronize_rcu'; did you mean 'isc_qsbr_synchronize_rcu'? [-Werror=implicit-function-declaration]
106 | #define synchronize_rcu() isc_qsbr_syncronize_rcu()
| ^~~~~~~~~~~~~~~~~~~~~~~
view.c:542:25: note: in expansion of macro 'synchronize_rcu'
542 | synchronize_rcu();
| ^~~~~~~~~~~~~~~
```
It's a permanent failure in the `pairwise` CI job.
isc-projects/bind9!7990 seems like the culprit.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Mark AndrewsMark Andrewshttps://gitlab.isc.org/isc-projects/bind9/-/issues/4115Assert on validator.c:3205: INSIST(__v > 0) during resolver benchmark2023-06-06T17:34:23ZPetr Špačekpspacek@isc.orgAssert on validator.c:3205: INSIST(__v > 0) during resolver benchmark### Summary
First sketch. I will supply more details next week, hopefully.
Crash in DNSSEC validator. Exact cause unknown.
### BIND version used
* ~"Affects v9.19": v9.19.13
Started with 6ba2579c67ac0bda59739c0f4555d405967f608c. Work...### Summary
First sketch. I will supply more details next week, hopefully.
Crash in DNSSEC validator. Exact cause unknown.
### BIND version used
* ~"Affects v9.19": v9.19.13
Started with 6ba2579c67ac0bda59739c0f4555d405967f608c. Worked on commit da0f154bc750df345b8c5ea07d1a3c22e59002cd.
```
# /var/opt/bind9/bin/named/.libs/named -V
BIND 9.19.13 (Development Release) <id:66a3c6b>
running on Linux x86_64 6.2.12-1-ec2 #1 SMP PREEMPT_DYNAMIC Mon, 24 Apr 2023 20:44:00 +0000
built by make with '--without-python' '--disable-linux-caps' '--prefix=/usr' 'CC=' 'CFLAGS=' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS='
compiled by GCC 11.3.0
compiled with OpenSSL version: OpenSSL 3.0.2 15 Mar 2022
linked to OpenSSL version: OpenSSL 3.0.2 15 Mar 2022
compiled with libuv version: 1.43.0
linked to libuv version: 1.43.0
compiled with liburcu version: 0.13.1
compiled with libnghttp2 version: 1.43.0
linked to libnghttp2 version: 1.43.0
compiled with json-c version: 0.15
linked to json-c version: 0.15
compiled with zlib version: 1.2.11
linked to zlib version: 1.2.11
threads support is enabled
DNSSEC algorithms: RSASHA1 NSEC3RSASHA1 RSASHA256 RSASHA512 ECDSAP256SHA256 ECDSAP384SHA384 ED25519 ED448
DS algorithms: SHA-1 SHA-256 SHA-384
HMAC algorithms: HMAC-MD5 HMAC-SHA1 HMAC-SHA224 HMAC-SHA256 HMAC-SHA384 HMAC-SHA512
TKEY mode 2 support (Diffie-Hellman): no
TKEY mode 3 support (GSS-API): no
default paths:
named configuration: /usr/etc/named.conf
rndc configuration: /usr/etc/rndc.conf
nsupdate session key: /usr/var/run/named/session.key
named PID file: /usr/var/run/named/named.pid
named lock file: /usr/var/run/named/named.lock
```
Did not reproduce on ~"v9.18" or ~"v9.16" :relieved:
### Steps to reproduce
* Run https://gitlab.isc.org/isc-projects/bind9-shotgun-ci/-/pipelines/new for 5 minutes with load factor 10x. Dataset telco-eu-2023-03.
### What is the current *bug* behavior?
Crashes in the last minute.
### What is the expected *correct* behavior?
Well, no crash.
### Relevant logs and/or screenshots
Asserts on
```
validator.c:3205: INSIST(__v > 0)
```
Backtrace - best I was able to get so far:
```
(No debugging symbols found in /var/opt/bind9/bin/named/.libs/named)
...
Core was generated by `/var/opt/bind9/bin/named/.libs/named -c /etc/bind9/named.conf -g -n 16 -S 10000'.
...
(gdb) bt
#0 __GI_abort () at ./stdlib/abort.c:107
#1 0x0000560012ed77a4 in assertion_failed ()
#2 0x00007fd80e8e40c2 in isc_assertion_failed () from /var/opt/bind9/lib/isc/.libs/libisc-9.19.13.so
#3 0x00007fd80e7befc4 in dns_validator_unref () from /var/opt/bind9/lib/dns/.libs/libdns-9.19.13.so
#4 0x00007fd80e7bf0a7 in dns_validator_detach () from /var/opt/bind9/lib/dns/.libs/libdns-9.19.13.so
#5 0x00007fd80e7b73fb in validator_done_cb () from /var/opt/bind9/lib/dns/.libs/libdns-9.19.13.so
#6 0x00007fd80e8e4616 in isc.async_cb () from /var/opt/bind9/lib/isc/.libs/libisc-9.19.13.so
#7 0x00007fd80e0881ed in ?? () from /lib/x86_64-linux-gnu/libuv.so.1
#8 0x00007fd80e0a411e in ?? () from /lib/x86_64-linux-gnu/libuv.so.1
#9 0x00007fd80e08dc88 in uv_run () from /lib/x86_64-linux-gnu/libuv.so.1
#10 0x00007fd80e900a2f in loop_thread () from /var/opt/bind9/lib/isc/.libs/libisc-9.19.13.so
#11 0x00007fd80e9175d1 in thread_body () from /var/opt/bind9/lib/isc/.libs/libisc-9.19.13.so
#12 0x00007fd80e917619 in thread_run () from /var/opt/bind9/lib/isc/.libs/libisc-9.19.13.so
#13 0x00007fd80de90b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#14 0x00007fd80df22a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
```June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)https://gitlab.isc.org/isc-projects/bind9/-/issues/4093zt.c:176: REQUIRE(((zt) != ((void *)0) && ((const isc__magic_t *)(zt))->magic...2023-06-02T11:00:36ZMichal Nowakzt.c:176: REQUIRE(((zt) != ((void *)0) && ((const isc__magic_t *)(zt))->magic == ((('Z') << 24 | ('T') << 16 | ('b') << 8 | ('l'))))) failedI started BIND (80eb7c2) on FreeBSD 12.4-RELEASE-p1 (`~/bind9/bin/named/named -c ns4/named.conf -g &`), then started Flamethrower (`flame --dnssec -P udp -F inet -g file -f 100k_mixed.txt -Q 1500 -p 5300 -v 99 10.53.0.4`). Then I realize...I started BIND (80eb7c2) on FreeBSD 12.4-RELEASE-p1 (`~/bind9/bin/named/named -c ns4/named.conf -g &`), then started Flamethrower (`flame --dnssec -P udp -F inet -g file -f 100k_mixed.txt -Q 1500 -p 5300 -v 99 10.53.0.4`). Then I realized I wanted to do something else, so I terminated Flamethrower with Ctrl-C, then put BIND to the foreground with `fg`, hit Ctrl-C a bunch of times, and hit this assert:
```
zt.c:176: REQUIRE(((zt) != ((void *)0) && ((const isc__magic_t *)(zt))->magic == ((('Z') << 24 | ('T') << 16 | ('b') << 8 | ('l'))))) failed
```
<details><summary>full backtrace</summary>
```
[New LWP 100678]
[New LWP 101524]
[New LWP 101525]
[New LWP 101526]
[New LWP 101527]
Core was generated by `/home/newman/bind9/bin/named/.libs/named -c ns4/named.conf -g'.
Program terminated with signal SIGABRT, Aborted.
Sent by thr_kill() from pid 28899 and user 1001.
#0 0x000000080153bb8a in thr_kill () from /lib/libc.so.7
[Current thread is 1 (LWP 100678)]
#0 0x000000080153bb8a in thr_kill () from /lib/libc.so.7
#1 0x0000000801539f54 in raise () from /lib/libc.so.7
#2 0x00000008014b2449 in abort () from /lib/libc.so.7
#3 0x000000000023d9da in assertion_failed (file=0x800885429 "zt.c", line=176, type=<optimized out>, cond=0x800880794 "((zt) != ((void *)0) && ((const isc__magic_t *)(zt))->magic == ((('Z') << 24 | ('T') << 16 | ('b') << 8 | ('l'))))") at main.c:225
#4 0x0000000800302c4a in isc_assertion_failed (file=0x18946 <error: Cannot access memory at address 0x18946>, line=6, line@entry=176, type=type@entry=isc_assertiontype_require, cond=0x80153bbaa <thr_self+10> "\017\202\004J") at assertions.c:48
#5 0x0000000800a3f218 in dns_zt_find (zt=<optimized out>, name=name@entry=0x80c5a2810, options=<optimized out>, zonep=zonep@entry=0x7ffffffedb98) at zt.c:176
#6 0x00000008009fa699 in dns_view_findzonecut (view=0x801949500, name=0x80c5a2810, fname=fname@entry=0x7ffffffedc10, dcname=dcname@entry=0x7ffffffede20, now=1685018611, options=0, use_hints=<optimized out>, use_cache=<optimized out>, rdataset=0x80c5a2db8, sigrdataset=0x0) at view.c:1090
#7 0x00000008009c6f2a in resume_qmin (arg=<optimized out>) at resolver.c:4181
#8 0x0000000800302f8f in isc__async_cb (handle=<optimized out>) at async.c:112
#9 0x00000008010fcd1a in ?? () from /usr/local/lib/libuv.so.1
#10 0x000000080110e0d5 in ?? () from /usr/local/lib/libuv.so.1
#11 0x00000008010fd2c8 in uv_run () from /usr/local/lib/libuv.so.1
#12 0x0000000800315759 in loop_thread (arg=0x8018c4000) at loop.c:281
#13 0x0000000800315626 in isc_loopmgr_run (loopmgr=0x80188a190) at loop.c:452
#14 0x000000000023d659 in main (argc=<optimized out>, argv=0x7fffffffe9c8) at main.c:1532
```
```
[New LWP 100678]
[New LWP 101524]
[New LWP 101525]
[New LWP 101526]
[New LWP 101527]
Core was generated by `/home/newman/bind9/bin/named/.libs/named -c ns4/named.conf -g'.
Program terminated with signal SIGABRT, Aborted.
Sent by thr_kill() from pid 28899 and user 1001.
#0 0x000000080153bb8a in thr_kill () from /lib/libc.so.7
[Current thread is 1 (LWP 100678)]
Thread 5 (LWP 101527):
#0 0x00000008014e300a in _poll () from /lib/libc.so.7
No symbol table info available.
#1 0x0000000801317b46 in ?? () from /lib/libthr.so.3
No symbol table info available.
#2 0x000000080137125f in ?? () from /usr/local/lib/liburcu.so.8
No symbol table info available.
#3 0x0000000801314fd6 in ?? () from /lib/libthr.so.3
No symbol table info available.
#4 0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdf9fb000
Thread 4 (LWP 101526 "isc-loop-0003"):
#0 0x0000000801526f7a in _kevent () from /lib/libc.so.7
No symbol table info available.
#1 0x00000008013180e3 in ?? () from /lib/libthr.so.3
No symbol table info available.
#2 0x000000080110dc03 in ?? () from /usr/local/lib/libuv.so.1
No symbol table info available.
#3 0x00000008010fd2c8 in uv_run () from /usr/local/lib/libuv.so.1
No symbol table info available.
#4 0x0000000800315759 in loop_thread (arg=arg@entry=0x8018c5740) at loop.c:281
loop = 0x8018c5740
r = 4
ret = <optimized out>
#5 0x0000000800324473 in thread_body (wrap=0x80190f5e0) at thread.c:88
jemalloc_enforce_init = 0x8019be000
func = 0x8003156e0 <loop_thread>
ret = 0x0
arg = 0x8018c5740
#6 thread_run (wrap=0x80190f5e0) at thread.c:103
ret = <optimized out>
#7 0x0000000801314fd6 in ?? () from /lib/libthr.so.3
No symbol table info available.
#8 0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdfbfc000
Thread 3 (LWP 101525 "isc-loop-0002"):
#0 0x0000000801526f7a in _kevent () from /lib/libc.so.7
No symbol table info available.
#1 0x00000008013180e3 in ?? () from /lib/libthr.so.3
No symbol table info available.
#2 0x000000080110dc03 in ?? () from /usr/local/lib/libuv.so.1
No symbol table info available.
#3 0x00000008010fd2c8 in uv_run () from /usr/local/lib/libuv.so.1
No symbol table info available.
#4 0x0000000800315759 in loop_thread (arg=arg@entry=0x8018c4f80) at loop.c:281
loop = 0x8018c4f80
r = 4
ret = <optimized out>
#5 0x0000000800324473 in thread_body (wrap=0x80190f5c0) at thread.c:88
jemalloc_enforce_init = 0x8019bf000
func = 0x8003156e0 <loop_thread>
ret = 0x0
arg = 0x8018c4f80
#6 thread_run (wrap=0x80190f5c0) at thread.c:103
ret = <optimized out>
#7 0x0000000801314fd6 in ?? () from /lib/libthr.so.3
No symbol table info available.
#8 0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdfdfd000
Thread 2 (LWP 101524 "isc-loop-0001"):
#0 0x0000000801526f7a in _kevent () from /lib/libc.so.7
No symbol table info available.
#1 0x00000008013180e3 in ?? () from /lib/libthr.so.3
No symbol table info available.
#2 0x000000080110dc03 in ?? () from /usr/local/lib/libuv.so.1
No symbol table info available.
#3 0x00000008010fd2c8 in uv_run () from /usr/local/lib/libuv.so.1
No symbol table info available.
#4 0x0000000800315759 in loop_thread (arg=arg@entry=0x8018c47c0) at loop.c:281
loop = 0x8018c47c0
r = 4
ret = <optimized out>
#5 0x0000000800324473 in thread_body (wrap=0x80190f600) at thread.c:88
jemalloc_enforce_init = 0x8019bd000
func = 0x8003156e0 <loop_thread>
ret = 0x0
arg = 0x8018c47c0
#6 thread_run (wrap=0x80190f600) at thread.c:103
ret = <optimized out>
#7 0x0000000801314fd6 in ?? () from /lib/libthr.so.3
No symbol table info available.
#8 0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdfffe000
Thread 1 (LWP 100678):
#0 0x000000080153bb8a in thr_kill () from /lib/libc.so.7
No symbol table info available.
#1 0x0000000801539f54 in raise () from /lib/libc.so.7
No symbol table info available.
#2 0x00000008014b2449 in abort () from /lib/libc.so.7
No symbol table info available.
#3 0x000000000023d9da in assertion_failed (file=0x800885429 "zt.c", line=176, type=<optimized out>, cond=0x800880794 "((zt) != ((void *)0) && ((const isc__magic_t *)(zt))->magic == ((('Z') << 24 | ('T') << 16 | ('b') << 8 | ('l'))))") at main.c:225
No locals.
#4 0x0000000800302c4a in isc_assertion_failed (file=0x18946 <error: Cannot access memory at address 0x18946>, line=6, line@entry=176, type=type@entry=isc_assertiontype_require, cond=0x80153bbaa <thr_self+10> "\017\202\004J") at assertions.c:48
No locals.
#5 0x0000000800a3f218 in dns_zt_find (zt=<optimized out>, name=name@entry=0x80c5a2810, options=<optimized out>, zonep=zonep@entry=0x7ffffffedb98) at zt.c:176
qpr = {magic = 256, root_ref = 0, base = 0x80185e000, uctx = 0x7ffffffed850, methods = 0x801319c3c, tid = 100678}
pval = 0x0
exactopts = <optimized out>
exactmask = (DNS_ZTFIND_EXACT | DNS_ZTFIND_NOEXACT)
ival = <optimized out>
result = <optimized out>
#6 0x00000008009fa699 in dns_view_findzonecut (view=0x801949500, name=0x80c5a2810, fname=fname@entry=0x7ffffffedc10, dcname=dcname@entry=0x7ffffffede20, now=1685018611, options=0, use_hints=<optimized out>, use_cache=<optimized out>, rdataset=0x80c5a2db8, sigrdataset=0x0) at view.c:1090
zrdataset = {magic = 1145983826, methods = 0x0, link = {prev = 0xffffffffffffffff, next = 0xffffffffffffffff}, rdclass = 0, type = 0, ttl = 0, trust = 0, covers = 0, attributes = 0, count = 4294967295, resign = 0, private1 = 0x0, private2 = 0x0, private3 = 0x0, privateuint4 = 0, private5 = 0x0, private6 = 0x0, private7 = 0x0}
zsigrdataset = {magic = 1145983826, methods = 0x0, link = {prev = 0xffffffffffffffff, next = 0xffffffffffffffff}, rdclass = 0, type = 0, ttl = 0, trust = 0, covers = 0, attributes = 0, count = 4294967295, resign = 0, private1 = 0x0, private2 = 0x0, private3 = 0x0, privateuint4 = 0, private5 = 0x0, private6 = 0x0, private7 = 0x0}
zfixedname = {name = {magic = 1145983854, ndata = 0x0, length = 0, labels = 0, attributes = {absolute = false, readonly = false, dynamic = false, dynoffsets = false, nocompress = false, cache = false, answer = false, ncache = false, chaining = false, chase = false, wildcard = false, prerequisite = false, update = false, hasupdaterec = false}, offsets = 0x7ffffffed8e8 "\200<\242\001\b", buffer = 0x7ffffffed968, link = {prev = 0xffffffffffffffff, next = 0xffffffffffffffff}, list = {head = 0x0, tail = 0x0}}, offsets = "\200<\242\001\b\000\000\000\260\342\205\001\b", '\000' <repeats 11 times>, "p\331\376\377\377\177\000\000\001\322\061\001\b", '\000' <repeats 12 times>, "\367\244\001\b\000\000\000\200\027\243\001\b\000\000\000\200<\242\001\b", '\000' <repeats 11 times>, "c\200\207\000\000\000\000\000\260\342\205\001\b\000\000\000\000\f<\t\b\000\000\000\320\325\271\f\b\000\000\000\000\260\334\f\b\000\000\000\300F\210\001\b\000\000", buffer = {magic = 1114990113, base = 0x7ffffffed9a8, length = 255, used = 0, current = 0, active = 0, extra = 0, dynamic = false, link = {prev = 0xffffffffffffffff, next = 0xffffffffffffffff}, mctx = 0x0}, data = "\231\324J\001\b\000\000\000 \332\376\377\377\177\000\000\231\324J\001\b\000\000\000\060\332\376\377\377\177\000\000\000\342\253\t\b\000\000\000\020\241\200\001\b\000\000\000\000\f<\t\b\000\000\000\000X\211\001\b\000\000\000\070\332\376\377\377\177\000\000k\264\256Jp)\331o \000\000\000\000\000\000\000 \000\000\000\000\000\000\000\300F\210\001\b\000\000\000v\220\207\000\b\000\000\000\000\342\253\t\b\000\000\000`\332\376\377\377\177\000\000\071\205\061\000\b", '\000' <repeats 12 times>, "p\300\002\b\000\000\000p\263\334\f\b\000\000\000\000\260\334\f\b\000\000\000P\263\334\f\b\000\000\000v\220\207\000\b\000\000\000 \333\376\377\377\177\000\000\324\322\233\000\b\000\000\000"...}
db = 0x0
zone = 0x0
try_hints = false
use_zone = false
ztoptions = 0
zfname = 0x0
result = <optimized out>
is_cache = <optimized out>
#7 0x00000008009c6f2a in resume_qmin (arg=<optimized out>) at resolver.c:4181
ffixed = {name = {magic = 1145983854, ndata = 0x0, length = 0, labels = 0, attributes = {absolute = false, readonly = false, dynamic = false, dynoffsets = false, nocompress = false, cache = false, answer = false, ncache = false, chaining = false, chase = false, wildcard = false, prerequisite = false, update = false, hasupdaterec = false}, offsets = 0x7ffffffedc60 "\020\241\200\001\b", buffer = 0x7ffffffedce0, link = {prev = 0xffffffffffffffff, next = 0xffffffffffffffff}, list = {head = 0x0, tail = 0x0}}, offsets = "\020\241\200\001\b\000\000\000\240\024\000\000\000\000\000\000@\304\200\001\b\000\000\000\370\244\200\001\b\000\000\000\340\333\376\377\377\177\000\000\360\244\200\001\b\000\000\000\020\241\200\001\b\000\000\000\200\t\240\001\b", '\000' <repeats 27 times>, "\001\000\000\000d\000\000\000\363\251\207\000\b\000\000\000p\b\000\000\000\000\000\000x\331\376\377\377\177\000\000\300\243\200\001\b\000\000", buffer = {magic = 1114990113, base = 0x7ffffffedd20, length = 255, used = 0, current = 0, active = 0, extra = 0, dynamic = false, link = {prev = 0xffffffffffffffff, next = 0xffffffffffffffff}, mctx = 0x0}, data = "\360\335\376\377\377\177", '\000' <repeats 18 times>, "\031", '\000' <repeats 16 times>, "\254\210\005\b\000\000\000@\304\200\001\b\000\000\000\003,1\000\b\000\000\000 \336\376\377\377\177\000\000\231\324J\001\b\000\000\000@\304\200\001\b\000\000\000\242\307J\001\b", '\000' <repeats 28 times>, "\254\210\005\b\000\000\000\363\251\207\000\b", '\000' <repeats 27 times>, "\200$(\000\000\000\000\000\273\327\207\305\000\000\000\000@\336\376\377\377\177\000\000\001\322\061\001\b", '\000' <repeats 11 times>...}
dcfixed = {name = {magic = 1145983854, ndata = 0x0, length = 0, labels = 0, attributes = {absolute = false, readonly = false, dynamic = false, dynoffsets = false, nocompress = false, cache = false, answer = false, ncache = false, chaining = false, chase = false, wildcard = false, prerequisite = false, update = false, hasupdaterec = false}, offsets = 0x7ffffffede70 "k\264\256Jp)\331o\n", buffer = 0x7ffffffedef0, link = {prev = 0xffffffffffffffff, next = 0xffffffffffffffff}, list = {head = 0x0, tail = 0x0}}, offsets = "k\264\256Jp)\331o\n", '\000' <repeats 15 times>, "\031\000\000\000\000\000\000\000\020\337\376\377\377\177\000\000`}v\001\b\000\000\000\260\342\205\001\b", '\000' <repeats 11 times>, " \337\376\377\377\177\000\000\001\322\061\001\b\000\000\000k\264\256Jp)\331o\n\000\000\000\000\000\000\000\020\337\376\377\377\177\000\000k\264\256Jp)\331oP\026v\005\b\000\000\000\000\000\000\000\000\000\000", buffer = {magic = 1114990113, base = 0x7ffffffedf30, length = 255, used = 0, current = 0, active = 0, extra = 0, dynamic = false, link = {prev = 0xffffffffffffffff, next = 0xffffffffffffffff}, mctx = 0x0}, data = "\000>c\005\b\000\000\000{\343\206\000\b\000\000\000\200\276\255\005\b\000\000\000\300\276\255\005\b\000\000\000 \340\376\377\377\177\000\000\343>\215\000\b\000\000\000 \340\376\377\377\177\000\000\000\254\210\005\b\000\000\000\234\245\200\001\b", '\000' <repeats 11 times>, "\360\337\376\377\377\177\000\000\336\327J\001\b\000\000\000\020\241\200\001\b\000\000\000\000\214Z\f\b\000\000\000\260\342\205\001\b", '\000' <repeats 11 times>, " \340\376\377\377\177\000\000\001\322\061\001\b", '\000' <repeats 11 times>, "\370Wod\n:`\037p\257\210\005\b", '\000' <repeats 11 times>, "\200$(\000\000\000\000\000\360%(\000\000\000\000\000(\000\000\000\031\000\000\000\000"...}
resp = 0x0
dcname = 0x7ffffffede20
fname = 0x7ffffffedc10
findoptions = 0
res = 0x802c07000
fctx = 0x80c5a2800
result = ISC_R_FAILURE
#8 0x0000000800302f8f in isc__async_cb (handle=<optimized out>) at async.c:112
job = 0x802de9d40
jobs = {head = {node = {next = 0x8056334d0}}, tail = {p = 0x802de9090}}
loop = 0x8018c4000
ret = <optimized out>
node = 0x802de9d50
next = 0x80b8023d0
#9 0x00000008010fcd1a in ?? () from /usr/local/lib/libuv.so.1
No symbol table info available.
#10 0x000000080110e0d5 in ?? () from /usr/local/lib/libuv.so.1
No symbol table info available.
#11 0x00000008010fd2c8 in uv_run () from /usr/local/lib/libuv.so.1
No symbol table info available.
#12 0x0000000800315759 in loop_thread (arg=0x8018c4000) at loop.c:281
loop = 0x8018c4000
r = 0
ret = <optimized out>
#13 0x0000000800315626 in isc_loopmgr_run (loopmgr=0x80188a190) at loop.c:452
free_call_rcu_data = <optimized out>
#14 0x000000000023d659 in main (argc=<optimized out>, argv=0x7fffffffe9c8) at main.c:1532
result = <optimized out>
```
</details>June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Mark AndrewsMark Andrewshttps://gitlab.isc.org/isc-projects/bind9/-/issues/4085Stats channel shutdown crash in httpd.c:874: REQUIRE((mgr->flags & 0x00000001...2023-06-14T09:44:21ZMichal NowakStats channel shutdown crash in httpd.c:874: REQUIRE((mgr->flags & 0x00000001) == 0)If I modify the `respdiff-long` CI job to run with 20,000 instead of 100,000 queries, add a stats channel, and point [resource monitor](https://gitlab.isc.org/isc-projects/resource-monitor) to the channel on the instance in AWS autoscale...If I modify the `respdiff-long` CI job to run with 20,000 instead of 100,000 queries, add a stats channel, and point [resource monitor](https://gitlab.isc.org/isc-projects/resource-monitor) to the channel on the instance in AWS autoscaler, [BIND reliably crashes](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3404866).
```
23-May-2023 11:00:59.803 no longer listening on 10.53.0.10#5300
23-May-2023 11:00:59.815 stopping command channel on 127.0.0.1#953
23-May-2023 11:00:59.815 stopping statistics channel on 10.53.0.10#8080
23-May-2023 11:00:59.815 httpd.c:874: REQUIRE((mgr->flags & 0x00000001) == 0) failed
```
```
Core was generated by `/builds/isc-projects/bind9/bin/named/.libs/named -4 -g -c /builds/isc-projects/'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[Current thread is 1 (Thread 0x7f68e904f300 (LWP 17994))]
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f68eb55c537 in __GI_abort () at abort.c:79
#2 0x000055c084eff4c5 in assertion_failed (file=0x7f68ec06681d "httpd.c", line=874, type=isc_assertiontype_require, cond=0x7f68ec066778 "(mgr->flags & 0x00000001) == 0") at main.c:225
#3 0x00007f68ec02d950 in isc_assertion_failed (file=file@entry=0x7f68ec06681d "httpd.c", line=line@entry=874, type=type@entry=isc_assertiontype_require, cond=cond@entry=0x7f68ec066778 "(mgr->flags & 0x00000001) == 0") at assertions.c:48
#4 0x00007f68ec038a34 in httpd_request (handle=0x7f6855395080, eresult=eresult@entry=ISC_R_SUCCESS, region=region@entry=0x0, arg=0x7f682c445080) at httpd.c:874
#5 0x00007f68ec038ff6 in isc_httpdmgr_shutdown (httpdmgrp=httpdmgrp@entry=0x7f68e5539940) at httpd.c:948
#6 0x000055c084f31bf5 in shutdown_listener (listener=0x7f68e5539940) at statschannel.c:2983
#7 0x000055c084f327ef in named_statschannels_shutdown (server=server@entry=0x7f68e5525340) at statschannel.c:3345
#8 0x000055c084f23fc9 in shutdown_server (arg=0x7f68e5525340) at server.c:10021
#9 0x00007f68ec02dc95 in isc__async_cb (handle=<optimized out>) at async.c:112
#10 0x00007f68ebf35e93 in uv__async_io (loop=0x7f68e559b860, w=0x7f68e559ba28, events=1) at /usr/src/libuv-v1.44.1/src/unix/async.c:163
#11 0x00007f68ebf51bc3 in uv__io_poll (loop=0x7f68e559b860, timeout=29975) at /usr/src/libuv-v1.44.1/src/unix/epoll.c:374
#12 0x00007f68ebf368ae in uv_run (loop=0x7f68e559b860, mode=UV_RUN_DEFAULT) at /usr/src/libuv-v1.44.1/src/unix/core.c:391
#13 0x00007f68ec0405a7 in loop_thread (arg=arg@entry=0x7f68e559b840) at loop.c:281
#14 0x00007f68ec04f7b8 in thread_body (wrap=0x7f68e40b91c0) at thread.c:87
#15 0x00007f68ec04f832 in isc_thread_main (func=func@entry=0x7f68ec040524 <loop_thread>, arg=0x7f68e559b840) at thread.c:118
#16 0x00007f68ec041287 in isc_loopmgr_run (loopmgr=0x7f68e5582a80) at loop.c:452
#17 0x000055c084f0234f in main (argc=<optimized out>, argv=<optimized out>) at main.c:1532
```
isc-projects/bind9#3552 looks similar to this one but was reported before d8df29e37d99b7b94b15a306dcb2b7488c9a92ad added the failing assert.
[core.17994-backtrace.txt](/uploads/a74ad1705819b9d73892d9e559029233/core.17994-backtrace.txt)
[named.run](/uploads/44e6a096e43f92cc81ee8e6978ed8e60/named.run)
[named.conf](/uploads/ed2aeda9c0f8e6e462fd535dc5835bdb/named.conf)June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)https://gitlab.isc.org/isc-projects/bind9/-/issues/4082rrl test persistently fails in `system:clang:tsan`2023-05-26T12:43:42ZTom Krizekrrl test persistently fails in `system:clang:tsan``rrl` fails ([job#3401261](
https://gitlab.isc.org/isc-projects/bind9/-/jobs/3401261)) with:
```
I:rrl:"would limit" not found in log file.
I:rrl:exit status: 1
```
when executed under TSAN. It started happening consistently since [lib...`rrl` fails ([job#3401261](
https://gitlab.isc.org/isc-projects/bind9/-/jobs/3401261)) with:
```
I:rrl:"would limit" not found in log file.
I:rrl:exit status: 1
```
when executed under TSAN. It started happening consistently since [liburcu-qsbr changes](https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/7668), which significantly [increased the runtime](https://gitlab.isc.org/isc-projects/bind9/-/issues/4081) of `system:clang:tsan`. The MR might've just exposed some timing issue rather than being the cause of the problem.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Tom KrizekTom Krizekhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4081system:clang:tsan runtime doubled after switching to liburcu-qsbr2023-05-31T09:51:49ZTom Krizeksystem:clang:tsan runtime doubled after switching to liburcu-qsbrAfter merging the [liburcu-qsbr changes](https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/7668#note_374875), the total runtime of [system:clang:tsan](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3381815) increased to 42 min...After merging the [liburcu-qsbr changes](https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/7668#note_374875), the total runtime of [system:clang:tsan](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3381815) increased to 42 minutes up from around 19 minutes that it used to take [before](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3381669).
The runtime of a few tests increased by 3x-4x:
```
runtime with TSAN
rpz 4m 16m
rpzrecurse 6.5m 18m
kasp 3m 9m
rrsetorder 1m 4.5m
```
Side note: The increased runtime also unearthed some timing issues, making some tests fail more frequently in `system:clang:tsan` - notably, the [`upforwd`](https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/6809#note_374883) test (which should be fixed once #4072 is resolved) and [`rrl`](https://gitlab.isc.org/isc-projects/bind9/-/issues/4082).June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)https://gitlab.isc.org/isc-projects/bind9/-/issues/4077thread.c:60:15 error: dereference of possibly-NULL 'wrap' in thread_wrap()2023-05-19T09:07:19ZMichal Nowakthread.c:60:15 error: dereference of possibly-NULL 'wrap' in thread_wrap()Job [#3396030](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3396030) failed for 0d9e27dd7ed9d85f6a5b8ff846f2cfdfc92a7899:
```
thread.c: In function 'thread_wrap':
thread.c:60:15: error: dereference of possibly-NULL 'wrap' [CWE-690] [...Job [#3396030](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3396030) failed for 0d9e27dd7ed9d85f6a5b8ff846f2cfdfc92a7899:
```
thread.c: In function 'thread_wrap':
thread.c:60:15: error: dereference of possibly-NULL 'wrap' [CWE-690] [-Werror=analyzer-possible-null-dereference]
60 | *wrap = (struct thread_wrap){
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~
61 | .func = func,
| ~~~~~~~~~~~~~
62 | .arg = arg,
| ~~~~~~~~~~~
63 | };
| ~
'thread_wrap': events 1-2
|
| 59 | struct thread_wrap *wrap = malloc(sizeof(*wrap));
| | ^~~~~~~~~~~~~~~~~~~~~
| | |
| | (1) this call could return NULL
| 60 | *wrap = (struct thread_wrap){
| | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| | |
| | (2) 'wrap' could be NULL: unchecked value from (1)
| 61 | .func = func,
| | ~~~~~~~~~~~~~
| 62 | .arg = arg,
| | ~~~~~~~~~~~
| 63 | };
| | ~
|
```
This is because in 7d1ceaf35dbc25dfbca32deffa7f6ff8f452e75f the `RUNTIME_CHECK(wrap != NULL);` was dropped.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Michal NowakMichal Nowakhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4073Data race lib/dns/qp.c:563 in chunk_discount2023-05-20T07:27:45ZMichal NowakData race lib/dns/qp.c:563 in chunk_discountJob [#3390630](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3390630) failed for 55cc0715472b8b251c5692f9ae0a50e0d51ad796.
```
WARNING: ThreadSanitizer: data race
Write of size 4 at 0x000000000001 by thread T1:
#0 chunk_discou...Job [#3390630](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3390630) failed for 55cc0715472b8b251c5692f9ae0a50e0d51ad796.
```
WARNING: ThreadSanitizer: data race
Write of size 4 at 0x000000000001 by thread T1:
#0 chunk_discount lib/dns/qp.c:563
#1 chunk_free lib/dns/qp.c:591
#2 destroy_guts lib/dns/qp.c:1403
#3 qpmulti_destroy_cb lib/dns/qp.c:1441
#4 call_rcu_thread <null>
Previous read of size 4 at 0x000000000001 by thread T2 (mutexes: write M1):
#0 reclaim_chunks_cb lib/dns/qp.c:663
#1 call_rcu_thread <null>
Location is heap block of size 168 at 0x000000000008 allocated by main thread:
#0 malloc <null>
#1 mallocx lib/isc/jemalloc_shim.h:65
#2 mem_get lib/isc/mem.c:305
#3 isc__mem_get lib/isc/mem.c:674
#4 dns_qpmulti_create lib/dns/qp.c:1375
#5 many_transactions tests/dns/qpmulti_test.c:370
#6 isc__async_cb lib/isc/async.c:112
#7 uv__async_io /usr/src/libuv-v1.44.1/src/unix/async.c:163
#8 thread_body lib/isc/thread.c:87
#9 isc_thread_main lib/isc/thread.c:118
#10 isc_loopmgr_run lib/isc/loop.c:452
#11 run_test_qpmulti tests/dns/qpmulti_test.c:388
#12 cmocka_run_one_test_or_fixture <null>
#13 __libc_start_call_main <null>
Mutex M1 (0x000000000022) created at:
#0 pthread_mutex_init <null>
#1 dns_qpmulti_create lib/dns/qp.c:1380
#2 many_transactions tests/dns/qpmulti_test.c:370
#3 isc__async_cb lib/isc/async.c:112
#4 uv__async_io /usr/src/libuv-v1.44.1/src/unix/async.c:163
#5 thread_body lib/isc/thread.c:87
#6 isc_thread_main lib/isc/thread.c:118
#7 isc_loopmgr_run lib/isc/loop.c:452
#8 run_test_qpmulti tests/dns/qpmulti_test.c:388
#9 cmocka_run_one_test_or_fixture <null>
#10 __libc_start_call_main <null>
Thread T1 (running) created by main thread at:
#0 pthread_create <null>
#1 call_rcu_data_init <null>
#2 run_test_qpmulti tests/dns/qpmulti_test.c:388
#3 cmocka_run_one_test_or_fixture <null>
#4 __libc_start_call_main <null>
Thread T2 (running) created by main thread at:
#0 pthread_create <null>
#1 call_rcu_data_init <null>
#2 run_test_qpmulti tests/dns/qpmulti_test.c:388
#3 cmocka_run_one_test_or_fixture <null>
#4 __libc_start_call_main <null>
SUMMARY: ThreadSanitizer: data race lib/dns/qp.c:563 in chunk_discount
```
```
WARNING: ThreadSanitizer: data race
Write of size 1 at 0x000000000001 by thread T1:
#0 pthread_mutex_destroy <null>
#1 qpmulti_destroy_cb lib/dns/qp.c:1442
#2 call_rcu_thread <null>
Previous atomic read of size 1 at 0x000000000001 by thread T2 (mutexes: write M1):
#0 pthread_mutex_unlock <null>
#1 reclaim_chunks_cb lib/dns/qp.c:668
#2 call_rcu_thread <null>
Location is heap block of size 168 at 0x000000000007 allocated by main thread:
#0 malloc <null>
#1 mallocx lib/isc/jemalloc_shim.h:65
#2 mem_get lib/isc/mem.c:305
#3 isc__mem_get lib/isc/mem.c:674
#4 dns_qpmulti_create lib/dns/qp.c:1375
#5 many_transactions tests/dns/qpmulti_test.c:370
#6 isc__async_cb lib/isc/async.c:112
#7 uv__async_io /usr/src/libuv-v1.44.1/src/unix/async.c:163
#8 thread_body lib/isc/thread.c:87
#9 isc_thread_main lib/isc/thread.c:118
#10 isc_loopmgr_run lib/isc/loop.c:452
#11 run_test_qpmulti tests/dns/qpmulti_test.c:388
#12 cmocka_run_one_test_or_fixture <null>
#13 __libc_start_call_main <null>
Mutex M1 is already destroyed.
Thread T1 (running) created by main thread at:
#0 pthread_create <null>
#1 call_rcu_data_init <null>
#2 run_test_qpmulti tests/dns/qpmulti_test.c:388
#3 cmocka_run_one_test_or_fixture <null>
#4 __libc_start_call_main <null>
Thread T2 (running) created by main thread at:
#0 pthread_create <null>
#1 call_rcu_data_init <null>
#2 run_test_qpmulti tests/dns/qpmulti_test.c:388
#3 cmocka_run_one_test_or_fixture <null>
#4 __libc_start_call_main <null>
SUMMARY: ThreadSanitizer: data race in __interceptor_pthread_mutex_destroy
```
```
WARNING: ThreadSanitizer: data race
Write of size 8 at 0x000000000001 by thread T1:
#0 free <null>
#1 sdallocx lib/isc/jemalloc_shim.h:80
#2 mem_put lib/isc/mem.c:327
#3 isc__mem_putanddetach lib/isc/mem.c:560
#4 qpmulti_destroy_cb lib/dns/qp.c:1445
#5 call_rcu_thread <null>
Previous read of size 4 at 0x000000000001 by thread T2 (mutexes: write M1):
#0 reclaim_chunks_cb lib/dns/qp.c:663
#1 call_rcu_thread <null>
Mutex M1 is already destroyed.
Thread T1 (running) created by main thread at:
#0 pthread_create <null>
#1 call_rcu_data_init <null>
#2 run_test_qpmulti tests/dns/qpmulti_test.c:388
#3 cmocka_run_one_test_or_fixture <null>
#4 __libc_start_call_main <null>
Thread T2 (running) created by main thread at:
#0 pthread_create <null>
#1 call_rcu_data_init <null>
#2 run_test_qpmulti tests/dns/qpmulti_test.c:388
#3 cmocka_run_one_test_or_fixture <null>
#4 __libc_start_call_main <null>
SUMMARY: ThreadSanitizer: data race in __interceptor_free
```
Might be ~Duplicate of isc-projects/bind9#4070.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Tony FinchTony Finchhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4070Data race in pthread_mutex_destroy (xferquota)2023-05-22T09:58:02ZMichal NowakData race in pthread_mutex_destroy (xferquota)Job [#3387972](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3387972) failed for 15eaf9d3f2d9a3ab8e3ca9dc8ff63f07deed48b7.
The `xferquota` system test produces TSAN error in `system:clang:tsan` CI job.
```
WARNING: ThreadSanitizer: ...Job [#3387972](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3387972) failed for 15eaf9d3f2d9a3ab8e3ca9dc8ff63f07deed48b7.
The `xferquota` system test produces TSAN error in `system:clang:tsan` CI job.
```
WARNING: ThreadSanitizer: data race
Write of size 1 at 0x000000000001 by thread T1:
#0 pthread_mutex_destroy <null> (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#1 qpmulti_destroy_cb lib/dns/qp.c:1442:2 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#2 <null> <null> (BuildId: 317457052cc2fe394e6cea9241fff1d4303f7d0a)
Previous atomic read of size 1 at 0x000000000001 by thread T2 (mutexes: write M1):
#0 pthread_mutex_unlock <null> (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#1 reclaim_chunks_cb lib/dns/qp.c:668:2 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#2 <null> <null> (BuildId: 317457052cc2fe394e6cea9241fff1d4303f7d0a)
Location is heap block of size 168 at 0x000000000007 allocated by main thread:
#0 malloc <null> (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#1 mallocx lib/isc/./jemalloc_shim.h:65:14 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#2 mem_get lib/isc/mem.c:305:8
#3 isc__mem_get lib/isc/mem.c:674:8 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#4 dns_qpmulti_create lib/dns/qp.c:1375:25 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#5 dns_zt_create lib/dns/zt.c:104:2 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#6 dns_view_create lib/dns/view.c:137:2 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#7 create_view bin/named/server.c:6440:11 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#8 load_configuration bin/named/server.c:9140:12 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#9 run_server bin/named/server.c:9982:2 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#10 isc__async_cb lib/isc/async.c:112:3 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#11 uv__async_io /usr/src/libuv-v1.44.1/src/unix/async.c:163:5 (BuildId: 120c450d14885aa5308bc95c4ea77de2c2b1cc36)
#12 thread_body lib/isc/thread.c:87:8 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#13 isc_thread_main lib/isc/thread.c:118:2
#14 isc_loopmgr_run lib/isc/loop.c:452:2 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#15 main bin/named/main.c:1532:2 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
Mutex M1 (0x000000000001) created at:
#0 pthread_mutex_init <null> (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#1 dns_qpmulti_create lib/dns/qp.c:1380:2 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#2 dns_zt_create lib/dns/zt.c:104:2 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#3 dns_view_create lib/dns/view.c:137:2 (BuildId: a40b420aa20f73afe0e46a58e58153e948670aef)
#4 create_view bin/named/server.c:6440:11 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#5 load_configuration bin/named/server.c:9140:12 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#6 run_server bin/named/server.c:9982:2 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#7 isc__async_cb lib/isc/async.c:112:3 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#8 uv__async_io /usr/src/libuv-v1.44.1/src/unix/async.c:163:5 (BuildId: 120c450d14885aa5308bc95c4ea77de2c2b1cc36)
#9 thread_body lib/isc/thread.c:87:8 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#10 isc_thread_main lib/isc/thread.c:118:2
#11 isc_loopmgr_run lib/isc/loop.c:452:2 (BuildId: 177803d1e9f1786301abb0f570f88676ff9a5f04)
#12 main bin/named/main.c:1532:2 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
Thread T1 (running) created by main thread at:
#0 pthread_create <null> (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#1 <null> <null> (BuildId: 317457052cc2fe394e6cea9241fff1d4303f7d0a)
#2 main bin/named/main.c:1532:2 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
Thread T2 (running) created by main thread at:
#0 pthread_create <null> (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
#1 <null> <null> (BuildId: 317457052cc2fe394e6cea9241fff1d4303f7d0a)
#2 main bin/named/main.c:1532:2 (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098)
SUMMARY: ThreadSanitizer: data race (BuildId: 7e82076ead252161dd1e856775766c8d7ccea098) in pthread_mutex_destroy
```
In daily CI pipelines, it started on Saturday when the `fanf-urcu-qp` branch was merged, and happens with 75 % reliability since.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Tony FinchTony Finchhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4069upforwd test failure2023-05-17T14:21:58ZEvan Huntupforwd test failureWhen CI is running low, the upforwd test fails. Previously this could cause an assertion failure, which has now been fixed by #4064, but we still need to address the failure.
The failure is described at https://gitlab.isc.org/isc-projec...When CI is running low, the upforwd test fails. Previously this could cause an assertion failure, which has now been fixed by #4064, but we still need to address the failure.
The failure is described at https://gitlab.isc.org/isc-projects/bind9/-/issues/4064#note_372499. It usually looks like this:
```
I:upforwd:checking DNSTAP logging of UPDATE forwarded update replies (26)
I:upforwd:UQ=4 UR=1
I:upforwd:failed
...
I:upforwd:checking DNSTAP logging of UPDATE forwarded update replies (28)
I:upforwd:UQ=3 UR=1
I:upforwd:failed
```
As I said above, I suspect this is because the test platform is slow: some update forwarding requests are timing out before replies are sent, and so we count more requests (UQ) than replies (UR) in the DNSTAP stats.
I'd love to find out why these CI jobs are so draggy. It shouldn't be taking 30-40 minutes for a system test job to run. It's possible there's some environmental issue, and if we fixed it the test failures would go away. That said, however, I think we can fix the test so it's more resilient.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Tony FinchTony Finchhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4068CID 455002: Error handling issues in lib/isc/loop.c2023-05-15T20:49:24ZMichal NowakCID 455002: Error handling issues in lib/isc/loop.cI think this Coverity Scan issue was triggered by adding two `uv_async_send()` calls in 7b1d985de2f3be760ea5fedd191482fb943ed934 that check return values, so Coverity thinks we might want to check `uv_async_send()`s return value elsewher...I think this Coverity Scan issue was triggered by adding two `uv_async_send()` calls in 7b1d985de2f3be760ea5fedd191482fb943ed934 that check return values, so Coverity thinks we might want to check `uv_async_send()`s return value elsewhere as well:
```
/lib/isc/loop.c: 483 in isc_loopmgr_pause()
477
478 /* Skip current loop */
479 if (i == isc_tid()) {
480 continue;
481 }
482
>>> CID 455002: Error handling issues (CHECKED_RETURN)
>>> Calling "uv_async_send" without checking return value (as is done elsewhere 5 out of 6 times).
483 uv_async_send(&loop->pause_trigger);
484 }
485
486 RUNTIME_CHECK(atomic_compare_exchange_strong(&loopmgr->paused,
487 &(bool){ false }, true));
488 pause_loop(CURRENT_LOOP(loopmgr));
```
For your consideration @fanf.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Tony FinchTony Finchhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4067Configuring with --with-liburcu=qsbr won't build2023-05-15T20:49:47ZMichal NowakConfiguring with --with-liburcu=qsbr won't buildThe [`pairwise`](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3385043) test fails with
```
In file included from ../../../lib/isc/include/isc/job.h:31,
from ../../../lib/isc/include/isc/async.h:28,
...The [`pairwise`](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3385043) test fails with
```
In file included from ../../../lib/isc/include/isc/job.h:31,
from ../../../lib/isc/include/isc/async.h:28,
from ../../../lib/dns/rbtdb.c:22:
../../../lib/dns/rbtdb.c: In function 'rdataset_addglue':
../../../lib/isc/include/isc/urcu.h:113:3: error: expected expression before 'if'
113 | if (!urcu_qsbr_read_ongoing()) { \
| ^~
../../../lib/isc/include/isc/urcu.h:120:30: note: in expansion of macro 'isc_qsbr_rcu_dereference'
120 | #define rcu_dereference(ptr) isc_qsbr_rcu_dereference(ptr)
| ^~~~~~~~~~~~~~~~~~~~~~~~
../../../lib/dns/rbtdb.c:9857:23: note: in expansion of macro 'rcu_dereference'
9857 | rbtdb_glue_t *glue = rcu_dereference(header->glue_list);
| ^~~~~~~~~~~~~~~
```
for the following configuration:
```
--disable-developer --enable-warn-error --with-liburcu=qsbr --disable-kqueue --enable-epoll --disable-devpoll --disable-geoip --with-locktype=adaptive --enable-doh --with-libnghttp2=auto --enable-pthread-rwlock --with-gssapi=auto --with-lmdb=auto --without-libxml2 --with-json-c=detect --with-zlib=auto --without-libsystemd --enable-tcp-fastopen --with-readline=yes --enable-chroot --enable-fixed-rrset --disable-dnstap --without-libidn2 --with-cmocka=yes --without-jemalloc --disable-leak-detection --enable-singletrace --enable-querytrace --disable-auto-validation --enable-dnsrps --enable-dnsrps-dl --disable-full-report
```
`--with-liburcu=qsbr` is the culprit here.
Starts with !7895.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Tony FinchTony Finchhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4064upforwd test causes assertion failure in streamdns.c2023-05-17T13:49:54ZTony Finchupforwd test causes assertion failure in streamdns.cI modified the `upforwd` test to add a delay:
```
--- bin/tests/system/upforwd/tests.sh
+++ bin/tests/system/upforwd/tests.sh
@@ -23,6 +23,8 @@ RNDCCMD="$RNDC -p ${CONTROLPORT} -c ../common/rndc.conf"
status=0
n=1
capture_dnstap() {
+...I modified the `upforwd` test to add a delay:
```
--- bin/tests/system/upforwd/tests.sh
+++ bin/tests/system/upforwd/tests.sh
@@ -23,6 +23,8 @@ RNDCCMD="$RNDC -p ${CONTROLPORT} -c ../common/rndc.conf"
status=0
n=1
capture_dnstap() {
+ echo_i "sleeping 15 seconds"
+ sleep 15
retry_quiet 20 test -f ns3/dnstap.out && mv ns3/dnstap.out dnstap.out.$n
$RNDCCMD -s 10.53.0.3 dnstap -reopen
}
```
With this change, I get the following failure fairly repeatably:
```
12-May-2023 18:59:21.745 dispatch 0x7b5000050000: TCP read:timed out:requests 20
12-May-2023 18:59:21.745 dispatch 0x7b5000050000: TCP response 0x7b5000050200: reading from 0x7b4c00040180
12-May-2023 18:59:21.745 netmgr/streamdns.c:834: REQUIRE(sock->recv_handle == ((void *)0)) failed
```
The failures also happen with revision 88cf7e7e9a5c70201d3f981e44b372785d19fb85 (main yesterday before RCU changes), when compiled with thread sanitizer and without.
I am running BIND on my Debian Bullseye box.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)https://gitlab.isc.org/isc-projects/bind9/-/issues/4023LeakSanitizer detected memory leak in nsupdate2023-05-11T14:05:33ZMichal NowakLeakSanitizer detected memory leak in nsupdateJob [#3336065](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3336065) failed in `system:clang:asan` which has Clang 16.
```
$NSUPDATE -D -S -A CA/CA-other.pem -K CA/certs/srv01.client01.example.nil.key -E CA/certs/client01.crt01.examp...Job [#3336065](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3336065) failed in `system:clang:asan` which has Clang 16.
```
$NSUPDATE -D -S -A CA/CA-other.pem -K CA/certs/srv01.client01.example.nil.key -E CA/certs/client01.crt01.example.nil.pem -k ns1/ddns.key <<END > nsupdate.out.test$n 2>&1
server 10.53.0.1 ${EXTRAPORT2}
update add dot-fsmt-auth-bad.example.nil. 600 A 10.10.10.3
send
END
```
```
setup_system()
Creating key...
Creating key...
namefromtext
keycreate
reset_system()
user_interaction()
do_next_command()
do_next_command()
evaluate_update()
update_addordelete()
do_next_command()
start_update()
dns_request_create: TLS error
=================================================================
==26697==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 16 byte(s) in 1 object(s) allocated from:
#0 0x55ad0ab4e3de in malloc (/builds/isc-projects/bind9/bin/nsupdate/.libs/nsupdate+0xc93de) (BuildId: cb6f56478f03cc1cdba7bb8d05bbc4bef756b573)
#1 0x7f43ee89a115 in mallocx /builds/isc-projects/bind9/lib/isc/./jemalloc_shim.h:65:14
#2 0x7f43ee89a115 in mem_get /builds/isc-projects/bind9/lib/isc/mem.c:304:8
#3 0x7f43ee899e31 in isc__mem_get /builds/isc-projects/bind9/lib/isc/mem.c:667:8
#4 0x55ad0ab9e371 in sendrequest /builds/isc-projects/bind9/bin/nsupdate/nsupdate.c:2947:12
#5 0x55ad0ab8f770 in start_update /builds/isc-projects/bind9/bin/nsupdate/nsupdate.c:3405:2
#6 0x55ad0ab8f770 in getinput /builds/isc-projects/bind9/bin/nsupdate/nsupdate.c:3499:2
#7 0x7f43ee891721 in setup_jobs_cb /builds/isc-projects/bind9/lib/isc/loop.c:255:3
#8 0x7f43ee80a13f in isc__async_cb /builds/isc-projects/bind9/lib/isc/async.c:84:3
#9 0x7f43ecd08e92 in uv__async_io /usr/src/libuv-v1.44.1/src/unix/async.c:163:5
#10 0x7f43ecd24bc2 in uv__io_poll /usr/src/libuv-v1.44.1/src/unix/epoll.c:374:11
#11 0x7f43ecd098ad in uv_run /usr/src/libuv-v1.44.1/src/unix/core.c:391:5
#12 0x7f43ee88bc6a in loop_run /builds/isc-projects/bind9/lib/isc/loop.c:272:6
#13 0x7f43ee88bc6a in loop_thread /builds/isc-projects/bind9/lib/isc/loop.c:299:2
#14 0x7f43ee88b8a7 in isc_loopmgr_run /builds/isc-projects/bind9/lib/isc/loop.c:473:2
#15 0x55ad0ab8afcb in main /builds/isc-projects/bind9/bin/nsupdate/nsupdate.c:3536:2
#16 0x7f43ecd74d09 in __libc_start_main csu/../csu/libc-start.c:308:16
SUMMARY: AddressSanitizer: 16 byte(s) leaked in 1 allocation(s).
```June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Michal NowakMichal Nowakhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4022Three "dnssec" failures on Fedora 382023-05-18T13:54:06ZMichal NowakThree "dnssec" failures on Fedora 38The `dnssec` system test [fails](https://gitlab.isc.org/isc-projects/bind9/-/pipelines/134795) in three tests on Fedora 38. Supposedly, there were [no crypto changes in Fedora 38](https://fedoraproject.org/wiki/Releases/38/ChangeSet).
#...The `dnssec` system test [fails](https://gitlab.isc.org/isc-projects/bind9/-/pipelines/134795) in three tests on Fedora 38. Supposedly, there were [no crypto changes in Fedora 38](https://fedoraproject.org/wiki/Releases/38/ChangeSet).
# 1
```
I:dnssec: check that 'dnssec-signzone -F' failed with disallowed algorithm (110)
I:dnssec:failed
```
The test expects `fatal: dnskey 'example.com/RSASHA1/19857' failed to sign data` but gets `dnssec-signzone: fatal: No signing keys specified or found.` although `dnssec/signer/general/Kexample.com.+005+19857.{key,private}` files are present.
# 2
```
I:dnssec:check that 'dnssec-keygen -F' disables rsasha1 (232)
I:dnssec:failed
```
The test expects `unsupported algorithm: RSASHA1` but gets `dnssec-keygen: fatal: unsupported algorithm: rsasha1`.
# 3
```
I:dnssec:check that 'dnssec-keygen -F' disables nsec3rsasha1 (233)
I:dnssec:failed
```
The test expects `unsupported algorithm: NSEC3RSASHA1` but gets `dnssec-keygen: fatal: unsupported algorithm: nsec3rsasha1`.
For issues **2** and **3**, matching case-insentively with `-i` is enough.June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)https://gitlab.isc.org/isc-projects/bind9/-/issues/4019CID 452208: locking issue in tests/bench/qpmulti.c2023-05-12T20:43:31ZMichal NowakCID 452208: locking issue in tests/bench/qpmulti.cCoverity Scan claims a locking issue in a test code rooted in !7582.
```
/tests/bench/qpmulti.c: 296 in read_transactions()
290 if (result == ISC_R_SUCCESS) {
291 args->present++;
292 } else {
293 args->abs...Coverity Scan claims a locking issue in a test code rooted in !7582.
```
/tests/bench/qpmulti.c: 296 in read_transactions()
290 if (result == ISC_R_SUCCESS) {
291 args->present++;
292 } else {
293 args->absent++;
294 }
295 }
>>> CID 452208: API usage errors (LOCK)
>>> "dns_qpread_destroy" unlocks "args->multi->mutex" while it is unlocked.
296 dns_qpread_destroy(args->multi, &qp);
297 }
298 next_loop(args, start);
299 }
300
301 static void
```June 2023 (9.16.42, 9.16.42-S1, 9.18.16, 9.18.16-S1, 9.19.14)Tony FinchTony Finch