BIND issueshttps://gitlab.isc.org/isc-projects/bind9/-/issues2022-05-19T10:23:56Zhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3350Document the time format of rndc dnssec -checkds -when <time>2022-05-19T10:23:56ZJan "Yenya" KasprzakDocument the time format of rndc dnssec -checkds -when <time>### Description
Setting the actual time of DS publication or withdrawal using
rndc dnssec -checkds -when <time> published <domain>
is documented only with this sentence in the rndc(8) manual page:
The time that the DS has bee...### Description
Setting the actual time of DS publication or withdrawal using
rndc dnssec -checkds -when <time> published <domain>
is documented only with this sentence in the rndc(8) manual page:
The time that the DS has been published or
withdrawn is set to now, unless otherwise specified with the ar‐
gument -when time.
But the actual time format accepted is apparently not documented anywhere.
I tried to use the same format rndc prints when the -checkds command is executed
without -when parameter, but it did not work either:
$ rndc dnssec -checkds published my.dom.ain
Marked DS as published since 13-May-2022 10:44:05.000
$ rndc dnssec -checkds -when '13-May-2022 10:44:05.000' published my.dom.ain
rndc: 'dnssec' failed: syntax error
I tried several other time formats, but was not able to figure out which time format
is accepted by the -when switch. Also, the "syntax error" message is not very helpful there.
This is rndc from BIND 9.16.27.
### Request
Please document the time format(s) accepted by the -when switch, and maybe improve
the diagnostics message printed when some of the switches have invalid arguments.
Thanks!June 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)https://gitlab.isc.org/isc-projects/bind9/-/issues/3349"managed-keys" zone is created even with "dnssec-validation no;"2022-08-09T15:23:43ZMichał Kępień"managed-keys" zone is created even with "dnssec-validation no;"With the following configuration file:
```
options {
dnssec-validation no;
};
```
`named` logs the following lines upon startup:
```
13-May-2022 07:06:05.967 set up managed keys zone for view _default, file 'managed-keys.bind'
13-May...With the following configuration file:
```
options {
dnssec-validation no;
};
```
`named` logs the following lines upon startup:
```
13-May-2022 07:06:05.967 set up managed keys zone for view _default, file 'managed-keys.bind'
13-May-2022 07:06:05.971 managed-keys-zone: loaded serial 0
```
and indeed creates a zone attached to the `_default` view which [only
contains a SOA record][1].
The same thing happens for `dnssec-validation yes;`.
This is harmless, but confusing at best, for two reasons:
1. The `managed-keys.bind` file is never written to, despite what the
log message suggests.
2. No managed keys are ever refreshed or used.
Introduced by commit 778a01b1aa76273d4a28c7559a509edc7a00ec95, i.e. in
BIND 9.7.1.
See also #1961 for more logging confusion related to RFC 5011.
[1]: https://gitlab.isc.org/isc-projects/bind9/-/blob/8fbb75e4571bd3a9595977915279f8ffc9f40c89/lib/dns/zone.c#L4930-4940June 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)https://gitlab.isc.org/isc-projects/bind9/-/issues/3347ipv4only.arpa instantiation should not be disabled forward only declarations ...2022-05-11T22:18:17ZMark Andrewsipv4only.arpa instantiation should not be disabled forward only declarations without forwardersJune 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)https://gitlab.isc.org/isc-projects/bind9/-/issues/3346Empty-zones should not be disabled forward only declarations without forwarders2022-05-11T22:20:00ZMark AndrewsEmpty-zones should not be disabled forward only declarations without forwardersJune 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)https://gitlab.isc.org/isc-projects/bind9/-/issues/3345Insecurity proof failed resolving 'a.b.keyless.example/A/IN' in dnssec test2022-06-01T00:42:57ZMichal NowakInsecurity proof failed resolving 'a.b.keyless.example/A/IN' in dnssec test`dnssec:checking that validation fails when key record is missing using dns_client` test [failed](https://gitlab.isc.org/isc-private/bind9/-/jobs/2502557) on `v9_16`.
In `delv.out71` broken trust chain was expected, but failed insecurit...`dnssec:checking that validation fails when key record is missing using dns_client` test [failed](https://gitlab.isc.org/isc-private/bind9/-/jobs/2502557) on `v9_16`.
In `delv.out71` broken trust chain was expected, but failed insecurity proof was found instead:
```
;; validating a.b.keyless.example/A: no valid signature found
;; insecurity proof failed resolving 'a.b.keyless.example/A/IN': 10.53.0.4#5000
;; resolution failed: insecurity proof failed
```
`ns4/named.run`:
```
10-May-2022 09:45:22.249 validating a.b.keyless.example/DS: validate_neg_rrset: creating validator for a.b.keyless.example NSEC
10-May-2022 09:45:22.249 validating a.b.keyless.example/NSEC: starting
10-May-2022 09:45:22.249 validating a.b.keyless.example/NSEC: attempting positive response validation
10-May-2022 09:45:22.249 validating a.b.keyless.example/NSEC: keyset with trust secure
10-May-2022 09:45:22.249 validating a.b.keyless.example/NSEC: verify rdataset (keyid=40810): success
10-May-2022 09:45:22.249 validating a.b.keyless.example/NSEC: marking as secure, noqname proof not needed
10-May-2022 09:45:22.249 validator @0x61d0001f4a90: dns_validator_destroy
```
This is how it looks like when the test passes:
`delv.out71`:
```
;; validating a.b.keyless.example/A: no valid signature found
;; validating a.b.keyless.example/NSEC: no valid signature found
;; no valid RRSIG resolving 'a.b.keyless.example/DS/IN': 10.53.0.4#5300
;; broken trust chain resolving 'a.b.keyless.example/A/IN': 10.53.0.4#5300
;; resolution failed: broken trust chain
```
`ns4/named.run`:
```
10-May-2022 16:54:57.976 validating a.b.keyless.example/DS: validate_neg_rrset: creating validator for a.b.keyless.example NSEC
10-May-2022 16:54:57.976 validating a.b.keyless.example/NSEC: starting
10-May-2022 16:54:57.976 validating a.b.keyless.example/NSEC: attempting positive response validation
10-May-2022 16:54:57.976 validating a.b.keyless.example/NSEC: no valid signature found
10-May-2022 16:54:57.976 validating a.b.keyless.example/NSEC: falling back to insecurity proof
10-May-2022 16:54:57.976 validating a.b.keyless.example/NSEC: checking existence of DS at 'example'
10-May-2022 16:54:57.976 validating a.b.keyless.example/NSEC: checking existence of DS at 'keyless.example'
10-May-2022 16:54:57.976 validating a.b.keyless.example/NSEC: checking existence of DS at 'b.keyless.example'
10-May-2022 16:54:57.976 validating a.b.keyless.example/NSEC: checking existence of DS at 'a.b.keyless.example'
10-May-2022 16:54:57.976 validating a.b.keyless.example/NSEC: continuing validation would lead to deadlock: aborting validation
10-May-2022 16:54:57.976 validating a.b.keyless.example/NSEC: deadlock found (create_fetch)
10-May-2022 16:54:57.976 validator @0x7fbff0026ae0: dns_validator_destroy
10-May-2022 16:54:57.976 validating a.b.keyless.example/DS: in validator_callback_nsec
10-May-2022 16:54:57.976 validating a.b.keyless.example/DS: validator_callback_nsec: got no valid RRSIG
10-May-2022 16:54:57.976 validating a.b.keyless.example/DS: resuming validate_nx
10-May-2022 16:54:57.976 validating a.b.keyless.example/DS: nonexistence proof(s) not found
10-May-2022 16:54:57.976 validating a.b.keyless.example/DS: checking existence of DS at 'example'
10-May-2022 16:54:57.976 validating a.b.keyless.example/DS: checking existence of DS at 'keyless.example'
10-May-2022 16:54:57.976 validating a.b.keyless.example/DS: checking existence of DS at 'b.keyless.example'
10-May-2022 16:54:57.976 validating a.b.keyless.example/DS: checking existence of DS at 'a.b.keyless.example'
10-May-2022 16:54:57.976 validating a.b.keyless.example/DS: continuing validation would lead to deadlock: aborting validation
10-May-2022 16:54:57.976 validating a.b.keyless.example/DS: deadlock found (create_fetch)
10-May-2022 16:54:57.976 validator @0x7fbff0024610: dns_validator_destroy
10-May-2022 16:54:57.976 no valid RRSIG resolving 'a.b.keyless.example/DS/IN': 10.53.0.3#5300
```June 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)https://gitlab.isc.org/isc-projects/bind9/-/issues/3344ThreadSanitizer: data race lib/isc/netmgr/tcpdns.c:1091:12 in isc__nm_tcpdns_...2022-05-30T12:17:58ZMichal NowakThreadSanitizer: data race lib/isc/netmgr/tcpdns.c:1091:12 in isc__nm_tcpdns_sendJob [#2502181](https://gitlab.isc.org/isc-private/bind9/-/jobs/2502181) failed for 4df5a54e9b61aa6fd8b9dad3adaa1d39340ae62b on the `v9_16_sub` branch, which is basically 9.16.28-S1.
```
WARNING: ThreadSanitizer: data race
Read of size...Job [#2502181](https://gitlab.isc.org/isc-private/bind9/-/jobs/2502181) failed for 4df5a54e9b61aa6fd8b9dad3adaa1d39340ae62b on the `v9_16_sub` branch, which is basically 9.16.28-S1.
```
WARNING: ThreadSanitizer: data race
Read of size 8 at 0x000000000001 by thread T1:
#0 isc__nm_tcpdns_send lib/isc/netmgr/tcpdns.c:1091:12
#1 isc_nm_send lib/isc/netmgr/netmgr.c:2438:3
#2 client_sendpkg lib/ns/client.c:355:2
#3 ns_client_send lib/ns/client.c:697:3
#4 query_send lib/ns/query.c:584:2
#5 ns_query_done lib/ns/query.c:11959:2
#6 query_respond lib/ns/query.c:8435:10
#7 query_prepresponse lib/ns/query.c:10969:10
#8 query_gotanswer lib/ns/query.c
#9 query_resume lib/ns/query.c:6886:10
#10 fetch_callback lib/ns/query.c:6442:12
#11 task_run lib/isc/task.c:851:5
#12 isc_task_run lib/isc/task.c:944:10
#13 process_netievent lib/isc/netmgr/netmgr.c
#14 process_queue lib/isc/netmgr/netmgr.c:1040:16
#15 process_all_queues lib/isc/netmgr/netmgr.c:814:25
#16 async_cb lib/isc/netmgr/netmgr.c:842:6
#17 uv__async_io /usr/src/libuv-v1.43.0/src/unix/async.c:163:5
#18 isc__trampoline_run lib/isc/trampoline.c:209:11
Previous write of size 8 at 0x000000000001 by thread T2:
#0 isc__nm_tcpdns_send lib/isc/netmgr/tcpdns.c:1092:23
#1 isc_nm_send lib/isc/netmgr/netmgr.c:2438:3
#2 client_sendpkg lib/ns/client.c:355:2
#3 ns_client_send lib/ns/client.c:697:3
#4 query_send lib/ns/query.c:584:2
#5 ns_query_done lib/ns/query.c:11959:2
#6 query_respond lib/ns/query.c:8435:10
#7 query_prepresponse lib/ns/query.c:10969:10
#8 query_gotanswer lib/ns/query.c
#9 query_resume lib/ns/query.c:6886:10
#10 fetch_callback lib/ns/query.c:6442:12
#11 task_run lib/isc/task.c:851:5
#12 isc_task_run lib/isc/task.c:944:10
#13 process_netievent lib/isc/netmgr/netmgr.c
#14 process_queue lib/isc/netmgr/netmgr.c:1040:16
#15 process_all_queues lib/isc/netmgr/netmgr.c:814:25
#16 async_cb lib/isc/netmgr/netmgr.c:842:6
#17 uv__async_io /usr/src/libuv-v1.43.0/src/unix/async.c:163:5
#18 isc__trampoline_run lib/isc/trampoline.c:209:11
Location is heap block of size 1361 at 0x000000000018 allocated by thread T3:
#0 malloc <null>
#1 default_memalloc lib/isc/mem.c:715:8
#2 mem_get lib/isc/mem.c:624:8
#3 mem_allocateunlocked lib/isc/mem.c:1289:8
#4 isc___mem_allocate lib/isc/mem.c:1309:7
#5 isc__mem_allocate lib/isc/mem.c:2399:10
#6 isc___mem_get lib/isc/mem.c:1059:11
#7 isc__mem_get lib/isc/mem.c:2378:10
#8 accept_connection lib/isc/netmgr/tcpdns.c:956:10
#9 tcpdns_connection_cb lib/isc/netmgr/tcpdns.c:624:11
#10 uv__server_io /usr/src/libuv-v1.43.0/src/unix/stream.c:570:5
#11 isc__trampoline_run lib/isc/trampoline.c:209:11
Thread T1 (running) created by main thread at:
#0 pthread_create <null>
#1 isc_thread_create lib/isc/pthreads/thread.c:81:8
#2 isc__netmgr_create lib/isc/netmgr/netmgr.c:370:3
#3 isc_managers_create lib/isc/managers.c:35:2
#4 create_managers bin/named/./main.c:931:11
#5 setup bin/named/./main.c:1256:11
#6 main bin/named/./main.c:1576:2
Thread T2 (running) created by main thread at:
#0 pthread_create <null>
#1 isc_thread_create lib/isc/pthreads/thread.c:81:8
#2 isc__netmgr_create lib/isc/netmgr/netmgr.c:370:3
#3 isc_managers_create lib/isc/managers.c:35:2
#4 create_managers bin/named/./main.c:931:11
#5 setup bin/named/./main.c:1256:11
#6 main bin/named/./main.c:1576:2
Thread T3 (running) created by main thread at:
#0 pthread_create <null>
#1 isc_thread_create lib/isc/pthreads/thread.c:81:8
#2 isc__netmgr_create lib/isc/netmgr/netmgr.c:370:3
#3 isc_managers_create lib/isc/managers.c:35:2
#4 create_managers bin/named/./main.c:931:11
#5 setup bin/named/./main.c:1256:11
#6 main bin/named/./main.c:1576:2
SUMMARY: ThreadSanitizer: data race lib/isc/netmgr/tcpdns.c:1091:12 in isc__nm_tcpdns_send
```June 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)https://gitlab.isc.org/isc-projects/bind9/-/issues/3341Extended DNS errors timing issue in serve-stale system test2023-07-11T10:00:46ZMichal NowakExtended DNS errors timing issue in serve-stale system test`serve-stale:check data.example (max-stale-ttl default)` test [failed](https://gitlab.isc.org/isc-private/bind9/-/jobs/2501631) because `dig` received `EDE: 3 (Stale Answer): (query within stale refresh time window)` instead of the expec...`serve-stale:check data.example (max-stale-ttl default)` test [failed](https://gitlab.isc.org/isc-private/bind9/-/jobs/2501631) because `dig` received `EDE: 3 (Stale Answer): (query within stale refresh time window)` instead of the expected `EDE: 3 (Stale Answer): (resolver failure)` response; looks like a timing issue on a system under pressure.June 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/3335Confusing parental-source documentation2022-05-16T11:54:19ZMatthijs Mekkingmatthijs@isc.orgConfusing parental-source documentationparental-source determines which local source address, and optionally UDP port, is used to send parental DS queries. **This address must appear in the secondary server’s parental-agents zone clause.** This statement sets the parental-sou...parental-source determines which local source address, and optionally UDP port, is used to send parental DS queries. **This address must appear in the secondary server’s parental-agents zone clause.** This statement sets the parental-source for all zones, but can be overridden on a per-zone or per-view basis by including a parental-source statement within the zone or view block in the configuration file.
The bold line is a copy paste error from `notify-source`.June 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/3332[ISC-support #20671] DNS_EDNSOPTIONS macro2022-05-06T07:58:31ZEverett Fulton[ISC-support #20671] DNS_EDNSOPTIONS macrohttps://support.isc.org/Ticket/Display.html?id=20671
A support customer has reported a potential issue with the value defined for DNS_EDNSOPTIONS, which is used in several INSISTs. From the Support ticket:
-----
In BIND 9.16.23, DNS_E...https://support.isc.org/Ticket/Display.html?id=20671
A support customer has reported a potential issue with the value defined for DNS_EDNSOPTIONS, which is used in several INSISTs. From the Support ticket:
-----
In BIND 9.16.23, DNS_EDNSOPTIONS is defined in
lib/dns/include/dns/message.h as:
> /*%< The number of EDNS options we know about. */
> #define DNS_EDNSOPTIONS 8
whereas message.h lists more options:
> /*%< EDNS0 extended OPT codes */
> #define DNS_OPT_LLQ 1 /*%< LLQ opt code */
> #define DNS_OPT_NSID 3 /*%< NSID opt code */
> #define DNS_OPT_CLIENT_SUBNET 8 /*%< client subnet opt code */
> #define DNS_OPT_EXPIRE 9 /*%< EXPIRE opt code */
> #define DNS_OPT_COOKIE 10 /*%< COOKIE opt code */
> #define DNS_OPT_TCP_KEEPALIVE 11 /*%< TCP keepalive opt code */
> #define DNS_OPT_PAD 12 /*%< PAD opt code */
> #define DNS_OPT_KEY_TAG 14 /*%< Key tag opt code */
> #define DNS_OPT_EDE 15 /*%< Extended DNS Error opt code */
> #define DNS_OPT_CLIENT_TAG 16 /*%< Client tag opt code */
> #define DNS_OPT_SERVER_TAG 17 /*%< Server tag opt code */
>
> #define DNS_OPT_PROTOSS 20292 /*%< Cisco/OpenDNS umbrella */
The DNS_EDNSOPTIONS macro is used in assertions in mutiple places. While
the current usage may not be vulnerable (I haven't checked every case),
the macro value probably has to be adjusted.
-----
9.16.23 actually shows 7, but this was later changed to 8.
./lib/dns/include/dns/message.h:#define DNS_EDNSOPTIONS 7June 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/3327fetches-per-server has been broken since 9.162022-06-15T13:45:59ZEvan Huntfetches-per-server has been broken since 9.16While working on another issue I noticed the output of the `fetchlimit` system test didn't match my recollection of how it used to look. I've almost certainly seen this before, but it didn't rise to the level of consciousness that would ...While working on another issue I noticed the output of the `fetchlimit` system test didn't match my recollection of how it used to look. I've almost certainly seen this before, but it didn't rise to the level of consciousness that would prompt me to confirm, but this time it did, and sure enough, up until 9.15.7 the test output looked like this:
```
I:fetchlimit:checking recursing clients are dropped at the per-server limit
I:fetchlimit:clients: 20
I:fetchlimit:clients: 40
I:fetchlimit:clients: 60
I:fetchlimit:clients: 80
I:fetchlimit:clients: 100
I:fetchlimit:clients: 120
I:fetchlimit:clients: 140
I:fetchlimit:clients: 160
I:fetchlimit:clients: 180
I:fetchlimit:clients: 180
I:fetchlimit:clients: 180
I:fetchlimit:clients: 177
I:fetchlimit:clients: 165
I:fetchlimit:clients: 152
I:fetchlimit:clients: 141
I:fetchlimit:clients: 131
I:fetchlimit:clients: 122
I:fetchlimit:clients: 114
I:fetchlimit:clients: 114
I:fetchlimit:clients: 107
```
...but now it's this:
```
I:fetchlimit:checking recursing clients are dropped at the per-server limit
I:fetchlimit:clients: 20
I:fetchlimit:clients: 40
I:fetchlimit:clients: 60
I:fetchlimit:clients: 80
I:fetchlimit:clients: 28
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
```
This is due to a logic error introduced in `maybe_adjust_quota()` in commit bad5a523c2e: Instead of updating the quota to the calculated `new_quota`, it's updated to `ISC_MIN(1, new_quota)`, which effectively means always 1. The test was only checking that the number of clients was below the specified limit, not that it was above a reasonable minimum, and so the error went unnoticed.
I would guess this has probably had some impact on recursive performance.June 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)Evan HuntEvan Hunthttps://gitlab.isc.org/isc-projects/bind9/-/issues/3302Key files are updated every keymgr run2022-05-17T02:51:44ZMatthijs Mekkingmatthijs@isc.orgKey files are updated every keymgr runAll the key related files are touched by BIND on every keymgr run, whether they are modified or not. If the keymgr is waiting for an event, it runs every "refresh key interval", which defaults to an hour.
Fix this by checking for modifi...All the key related files are touched by BIND on every keymgr run, whether they are modified or not. If the keymgr is waiting for an event, it runs every "refresh key interval", which defaults to an hour.
Fix this by checking for modification of keys and only write out keys if they were modified.June 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/326932-bit named sometimes fails to start in doth test2023-03-28T14:42:19ZMichal Nowak32-bit named sometimes fails to start in doth testJob [#2440515](https://gitlab.isc.org/isc-private/bind9/-/jobs/2440515) failed for cab15392afd841fe3b0bacd894003376d857459a:
```
S:doth:2022-04-11T08:39:50+0000
T:doth:1:A
A:doth:System test doth
I:doth:PORTS:30179,30180,30181,30182,3018...Job [#2440515](https://gitlab.isc.org/isc-private/bind9/-/jobs/2440515) failed for cab15392afd841fe3b0bacd894003376d857459a:
```
S:doth:2022-04-11T08:39:50+0000
T:doth:1:A
A:doth:System test doth
I:doth:PORTS:30179,30180,30181,30182,30183,30184,30185,30186,30187,30188,30189,30190,30191
I:doth:starting servers
I:doth:Couldn't start server /builds/isc-private/bind9/bin/named/named -D doth-ns1 -X named.lock -m record -c named.conf -d 99 -g -U 4 -T maxcachesize=2097152 >named.run 2>&1 & echo $! (pid=1600)
I:doth:failed
I:doth:ns1 didn't die when sent a SIGTERM
I:doth:starting servers failed
I:doth:No tests were found and run
I:doth:Core dump(s) found: doth/ns1/core.1600
D:doth:backtrace from doth/ns1/core.1600:
D:doth:--------------------------------------------------------------------------------
D:doth:Core was generated by `/builds/isc-private/bind9/bin/named/.libs/named -D doth-ns1 -X named.lock -m re'.
D:doth:Program terminated with signal SIGABRT, Aborted.
D:doth:#0 0xf7f98069 in __kernel_vsyscall ()
D:doth:[Current thread is 1 (Thread 0xf4f384c0 (LWP 1600))]
D:doth:#0 0xf7f98069 in __kernel_vsyscall ()
D:doth:#1 0xf72b4cd2 in sigtimedwait () from /lib/i386-linux-gnu/libc.so.6
D:doth:#2 0xf747c13d in sigwait () from /lib/i386-linux-gnu/libpthread.so.0
D:doth:#3 0xf7cba3e4 in isc_app_ctxrun (ctx=<optimized out>) at app.c:236
D:doth:#4 0xf7cba57b in isc_app_run () at app.c:288
D:doth:#5 0x5663b81a in main (argc=16, argv=0xff99aa34) at main.c:1459
D:doth:--------------------------------------------------------------------------------
D:doth:full backtrace from doth/ns1/core.1600 saved in doth/ns1/core.1600-backtrace.txt
D:doth:core dump doth/ns1/core.1600 archived as doth/ns1/core.1600.gz
R:doth:FAIL
E:doth:2022-04-11T08:41:53+0000
FAIL doth (exit status: 1)
```
Lines about `named` listening on interfaces is missing (ns1 `named.run`):
```
11-Apr-2022 08:39:50.952 starting BIND 9.19.0 (Development Release) <id:cab1539>
11-Apr-2022 08:39:50.952 running on Linux x86_64 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07)
11-Apr-2022 08:39:50.952 built with '--disable-maintainer-mode' '--enable-developer' '--enable-option-checking=fatal' '--enable-dnstap' '--with-cmocka' '--with-libxml2' '--with-json-c' '--build=x86_64-linux-gnu' '--host=i686-linux-gnu' '--with-libidn2' '--with-readline=libedit' 'build_alias=x86_64-linux-gnu' 'host_alias=i686-linux-gnu' 'CFLAGS=-fno-omit-frame-pointer -fno-optimize-sibling-calls -O1 -g -Wall -Wextra'
11-Apr-2022 08:39:50.952 running as: named -D doth-ns1 -X named.lock -m record -c named.conf -d 99 -g -U 4 -T maxcachesize\=2097152
11-Apr-2022 08:39:50.952 compiled by GCC 10.2.1 20210110
11-Apr-2022 08:39:50.952 compiled with OpenSSL version: OpenSSL 1.1.1k 25 Mar 2021
11-Apr-2022 08:39:50.952 linked to OpenSSL version: OpenSSL 1.1.1k 25 Mar 2021
11-Apr-2022 08:39:50.952 compiled with libxml2 version: 2.9.10
11-Apr-2022 08:39:50.952 linked to libxml2 version: 20910
11-Apr-2022 08:39:50.952 compiled with json-c version: 0.15
11-Apr-2022 08:39:50.952 linked to json-c version: 0.15
11-Apr-2022 08:39:50.952 compiled with zlib version: 1.2.11
11-Apr-2022 08:39:50.952 linked to zlib version: 1.2.11
11-Apr-2022 08:39:50.952 ----------------------------------------------------
11-Apr-2022 08:39:50.952 BIND 9 is maintained by Internet Systems Consortium,
11-Apr-2022 08:39:50.952 Inc. (ISC), a non-profit 501(c)(3) public-benefit
11-Apr-2022 08:39:50.952 corporation. Support and training for BIND 9 are
11-Apr-2022 08:39:50.952 available at https://www.isc.org/support
11-Apr-2022 08:39:50.952 ----------------------------------------------------
11-Apr-2022 08:39:50.952 found 12 CPUs, using 12 worker threads
11-Apr-2022 08:39:50.952 using 4 UDP listeners per interface
11-Apr-2022 08:39:50.960 Registering DLZ_dlopen driver
11-Apr-2022 08:39:50.960 Registering SDLZ driver 'dlopen'
11-Apr-2022 08:39:50.960 Registering DLZ driver 'dlopen'
11-Apr-2022 08:39:50.980 config.c: option 'trust-anchor-telemetry' is experimental and subject to change in the future
11-Apr-2022 08:39:50.980 loading configuration from '/builds/isc-private/bind9/bin/tests/system/doth/ns1/named.conf'
11-Apr-2022 08:39:50.980 unable to open '/usr/local/etc/bind.keys'; using built-in keys instead
11-Apr-2022 08:39:50.980 exclusive task mode: starting
11-Apr-2022 08:39:51.004 exclusive task mode: started
11-Apr-2022 08:39:51.004 set maximum stack size to 18446744073709551615: success
11-Apr-2022 08:39:51.004 set maximum data size to 18446744073709551615: success
11-Apr-2022 08:39:51.004 set maximum core size to 18446744073709551615: success
11-Apr-2022 08:39:51.004 set maximum open files to 18446744073709551615: success
11-Apr-2022 08:39:51.004 looking for GeoIP2 databases in '/usr/share/GeoIP'
11-Apr-2022 08:39:51.004 unable to open GeoIP2 database '/usr/share/GeoIP/GeoIP2-Country.mmdb' (status 1)
11-Apr-2022 08:39:51.004 unable to open GeoIP2 database '/usr/share/GeoIP/GeoLite2-Country.mmdb' (status 1)
11-Apr-2022 08:39:51.004 unable to open GeoIP2 database '/usr/share/GeoIP/GeoIP2-City.mmdb' (status 1)
11-Apr-2022 08:39:51.004 unable to open GeoIP2 database '/usr/share/GeoIP/GeoLite2-City.mmdb' (status 1)
11-Apr-2022 08:39:51.004 unable to open GeoIP2 database '/usr/share/GeoIP/GeoIP2-ASN.mmdb' (status 1)
11-Apr-2022 08:39:51.004 unable to open GeoIP2 database '/usr/share/GeoIP/GeoLite2-ASN.mmdb' (status 1)
11-Apr-2022 08:39:51.004 unable to open GeoIP2 database '/usr/share/GeoIP/GeoIP2-ISP.mmdb' (status 1)
11-Apr-2022 08:39:51.004 unable to open GeoIP2 database '/usr/share/GeoIP/GeoIP2-Domain.mmdb' (status 1)
11-Apr-2022 08:39:51.004 using default UDP/IPv4 port range: [32768, 60999]
11-Apr-2022 08:39:51.004 using default UDP/IPv6 port range: [32768, 60999]
```
Happened a day before in the [2439503](https://gitlab.isc.org/isc-projects/bind9/-/jobs/2439503) job, again `doth` system test.
Note that this job employs 32-bit BIND with 32-bit dependencies on otherwise 64-bit system.
[core.1600-backtrace.txt](/uploads/b99233defa71f1ab35bdc5c021447cc4/core.1600-backtrace.txt)June 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)Artem BoldarievArtem Boldarievhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3064Disable the interface-interval timer on systems where route/netlink messages ...2022-05-30T10:44:26ZArtem BoldarievDisable the interface-interval timer on systems where route/netlink messages workThis issue is essentially a follow-up from #3059 and !5638.
The idea is that there is no point in having `interface-interval` on the systems where a kernel reliably informs us about the state of network interfaces. On such systems, havi...This issue is essentially a follow-up from #3059 and !5638.
The idea is that there is no point in having `interface-interval` on the systems where a kernel reliably informs us about the state of network interfaces. On such systems, having `interface-interval` is redundant. At the very least the list of such systems includes all Linux distributions, where NetLink messages are available for that.
We may wish to disable the timer with a `WARNING` message in the log earlier during 9.19 release series to see how well it will work on practice (it should work just fine, IMO).June 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)Artem BoldarievArtem Boldarievhttps://gitlab.isc.org/isc-projects/bind9/-/issues/2801unit test netmgr_test of 9.17.15 fails reliably on s390x2022-05-30T10:14:14ZPetr Menšíkunit test netmgr_test of 9.17.15 fails reliably on s390x<!--
If the bug you are reporting is potentially security-related - for example,
if it involves an assertion failure or other crash in `named` that can be
triggered repeatedly - then please do *NOT* report it here, but send an
email to [...<!--
If the bug you are reporting is potentially security-related - for example,
if it involves an assertion failure or other crash in `named` that can be
triggered repeatedly - then please do *NOT* report it here, but send an
email to [security-officer@isc.org](security-officer@isc.org).
-->
### Summary
netmgr_test fails always on 9.17.15 builds
### BIND version used
```
BIND 9.17.15 (Development Release) <id:a3a1875>
running on Linux s390x 5.13.0-0.rc7.20210624git7426cedc7dad.54.fc35.s390x #1 SMP Thu Jun 24 15:11:21 UTC 2021
built by make with '--build=s390x-ibm-linux-gnu' '--host=s390x-ibm-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--localstatedir=/var' '--with-pic' '--disable-static' '--includedir=/usr/include/bind9' '--with-tuning=large' '--with-libidn2' '--with-maxminddb' '--with-gssapi=yes' '--with-lmdb=yes' '--with-json-c' '--enable-dnstap' '--with-cmocka' '--enable-fixed-rrset' '--enable-full-report' 'build_alias=s390x-ibm-linux-gnu' 'host_alias=s390x-ibm-linux-gnu' 'CC=gcc' 'CFLAGS= -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=zEC12 -mtune=z13 -fasynchronous-unwind-tables -fstack-clash-protection' 'LDFLAGS=-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld ' 'LT_SYS_LIBRARY_PATH=/usr/lib64:' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'
compiled by GCC 11.1.1 20210623 (Red Hat 11.1.1-6)
compiled with OpenSSL version: OpenSSL 1.1.1k FIPS 25 Mar 2021
linked to OpenSSL version: OpenSSL 1.1.1k FIPS 25 Mar 2021
compiled with libuv version: 1.41.0
linked to libuv version: 1.41.0
compiled with libnghttp2 version: 1.43.0
linked to libnghttp2 version: 1.43.0
compiled with libxml2 version: 2.9.12
linked to libxml2 version: 20912
compiled with json-c version: 0.14
linked to json-c version: 0.14
compiled with zlib version: 1.2.11
linked to zlib version: 1.2.11
linked to maxminddb version: 1.5.2
compiled with protobuf-c version: 1.3.3
linked to protobuf-c version: 1.3.3
threads support is enabled
default paths:
named configuration: /etc/named.conf
rndc configuration: /etc/rndc.conf
DNSSEC root key: /etc/bind.keys
nsupdate session key: /var/run/named/session.key
named PID file: /var/run/named/named.pid
named lock file: /var/run/named/named.lock
geoip-directory: /usr/share/GeoIP
```
### Steps to reproduce
- build 9.17.15 on Fedora rawhide, arch s390x (fails on f34 too)
- make unit
What I have used:
- git clone https://src.fedoraproject.org/forks/pemensik/rpms/bind.git
- cd bind
- git checkout v9_17
- fedpkg builddep bind.spec
- fedpkg --release rawhide local
- or fedpkg --release rawhide scratch-build --arch s390x --srpm
### What is the current *bug* behavior?
```
make[5]: Entering directory '/builddir/build/BUILD/bind-9.17.15/build/lib/isc/tests'
PASS: aes_test
PASS: buffer_test
PASS: counter_test
PASS: crc64_test
PASS: doh_test
PASS: errno_test
PASS: file_test
PASS: hash_test
PASS: heap_test
PASS: hmac_test
PASS: ht_test
PASS: lex_test
PASS: md_test
PASS: mem_test
PASS: netaddr_test
FAIL: netmgr_test
PASS: parse_test
PASS: pool_test
PASS: quota_test
PASS: radix_test
PASS: random_test
PASS: regex_test
PASS: result_test
PASS: safe_test
PASS: siphash_test
PASS: sockaddr_test
PASS: socket_test
PASS: symtab_test
PASS: task_test
PASS: taskpool_test
PASS: time_test
PASS: timer_test
============================================================================
Testsuite summary for BIND 9.17.15
============================================================================
# TOTAL: 32
# PASS: 31
# SKIP: 0
# XFAIL: 0
# FAIL: 1
# XPASS: 0
# ERROR: 0
============================================================================
See lib/isc/tests/test-suite.log
Please report to info@isc.org
============================================================================``
```
### What is the expected *correct* behavior?
All PASS, FAIL: 0
### Relevant configuration files
Used [bind.spec](https://src.fedoraproject.org/fork/pemensik/rpms/bind/blob/v9_17/f/bind.spec).
### Relevant logs and/or screenshots
```
$ ./netmgr_test
[==========] Running 81 test(s).
[ RUN ] mock_listenudp_uv_udp_open
[ OK ] mock_listenudp_uv_udp_open
[ RUN ] mock_listenudp_uv_udp_bind
[ OK ] mock_listenudp_uv_udp_bind
[ RUN ] mock_listenudp_uv_udp_recv_start
[ OK ] mock_listenudp_uv_udp_recv_start
[ RUN ] mock_udpconnect_uv_udp_open
[ OK ] mock_udpconnect_uv_udp_open
[ RUN ] mock_udpconnect_uv_udp_bind
[ OK ] mock_udpconnect_uv_udp_bind
[ RUN ] mock_udpconnect_uv_udp_connect
[ OK ] mock_udpconnect_uv_udp_connect
[ RUN ] mock_udpconnect_uv_recv_buffer_size
[ OK ] mock_udpconnect_uv_recv_buffer_size
[ RUN ] mock_udpconnect_uv_send_buffer_size
[ OK ] mock_udpconnect_uv_send_buffer_size
[ RUN ] udp_noop
[ OK ] udp_noop
[ RUN ] udp_noresponse
[ OK ] udp_noresponse
[ RUN ] udp_timeout_recovery
[ OK ] udp_timeout_recovery
[ RUN ] udp_recv_one
$ echo $?
255
```
```
(gdb) bt
#0 __GI_exit (status=status@entry=-1) at exit.c:143
#1 0x000003fffde8296e in exit_test (quit_application=1) at /usr/src/debug/cmocka-1.1.5-9.fc35.s390x/src/cmocka.c:408
#2 0x000003fffde82a7a in _fail (file=file@entry=0x2aa00018c8c "../../../../lib/isc/tests/netmgr_test.c",
line=<optimized out>) at /usr/src/debug/cmocka-1.1.5-9.fc35.s390x/src/cmocka.c:2196
#3 0x000003fffde82b3c in _assert_true (result=0, line=<optimized out>,
file=0x2aa00018c8c "../../../../lib/isc/tests/netmgr_test.c",
expression=0x2aa00018f3a "region->length >= sizeof(magic)")
at /usr/src/debug/cmocka-1.1.5-9.fc35.s390x/src/cmocka.c:1730
#4 0x000002aa000078c2 in listen_read_cb (handle=<optimized out>, eresult=<optimized out>,
region=region@entry=0x3fffd0395c8, cbarg=0x0) at ../../../../lib/isc/tests/netmgr_test.c:537
#5 0x000003fffdda9b16 in isc__nm_async_readcb (worker=worker@entry=0x0, ev0=ev0@entry=0x3fffd039680)
at ../../../lib/isc/netmgr/netmgr.c:2739
#6 0x000003fffdda9c98 in isc__nm_readcb (sock=sock@entry=0x2aa00e5fd30, uvreq=<optimized out>, eresult=eresult@entry=0)
at ../../../lib/isc/netmgr/netmgr.c:2714
#7 0x000002aa00005cba in udp_recv_cb (handle=<optimized out>, nrecv=0, buf=0x3fffd0398b8, addr=0x3fffd039748,
flags=<optimized out>) at ../../../../lib/isc/tests/../netmgr/udp.c:420
#8 0x000003fffdd22cc2 in uv__udp_recvmsg (handle=0x2aa00e603f0) at src/unix/udp.c:304
#9 uv__udp_io (loop=<optimized out>, w=0x2aa00e60470, revents=<optimized out>) at src/unix/udp.c:180
#10 0x000003fffdd26b78 in uv__io_poll (loop=0x2aa004d68b0, timeout=<optimized out>) at src/unix/linux-core.c:462
#11 0x000003fffdd163e0 in uv_run (loop=loop@entry=0x2aa004d68b0, mode=mode@entry=UV_RUN_DEFAULT) at src/unix/core.c:385
#12 0x000003fffddad4f0 in nm_thread (worker0=0x2aa004d68a0) at ../../../lib/isc/netmgr/netmgr.c:746
#13 0x000003fffddf2696 in isc__trampoline_run (arg=0x2aa0057c850) at ../../../lib/isc/trampoline.c:184
#14 0x000003fffdb9a8a2 in start_thread (arg=<optimized out>) at pthread_create.c:429
#15 0x000003fffdc12d8e in thread_start () at ../sysdeps/unix/sysv/linux/s390/s390-64/clone.S:67
(gdb) frame 4
#4 0x000002aa000078c2 in listen_read_cb (handle=<optimized out>, eresult=<optimized out>,
region=region@entry=0x3fffd0395c8, cbarg=0x0) at ../../../../lib/isc/tests/netmgr_test.c:537
537 assert_true(region->length >= sizeof(magic));
(gdb) p region->length
$1 = 0
(gdb) p sizeof(magic)
$2 = 8
(gdb) p region
$3 = (isc_region_t *) 0x3fffd0395c8
(gdb) p *region
$4 = {base = 0x2aa0057d270 "", length = 0}
```
```
# cat /proc/cpuinfo
vendor_id : IBM/S390
# processors : 2
bogomips per cpu: 3033.00
max thread id : 0
features : esan3 zarch stfle msa ldisp eimm dfp edat etf3eh highgprs te vx sie
facilities : 0 1 2 3 4 6 7 8 9 10 12 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 30 31 32 33 34 35 36 37 40 41 42 43 44 45 46 47 48 49 50 51 52 53 55 57 73 74 75 76 77 80 81 82 128 129 131
cache0 : level=1 type=Data scope=Private size=128K line_size=256 associativity=8
cache1 : level=1 type=Instruction scope=Private size=96K line_size=256 associativity=6
cache2 : level=2 type=Data scope=Private size=2048K line_size=256 associativity=8
cache3 : level=2 type=Instruction scope=Private size=2048K line_size=256 associativity=8
cache4 : level=3 type=Unified scope=Shared size=65536K line_size=256 associativity=16
cache5 : level=4 type=Unified scope=Shared size=491520K line_size=256 associativity=30
processor 0: version = FF, identification = 3233E8, machine = 2964
processor 1: version = FF, identification = 3233E8, machine = 2964
cpu number : 0
physical id : 0
core id : 0
book id : 0
drawer id : 0
dedicated : 0
address : 0
siblings : 1
cpu cores : 1
version : FF
identification : 3233E8
machine : 2964
cpu MHz dynamic : 5000
cpu MHz static : 5000
cpu number : 1
physical id : 1
core id : 1
book id : 1
drawer id : 1
dedicated : 0
address : 1
siblings : 1
cpu cores : 1
version : FF
identification : 3233E8
machine : 2964
cpu MHz dynamic : 5000
cpu MHz static : 5000
```
Strange is this failure is not printed to stdout or stderr, it just exits from test with 255 error code.
### Possible fixes
(If you can, link to the line of code that might be responsible for the
problem.)
It reliably fails on Fedora 34 too. No idea why it always fails on s390x. I have seen also failure on x86_64 builder, which I could not reproduce on my own machines. s390x fails always. Could be related to CPU count?June 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)https://gitlab.isc.org/isc-projects/bind9/-/issues/1611Detect insane dnssec-policies2022-05-31T16:21:20ZMatthijs Mekkingmatthijs@isc.orgDetect insane dnssec-policiesnamed-checkconf should warn if a crazy policy is configured. This is a twofold piece of work:
1. Determine what are crazy policies (multiple keys with same role and algorithm, very short key lifetime, ...).
2. Implement the checks:
Key...named-checkconf should warn if a crazy policy is configured. This is a twofold piece of work:
1. Determine what are crazy policies (multiple keys with same role and algorithm, very short key lifetime, ...).
2. Implement the checks:
Keys:
- [X] Error if there is no KSK. Error if there is no ZSK. There should always be a set of keys that such that there is at least one key with the KSK role and one with the ZSK role. This may be the same key (CSK). #3142
- [X] Warn if multiple keys with the same role and algorithm are configured.
- [X] Warn if key lifetimes are less than one month (30 days).
Signature timings:
- [X] Error if the signature refresh is larger than the signature validity.
Rollover timings:
- [X] Error if the lifetime of a key is shorter than the rollover process duration.June 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/1601Insane dnssec-policy leads to weird times in state files2022-05-31T16:20:01ZMatthijs Mekkingmatthijs@isc.orgInsane dnssec-policy leads to weird times in state filesInsane policy with a short key lifetime and a long publish safety leads to weird state files:
```
dnssec-policy tall {
keys {
ksk key-directory lifetime P1Y algorithm 13;
zsk key-directory lifetim...Insane policy with a short key lifetime and a long publish safety leads to weird state files:
```
dnssec-policy tall {
keys {
ksk key-directory lifetime P1Y algorithm 13;
zsk key-directory lifetime PT6H algorithm 13;
};
zone-max-ttl 1d;
publish-safety 1w;
};
```
this is the predecessor key's state:
```
; This is the state of key 35225, for sisotowbell.org.
Algorithm: 13
Length: 256
Lifetime: 604800
Successor: 6279
KSK: no
ZSK: yes
Generated: 20200206082447 (Thu Feb 6 00:24:47 2020)
Published: 20200206082447 (Thu Feb 6 00:24:47 2020)
Active: 20200206082447 (Thu Feb 6 00:24:47 2020)
Retired: 20200213082447 (Thu Feb 13 00:24:47 2020)
DNSKEYChange: 20200206082447 (Thu Feb 6 00:24:47 2020)
ZRRSIGChange: 20200206082447 (Thu Feb 6 00:24:47 2020)
DNSKEYState: rumoured
ZRRSIGState: rumoured
GoalState: omnipresent
```
note that it's active today, and retired a week from now
this is the successor key's state:
```
; This is the state of key 6279, for sisotowbell.org.
Algorithm: 13
Length: 256
Lifetime: 21600
Predecessor: 35225
KSK: no
ZSK: yes
Generated: 20200206083705 (Thu Feb 6 00:37:05 2020)
Published: 20200206071947 (Wed Feb 5 23:19:47 2020)
Active: 20200213082447 (Thu Feb 13 00:24:47 2020)
Retired: 20200206083713 (Thu Feb 6 00:37:13 2020)
DNSKEYChange: 20200206083713 (Thu Feb 6 00:37:13 2020)
ZRRSIGChange: 20200206083713 (Thu Feb 6 00:37:13 2020)
DNSKEYState: unretentive
ZRRSIGState: unretentive
GoalState: hidden
```June 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/1223Add a section on using RPZ to the ARM2022-05-16T12:20:01ZVicky Riskvicky@isc.orgAdd a section on using RPZ to the ARMWe have additional notes on using several advanced features in the ARM. We should have such a section for RPZ - a complicated feature that is in widespread use that is pretty critical to those who use it. We need more than a command ref...We have additional notes on using several advanced features in the ARM. We should have such a section for RPZ - a complicated feature that is in widespread use that is pretty critical to those who use it. We need more than a command reference here.
Please also address WHY we expire RPZ zones, and whether a query for an expired RPZ zone should fail open or closed.June 2022 (9.16.30, 9.16.30-S1, 9.18.4, 9.19.2)Evan HuntEvan Hunthttps://gitlab.isc.org/isc-projects/bind9/-/issues/3441Recursive "stress" tests hang intermittently for BIND 9.16(-S)2022-07-12T07:10:16ZMichał KępieńRecursive "stress" tests hang intermittently for BIND 9.16(-S)For the past few days, recursive "stress" tests for `v9_16` and
`v9_16_sub` have been intermittently failing in weird ways:
- hangs during statistics collection at the end of the test:
- Fedora, amd64:
- https://gitl...For the past few days, recursive "stress" tests for `v9_16` and
`v9_16_sub` have been intermittently failing in weird ways:
- hangs during statistics collection at the end of the test:
- Fedora, amd64:
- https://gitlab.isc.org/isc-projects/bind9/-/jobs/2624245
- https://gitlab.isc.org/isc-private/bind9/-/jobs/2626723
- Fedora, arm64:
- https://gitlab.isc.org/isc-private/bind9/-/jobs/2624543
- https://gitlab.isc.org/isc-private/bind9/-/jobs/2626726
- https://gitlab.isc.org/isc-private/bind9/-/jobs/2629480
- core dumps:
- Fedora, amd64:
- https://gitlab.isc.org/isc-projects/bind9/-/jobs/2629186
Not all recursive "stress" test jobs are failing, though, and no such
problems have been occurring for `v9_18` or `main`. Given that the
problem started within the past week, !6419 (9.16-specific) is the most
likely culprit, but a quick glance at the changes it contains does not
raise any immediate red flags.
I managed to collect a core dump from a hung `named` instance and I
found out that all working threads are blocked on the same mutex (one of
the fetch context bucket locks). This suggests a missing mutex unlock
somewhere and explains why the server is unresponsive to queries, `rndc`
commands, etc. The question is how it got into that state, given that
it was seemingly working fine before !6419 and that the latter does not
look immediately suspicious.
**This issue is a blocker for releasing BIND 9.16.31 and 9.16.31-S1.**July 2022 (9.16.31, 9.16.31-S1, 9.18.5, 9.19.3)Michał KępieńMichał Kępieńhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3437CDS error window too small2022-07-05T13:12:27ZMark AndrewsCDS error window too smallJob [#2622252](https://gitlab.isc.org/isc-private/bind9/-/jobs/2622252) failed for isc-private/bind9@36c4446d40e7765d0fec46109cf9e765ed7c3147:
It looks like we need to allow 10 seconds instead of 3 seconds
also `D:cds:stderr did not ma...Job [#2622252](https://gitlab.isc.org/isc-private/bind9/-/jobs/2622252) failed for isc-private/bind9@36c4446d40e7765d0fec46109cf9e765ed7c3147:
It looks like we need to allow 10 seconds instead of 3 seconds
also `D:cds:stderr did not match ''` is spurious. It should only be emitted when we are searching for $err.
```
I:cds:correct signature inception time (13)
D:cds:stderr did not match ''
D:cds:bad inception time 3609 at checktime.pl line 27, <> line 28.
I:cds:failed
D:cds:exit status does not match 0
I:cds:failed
I:cds:in-place reads modification time (14)
```
```
dnssec-cds: child records must not be signed before 20220704070137 (1656918097)
dnssec-cds: which child DNSKEY records match parent DS records?
dnssec-cds: no matching DS for DNSKEY 58053 8
dnssec-cds: found matching DS 38830 8 1
dnssec-cds: no matching DS for DNSKEY 23492 8
dnssec-cds: verify DNSKEY signature(s)
dnssec-cds: skip RRSIG by key 23492: no matching (C)DS
dnssec-cds: found RRSIG by key 38830
dnssec-cds: this is the oldest so far 20220704080146 (1656921706)
dnssec-cds: skip RRSIG by key 58053: no matching (C)DS
dnssec-cds: verify CDS signature(s)
dnssec-cds: skip RRSIG by key 23492: no matching (C)DS
dnssec-cds: found RRSIG by key 38830
dnssec-cds: skip RRSIG by key 58053: no matching (C)DS
dnssec-cds: child signature inception time 20220704080146 (1656921706)
dnssec-cds: from RRSIG DNSKEY by key 38830
dnssec-cds: which child DNSKEY records match new DS records?
dnssec-cds: no matching DS for DNSKEY 58053 8
dnssec-cds: found matching DS 38830 8 1
dnssec-cds: no matching DS for DNSKEY 23492 8
dnssec-cds: verify DNSKEY signature(s)
dnssec-cds: skip RRSIG by key 23492: no matching (C)DS
dnssec-cds: found RRSIG by key 38830
dnssec-cds: skip RRSIG by key 58053: no matching (C)DS
dnssec-cds: debug 1: calling free_rbtdb(cds.test)
dnssec-cds: debug 1: done free_rbtdb(cds.test)
dnssec-cds: debug 1: calling free_rbtdb(cds.test)
dnssec-cds: debug 1: done free_rbtdb(cds.test)
```July 2022 (9.16.31, 9.16.31-S1, 9.18.5, 9.19.3)https://gitlab.isc.org/isc-projects/bind9/-/issues/3436Release Checklist for BIND 9.16.31, 9.16.31-S1, 9.18.5, 9.19.32022-07-21T14:37:47ZMichał KępieńRelease Checklist for BIND 9.16.31, 9.16.31-S1, 9.18.5, 9.19.3## Release Schedule
**Code Freeze:** Wednesday, July 6th, 2022
**Tagging Deadline:** Monday, July 11th, 2022
**Public Release:** Wednesday, July 20th, 2022
## Documentation Review Links
**Closed issues assigned to the milestone with...## Release Schedule
**Code Freeze:** Wednesday, July 6th, 2022
**Tagging Deadline:** Monday, July 11th, 2022
**Public Release:** Wednesday, July 20th, 2022
## Documentation Review Links
**Closed issues assigned to the milestone without a release note:**
- [9.19.3](https://gitlab.isc.org/isc-projects/bind9/-/issues/?sort=created_asc&state=closed&milestone_title=July%202022%20%289.16.31,%209.16.31-S1,%209.18.5,%209.19.3%29¬%5Blabel_name%5D%5B%5D=Release%20Notes¬%5Blabel_name%5D%5B%5D=Duplicate&label_name%5B%5D=v9.19)
- [9.18.5](https://gitlab.isc.org/isc-projects/bind9/-/issues/?sort=created_asc&state=closed&milestone_title=July%202022%20%289.16.31,%209.16.31-S1,%209.18.5,%209.19.3%29¬%5Blabel_name%5D%5B%5D=Release%20Notes¬%5Blabel_name%5D%5B%5D=Duplicate&label_name%5B%5D=v9.18)
- [9.16.31](https://gitlab.isc.org/isc-projects/bind9/-/issues/?sort=created_asc&state=closed&milestone_title=July%202022%20%289.16.31,%209.16.31-S1,%209.18.5,%209.19.3%29¬%5Blabel_name%5D%5B%5D=Release%20Notes¬%5Blabel_name%5D%5B%5D=Duplicate&label_name%5B%5D=v9.16)
**Merge requests merged into the milestone without a release note:**
- [9.19.3](https://gitlab.isc.org/isc-projects/bind9/-/merge_requests?sort=merged_at&state=merged&milestone_title=July%202022%20(9.16.31%2C%209.16.31-S1%2C%209.18.5%2C%209.19.3)¬[label_name][]=Release%20Notes&target_branch=main)
- [9.18.5](https://gitlab.isc.org/isc-projects/bind9/-/merge_requests?sort=merged_at&state=merged&milestone_title=July%202022%20(9.16.31%2C%209.16.31-S1%2C%209.18.5%2C%209.19.3)¬[label_name][]=Release%20Notes&target_branch=v9_18)
- [9.16.31](https://gitlab.isc.org/isc-projects/bind9/-/merge_requests?sort=merged_at&state=merged&milestone_title=July%202022%20(9.16.31%2C%209.16.31-S1%2C%209.18.5%2C%209.19.3)¬[label_name][]=Release%20Notes&target_branch=v9_16)
**Merge requests merged into the milestone without a `CHANGES` entry:**
- [9.19.3](https://gitlab.isc.org/isc-projects/bind9/-/merge_requests?sort=merged_at&state=merged&milestone_title=July%202022%20(9.16.31%2C%209.16.31-S1%2C%209.18.5%2C%209.19.3)&label_name[]=No%20CHANGES&target_branch=main)
- [9.18.5](https://gitlab.isc.org/isc-projects/bind9/-/merge_requests?sort=merged_at&state=merged&milestone_title=July%202022%20(9.16.31%2C%209.16.31-S1%2C%209.18.5%2C%209.19.3)&label_name[]=No%20CHANGES&target_branch=v9_18)
- [9.16.31](https://gitlab.isc.org/isc-projects/bind9/-/merge_requests?sort=merged_at&state=merged&milestone_title=July%202022%20(9.16.31%2C%209.16.31-S1%2C%209.18.5%2C%209.19.3)&label_name[]=No%20CHANGES&target_branch=v9_16)
## Release Checklist
### Before the Code Freeze
- [x] ***(QA)*** Inform Support and Marketing of impending release (and give estimated release dates).
- [x] ***(QA)*** Ensure there are no permanent test failures on any platform.
- [x] ***(QA)*** Check Perflab to ensure there has been no unexplained drop in performance for the versions being released.
- [x] ***(QA)*** Check whether all issues assigned to the release milestone are resolved[^1].
- [x] ***(QA)*** Ensure that there are no outstanding merge requests in the private repository[^1] (Subscription Edition only).
- [x] ***(QA)*** Ensure all merge requests marked for backporting have been indeed backported.
- [x] ***(QA)*** Update GitLab settings for all maintained branches to disallow merging to them.
- [x] ***(QA)*** Announce (on Mattermost) that the code freeze is in effect.
### Before the Tagging Deadline
- [x] ***(QA)*** Ensure release notes are correct, ask Support and Marketing to check them as well.
- [x] ***(QA)*** Add a release marker to `CHANGES`.
- [x] ***(QA)*** Add a release marker to `CHANGES.SE` (Subscription Edition only).
- [x] ***(QA)*** Update BIND 9 version in `configure.ac` (9.18+) or `version` (9.16).
- [x] ***(QA)*** Rebuild `configure` using Autoconf on `docs.isc.org` (9.16).
- [x] ***(QA)*** Tag the releases in the private repository (`git tag -s -m "BIND 9.x.y" v9_x_y`).
### Before the ASN Deadline (for ASN Releases) or the Public Release Date (for Regular Releases)
- [x] ***(QA)*** Check that the formatting is correct for HTML and PDF versions of release notes.
- [x] ***(QA)*** Check that the formatting of the generated man pages is correct.
- [x] ***(QA)*** Verify GitLab CI results for the tags created and sign off on the releases to be published.
- [x] ***(QA)*** Update GitLab settings for all maintained branches to allow merging to them again.
- [x] ***(QA)*** Prepare and merge MRs resetting the release notes and updating the version string for each maintained branch.
- [x] ***(QA)*** Announce (on Mattermost) that the code freeze is over.
- [x] ***(QA)*** Request signatures for the tarballs, providing their location and checksums.
- [x] ***(Signers)*** Validate tarball checksums, sign tarballs, and upload signatures.
- [x] ***(QA)*** Verify tarball signatures and check tarball checksums again.
- [x] ***(Support)*** Pre-publish ASN and/or Subscription Edition tarballs so that packages can be built.
- [x] ***(QA)*** Build and test ASN and/or Subscription Edition packages.
- [x] ***(QA)*** Notify Support that the releases have been prepared.
- [x] ***(Support)*** Send out ASNs (if applicable).
### On the Day of Public Release
- [x] ***(Support)*** Wait for clearance from Security Officer to proceed with the public release (if applicable).
- [x] ***(Support)*** Place tarballs in public location on FTP site.
- [x] ***(Support)*** Publish links to downloads on ISC website.
- [x] ***(Support)*** Write release email to *bind-announce*.
- [x] ***(Support)*** Write email to *bind-users* (if a major release).
- [x] ***(Support)*** Send eligible customers updated links to the Subscription Edition (update the -S edition delivery tickets, even if those links were provided earlier via an ASN ticket).
- [x] ***(Support)*** Update tickets in case of waiting support customers.
- [x] ***(QA)*** Build and test any outstanding private packages.
- [x] ***(QA)*** Build public RPMs.
- [x] ***(SwEng)*** Build Debian/Ubuntu packages.
- [x] ***(SwEng)*** Update Docker images.
- [x] ***(QA)*** Inform Marketing of the release.
- [x] ***(Marketing)*** Post short note to Twitter.
- [x] ***(Marketing)*** Update [Wikipedia entry for BIND](https://en.wikipedia.org/wiki/BIND).
- [x] ***(Marketing)*** Write blog article (if a major release).
- [x] ***(QA)*** Ensure all new tags are annotated and signed.
- [x] ***(QA)*** Push tags for the published releases to the public repository.
- [x] ***(QA)*** Merge published release tags (non-linearly) back into the their relevant development/maintenance branches.
- [x] ***(QA)*** Sanitize confidential issues which are assigned to the current release milestone and do not describe a security vulnerability, then make them public.
- [x] ***(QA)*** Sanitize confidential issues which are assigned to older release milestones and describe security vulnerabilities, then make them public if appropriate[^2].
- [x] ***(QA)*** Update QA tools used in GitLab CI (e.g. Flake8, PyLint) by modifying the relevant `Dockerfile`.
[^1]: If not, use the time remaining until the tagging deadline to ensure all outstanding issues are either resolved or moved to a different milestone.
[^2]: As a rule of thumb, security vulnerabilities which have reproducers merged to the public repository are considered okay for full disclosure.July 2022 (9.16.31, 9.16.31-S1, 9.18.5, 9.19.3)Michal NowakMichal Nowak2022-07-20