BIND issueshttps://gitlab.isc.org/isc-projects/bind9/-/issues2023-04-06T12:27:32Zhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3658journal performance improvements - use asynchronous I/O2023-04-06T12:27:32ZPetr Špačekpspacek@isc.orgjournal performance improvements - use asynchronous I/OCurrently journal I/O is done synchronously, including fsync(). `fsync` is very slow (see #3556) and we probably cannot get rid of it completely, so I think we should use asynchronous I/O for journal processing. That would allow the serv...Currently journal I/O is done synchronously, including fsync(). `fsync` is very slow (see #3556) and we probably cannot get rid of it completely, so I think we should use asynchronous I/O for journal processing. That would allow the server to process normal queries while waiting for journal operations.
TODO: Test IXFR-in and querying the server in parallel.https://gitlab.isc.org/isc-projects/bind9/-/issues/3653update-policy logic logging/debugging2022-11-04T07:04:22ZJorisupdate-policy logic logging/debugging### Description
Currently debugging update-policy rules is opaque, only the decision is available.
It would be beneficial if it was possible to trace the individual steps taken, eg which rule was expanded to what (wildcard expansion, k...### Description
Currently debugging update-policy rules is opaque, only the decision is available.
It would be beneficial if it was possible to trace the individual steps taken, eg which rule was expanded to what (wildcard expansion, key names vs fqdn names) to which result.
### Request
Logging at the right trace level could be something like below. Note I'm by far not an expert in the update policies, my example might way off.
```update-policy
grant wrong-key-name name example.com ANY;
=> identity key wrong-key-name not found, aborted
grant key-name name example.com NS;
=> update request does not match name example.com, aborted
grant key-name name example.com MX;
=> update request does not match type, aborted
grant updater-key.example.com name example.com ANY;
=> identity host updater-key.example.com not found, aborted [ a keyname that looks like a fqdn? ]
=> request denied```
Similarly key/kerberos failures could be logged too.
### Links / references
In #235 a similar request is made, this elaborates the scope and suggested solution.https://gitlab.isc.org/isc-projects/bind9/-/issues/3649Feature request: configurable TCP timeouts on zone refresh queries2023-06-15T13:22:31ZCathy AlmondFeature request: configurable TCP timeouts on zone refresh queriesThere is nothing that can be configured to reduce the timeout when failing to reach an authoritative server with a refresh query over TCP (BIND default is "try-tcp-refresh yes;")
Customer input is :
```
Basically, if my slave is trying ...There is nothing that can be configured to reduce the timeout when failing to reach an authoritative server with a refresh query over TCP (BIND default is "try-tcp-refresh yes;")
Customer input is :
```
Basically, if my slave is trying to reach out to a third-party master and doesn't get a response in 10-15 seconds, I want it to move on to the next listed master in hopes of better results versus waiting for 2 minutes
```
A 'sort of' workaround for this is to allow the UDP timeout to happen ("try-tcp-refresh no;") but that takes away the possibility of being able to reach servers and to pull a zone transfer in the situation where UDP doesn't work, but TCP does.
We have configurable TCP timeouts for other BIND functions, but not for this.
See [Support ticket #21044](https://support.isc.org/Ticket/Display.html?id=21044)https://gitlab.isc.org/isc-projects/bind9/-/issues/3635Implement support for DNS over QUIC2023-02-15T19:00:14ZJeremy SakladImplement support for DNS over QUIC### Description
DNS-over-QUIC, specified in [RFC 9250][RFC 9250], has considerable advantages over the already-implemented options.
* With the debatable exception of DoH and HTTP/3, it is the only standardized encrypted DNS protocol to...### Description
DNS-over-QUIC, specified in [RFC 9250][RFC 9250], has considerable advantages over the already-implemented options.
* With the debatable exception of DoH and HTTP/3, it is the only standardized encrypted DNS protocol to operate over UDP.
* It avoids issues such as head-of-line blocking and potential for amplification attacks.
* It avoids the overhead of DNS-over-HTTPS.
### Request
DNS-over-QUIC should be offered wherever DNS-over-HTTPS or DNS-over-TLS is, at minimum. Its use should be encouraged over the others where applicable.
[RFC 9250][RFC 9250] emphasizes the following scopes of usage:
> * the "stub to recursive resolver" scenario (also called the "stub to recursive" scenario in this document)
> * the "recursive resolver to authoritative nameserver" scenario (also called the "recursive to authoritative" scenario in this document), and
> * the "nameserver to nameserver" scenario (mainly used for zone transfers (XFR) [RFC1995][RFC 1995] [RFC5936][RFC 5936]).
I believe that covers every function of BIND.
---
While not specific to DNS-over-QUIC, the implementation should be designed with future support for non-standard ports and SVBC records in mind. 53/udp is explicitly banned for use with this protocol, but it should eventually be possible to use any other non-standard port rather than 853/udp.
[RFC 1995]: https://www.rfc-editor.org/info/rfc1995
[RFC 5936]: https://www.rfc-editor.org/info/rfc5936
[RFC 9250]: https://www.rfc-editor.org/info/rfc9250Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3629Slow retries after timeouts2022-11-16T16:07:31Zgrzchr15Slow retries after timeoutsMeasurements from Neustar/UltraDNS for some DNS servers including BIND9,They think there is weird behavior:
https://ripe85.ripe.net/wp-content/uploads/presentations/96-dknight-fewnsmanyips-ripe85-dnswg.pdf
Page 8
Video: https://ripe85....Measurements from Neustar/UltraDNS for some DNS servers including BIND9,They think there is weird behavior:
https://ripe85.ripe.net/wp-content/uploads/presentations/96-dknight-fewnsmanyips-ripe85-dnswg.pdf
Page 8
Video: https://ripe85.ripe.net/archive/video/dave-knight_fewer-name-servers-more-addresses_main-20221027-112204.mp4 time 7:17 onwards
Measurements from Neustar/UltraDNS for some DNS servers including BIND9
- Bind Strongly prefers IPv6 - they think there is weird behavior.
- If one address is broken, penalize all higher numbered addresses until piling onto the last one?
- Slow to get an answer when retryingŠtěpán BalážikŠtěpán Balážikhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3623A combination of an RPZ with NSIP triggers & unusually (but not entirely) bro...2022-10-27T11:36:42ZGraham ClinchA combination of an RPZ with NSIP triggers & unusually (but not entirely) broken delegation gives SERVFAIL & "rpz NSIP rewrite X via Y NS address rewrite rrset failed: failure"<!--
If the bug you are reporting is potentially security-related - for example,
if it involves an assertion failure or other crash in `named` that can be
triggered repeatedly - then please do *NOT* report it here, but send an
email to [...<!--
If the bug you are reporting is potentially security-related - for example,
if it involves an assertion failure or other crash in `named` that can be
triggered repeatedly - then please do *NOT* report it here, but send an
email to [security-officer@isc.org](security-officer@isc.org).
-->
### Summary
When a configured response policy zone contains rpz-nsip triggers **and** NS record resolution does not complete successfully (but in an apparently unusual way), SERVFAIL is returned to queries that would otherwise succeed.
### BIND version used
```
BIND 9.18.8 (Stable Release) <id:35f5d35>
running on Darwin arm64 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:19:52 PDT 2022; root:xnu-8020.140.49~2/RELEASE_ARM64_T6000
built by make with '--prefix=/opt/homebrew/Cellar/bind/9.18.8' '--sysconfdir=/opt/homebrew/etc/bind' '--localstatedir=/opt/homebrew/var' '--with-json-c' '--with-libidn2=/opt/homebrew/opt/libidn2' '--with-openssl=/opt/homebrew/opt/openssl@3' '--without-lmdb' 'CC=clang' 'PKG_CONFIG_PATH=/opt/homebrew/opt/json-c/lib/pkgconfig:/opt/homebrew/opt/libidn2/lib/pkgconfig:/opt/homebrew/opt/libnghttp2/lib/pkgconfig:/opt/homebrew/opt/libuv/lib/pkgconfig:/opt/homebrew/opt/openssl@3/lib/pkgconfig' 'PKG_CONFIG_LIBDIR=/usr/lib/pkgconfig:/opt/homebrew/Library/Homebrew/os/mac/pkgconfig/12'
compiled by CLANG Apple LLVM 14.0.0 (clang-1400.0.29.202)
compiled with OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
linked to OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
compiled with libuv version: 1.44.2
linked to libuv version: 1.44.2
compiled with libnghttp2 version: 1.50.0
linked to libnghttp2 version: 1.50.0
compiled with libxml2 version: 2.9.4
linked to libxml2 version: 20904
compiled with json-c version: 0.16
linked to json-c version: 0.16
compiled with zlib version: 1.2.11
linked to zlib version: 1.2.11
threads support is enabled
DNSSEC algorithms: RSASHA1 NSEC3RSASHA1 RSASHA256 RSASHA512 ECDSAP256SHA256 ECDSAP384SHA384 ED25519 ED448
DS algorithms: SHA-1 SHA-256 SHA-384
HMAC algorithms: HMAC-MD5 HMAC-SHA1 HMAC-SHA224 HMAC-SHA256 HMAC-SHA384 HMAC-SHA512
TKEY mode 2 support (Diffie-Hellman): yes
TKEY mode 3 support (GSS-API): yes
default paths:
named configuration: /opt/homebrew/etc/bind/named.conf
rndc configuration: /opt/homebrew/etc/bind/rndc.conf
DNSSEC root key: /opt/homebrew/etc/bind/bind.keys
nsupdate session key: /opt/homebrew/var/run/named/session.key
named PID file: /opt/homebrew/var/run/named/named.pid
named lock file: /opt/homebrew/var/run/named/named.lock
```
### Steps to reproduce
Configure a minimal BIND 9 recursive resolver with a response policy zone that includes an rpz-nsip match, and then attempt to resolve www.britishairways.com (which appears to have an unusually broken partially lame delegation).
### What is the current *bug* behavior?
DNS resolution of "www.britishways.com" fails with SERVFAIL:
```
$ delv www.britishairways.com @::1
;; resolution failed: SERVFAIL
```
named (at -d 1) logs:
```
25-Oct-2022 22:53:35.007 client @0x1439ad560 ::1#53833 (www.britishairways.com): rpz NSIP rewrite www.britishairways.com via dnssec1-win.server.ntli.net NS address rewrite rrset failed: failure
25-Oct-2022 22:53:35.007 client @0x1439ad560 ::1#53833 (www.britishairways.com): query failed (SERVFAIL) for www.britishairways.com/IN/A at query.c:7232
```
### What is the expected *correct* behavior?
DNS resolution of "www.britishairways.com" should succeed:
```
$ delv www.britishairways.com @::1
; unsigned answer
www.britishairways.com. 60 IN CNAME www.ba.com.edgekey.net.
www.ba.com.edgekey.net. 21600 IN CNAME e8308.b.akamaiedge.net.
e8308.b.akamaiedge.net. 20 IN A 104.117.169.173
```
### Relevant configuration files
named.conf:
```
options {
response-policy {
zone "test.example.net" policy given;
};
};
zone "test.example.net" {
type primary;
file "test.example.net";
};
```
test.example.net zone file:
```
@ SOA ns1 hostmaster. (
2003080800 ; serial number
12h ; refresh
15m ; update retry
3w ; expiry
2h ; minimum
)
@ NS ns1
ns1 A 127.0.0.1
foo.com CNAME .
32.99.99.168.192.rpz-nsip CNAME .
```
Note that simply commenting out the final line in the zone file causes the problem to go away.
### Relevant logs and/or screenshots
named -d 2 -g output:
```
[...]
25-Oct-2022 22:57:14.370 fetch: www.britishairways.com/A
25-Oct-2022 22:57:14.371 fetch: _.com/A
25-Oct-2022 22:57:14.393 fetch: _.britishairways.com/A
25-Oct-2022 22:57:14.419 fetch: ns1.britishairways.com/AAAA
25-Oct-2022 22:57:14.419 fetch: ns2.britishairways.com/AAAA
25-Oct-2022 22:57:14.419 fetch: dnssec1-win.server.ntli.net/A
25-Oct-2022 22:57:14.419 fetch: dnssec1-win.server.ntli.net/AAAA
25-Oct-2022 22:57:14.419 fetch: dnssec2-win.server.ntli.net/A
25-Oct-2022 22:57:14.419 fetch: dnssec2-win.server.ntli.net/AAAA
25-Oct-2022 22:57:14.439 delete_node(): 0x600002c8edf0 www.britishairways.com (bucket 15)
25-Oct-2022 22:57:14.440 fetch: britishairways.com/DS
25-Oct-2022 22:57:14.456 fetch: com/DNSKEY
25-Oct-2022 22:57:14.477 fetch: dnssec1-win.server.ntli.net/A
25-Oct-2022 22:57:14.494 lame server resolving 'dnssec2-win.server.ntli.net' (in 'server.ntli.net'?): 194.168.4.237#53
25-Oct-2022 22:57:14.495 lame server resolving 'dnssec1-win.server.ntli.net' (in 'server.ntli.net'?): 194.168.4.237#53
25-Oct-2022 22:57:14.495 lame server resolving 'dnssec1-win.server.ntli.net' (in 'server.ntli.net'?): 194.168.4.237#53
25-Oct-2022 22:57:14.497 lame server resolving 'dnssec2-win.server.ntli.net' (in 'server.ntli.net'?): 194.168.4.237#53
25-Oct-2022 22:57:14.497 lame server resolving 'dnssec1-win.server.ntli.net' (in 'server.ntli.net'?): 194.168.4.237#53
25-Oct-2022 22:57:14.522 lame server resolving 'dnssec2-win.server.ntli.net' (in 'server.ntli.net'?): 62.253.162.237#53
25-Oct-2022 22:57:14.522 fetch: dns1.ntli.net/AAAA
25-Oct-2022 22:57:14.522 fetch: dns2.ntli.net/AAAA
25-Oct-2022 22:57:14.522 lame server resolving 'dnssec1-win.server.ntli.net' (in 'server.ntli.net'?): 62.253.162.237#53
25-Oct-2022 22:57:14.522 lame server resolving 'dnssec1-win.server.ntli.net' (in 'server.ntli.net'?): 62.253.162.237#53
25-Oct-2022 22:57:14.522 lame server resolving 'dnssec2-win.server.ntli.net' (in 'server.ntli.net'?): 62.253.162.237#53
25-Oct-2022 22:57:14.523 lame server resolving 'dnssec1-win.server.ntli.net' (in 'server.ntli.net'?): 62.253.162.237#53
25-Oct-2022 22:57:14.523 client @0x11a870560 ::1#52973 (www.britishairways.com): rpz NSIP rewrite www.britishairways.com via dnssec1-win.server.ntli.net NS address rewrite rrset failed: failure
25-Oct-2022 22:57:14.523 client @0x11a870560 ::1#52973 (www.britishairways.com): query failed (SERVFAIL) for www.britishairways.com/IN/A at query.c:7232
25-Oct-2022 22:57:14.523 fetch completed at resolver.c:4140 for dnssec1-win.server.ntli.net/A in 0.045906: failure/success [domain:server.ntli.net,referral:0,restart:2,qrysent:2,timeout:0,lame:2,quota:0,neterr:0,badresp:0,adberr:0,findfail:0,valfail:0]
```
### Possible fixes
Unclear, but appears to be a fault in the rpz-nsip processing when an "unusually unknown" NS IP is processed.https://gitlab.isc.org/isc-projects/bind9/-/issues/3618dynamic TTL shortening in auth after RR change2022-10-26T09:07:48ZPetr Špačekpspacek@isc.orgdynamic TTL shortening in auth after RR change### Description
TL;DR version: Withdrawing DS is a nightmare because TLDs have too long TTLs. COM with 1 day is a total nightmare and risk-averse bussinesses like google.com are not going to risk 1 day disruption of service => no prospe...### Description
TL;DR version: Withdrawing DS is a nightmare because TLDs have too long TTLs. COM with 1 day is a total nightmare and risk-averse bussinesses like google.com are not going to risk 1 day disruption of service => no prospect of deploying DNSSEC.
Long version:
https://indico.dns-oarc.net/event/44/contributions/962/
### Request
I'm considering an _experiment_, not a production-ready feature. Auth DNS is not a good place for what I'm going to propose, but I still think it is a nice experiment:
Add magic which shortens TTLs sent out in answers after RR modification. Say, in the first hour after modification shorten TTL of modified DS RR to 60 seconds. After that use the original TTL. (Of course we can invent any other schema, this is just a simple example.)
Obviously this requires knowing when RR was modified - and this is a nightmare by itself. For an experiment I think it would be good enough to look at RRSIG inception time to detect the initial window. Obviously this will have false positives after resigning, but for an experiment I think we don't need to care.
An experiment would allow us to detect if something breaks when TTL on RR and it's RRSIG do not match when sent as an answer from auth. (It should work, but you know how it is ...)
### Links / references
- https://indico.dns-oarc.net/event/44/contributions/962/
- https://chat.dns-oarc.net/community/pl/u36txi1cw3ykzx7iaos4rqo95c
- https://www.ripe.net/ripe/mail/archives/dns-wg/2021-December/003935.htmlhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3616empty zones slow down loading lots of zones2023-11-02T17:05:05ZPetr Špačekpspacek@isc.orgempty zones slow down loading lots of zones### Summary
Loading 1 M zones from textual named.conf & minimal zone file is much slower if `empty-zones-enable yes` (default) is in effect.
### BIND version used
BIND 9.19.7-dev (Development Release) 64287e4
### Steps to reproduce
1...### Summary
Loading 1 M zones from textual named.conf & minimal zone file is much slower if `empty-zones-enable yes` (default) is in effect.
### BIND version used
BIND 9.19.7-dev (Development Release) 64287e4
### Steps to reproduce
1. Generate config
```
for I in $(seq 1 1000000); do echo "zone z$I.example. { type primary; file \"db\"; };" >> named.conf; done
echo 'options { notify no; };' >> named.conf
```
2. Use minimal zone file:
[db](/uploads/dc86b8dc10f481b4d9b2f9718a7efd2a/db)
3. Run BIND:
```
named -g -c named.conf &> log
```
4. Observe CPU utilization going through the roof. Startup time is over 60 seconds on my laptop.
### What is the expected *correct* behavior?
It could be faster :-)
### Relevant logs and/or screenshots
Notice the timestamps - it takes quite long to process a single zone:
```
20-Oct-2022 15:55:37.781 automatic empty zone: 10.IN-ADDR.ARPA
20-Oct-2022 15:55:37.971 automatic empty zone: 16.172.IN-ADDR.ARPA
20-Oct-2022 15:55:38.148 automatic empty zone: 17.172.IN-ADDR.ARPA
20-Oct-2022 15:55:38.325 automatic empty zone: 18.172.IN-ADDR.ARPA
20-Oct-2022 15:55:38.505 automatic empty zone: 19.172.IN-ADDR.ARPA
```
Flamegraph, all threads combined:
![nothreads.svg](/uploads/30254da36ec0ee0be71ca1d86d0e2cde/nothreads.svg)
With
```
options { notify no; empty-zones-enable no; };
```
the load time decreases down to ~ 42 s, i.e. almost 1/3 decrease.
### Possible fixes
From a quick glance, it seems that `create_empty_zone()` is called in a cycle and it effectively walks the cfg-representation of zone list over and over, including conversion from text to wire format, and then compares zone names. This is done to detect forward zones with particular configuration.
Unless I'm missing something, we should be able to move empty zones creating to the very end of config processing and use forward table for way more efficient lookups which should lower the processing time for empty zones to almost zero.Not plannedTony FinchTony Finchhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3561Analyze the current lock contention2022-09-26T09:20:18ZOndřej SurýAnalyze the current lock contentionI've hacked `mutrace` and `named` to work again together and here are current results:
```
mutrace: Showing statistics for process named (PID: 1782915).
mutrace: 1802520 mutexes used.
Mutex #1326836 (0x0x55ec3f8dd640) first referenced ...I've hacked `mutrace` and `named` to work again together and here are current results:
```
mutrace: Showing statistics for process named (PID: 1782915).
mutrace: 1802520 mutexes used.
Mutex #1326836 (0x0x55ec3f8dd640) first referenced by:
/usr/local/lib/libmutrace.so(pthread_mutex_init+0xe8) [0x7fc95d0d87a8]
/home/ondrej/Projects/bind9/lib/dns/.libs/libdns-9.19.6-dev.so(dns_sdb_register+0xb7) [0x7fc95ca55f32]
Mutex #1332002 (0x0x55ec3fa3b4e0) first referenced by:
/usr/local/lib/libmutrace.so(pthread_mutex_init+0xe8) [0x7fc95d0d87a8]
/home/ondrej/Projects/bind9/lib/isc/.libs/libisc-9.19.6-dev.so(isc__task_create+0xbd) [0x7fc95cb98aee]
Mutex #1329994 (0x0x55ec3f9cea68) first referenced by:
/usr/local/lib/libmutrace.so(pthread_rwlock_init+0xca) [0x7fc95d0d942a]
/home/ondrej/Projects/bind9/lib/dns/.libs/libdns-9.19.6-dev.so(dns_rbtdb_create+0xfe) [0x7fc95c9e4cfc]
Mutex #1332378 (0x0x55ec3fa5b7a8) first referenced by:
/usr/local/lib/libmutrace.so(pthread_rwlock_init+0xca) [0x7fc95d0d942a]
/home/ondrej/Projects/bind9/lib/dns/.libs/libdns-9.19.6-dev.so(dns_adb_create+0x120) [0x7fc95c951561]
Mutex #1332379 (0x0x55ec3fa5bd30) first referenced by:
/usr/local/lib/libmutrace.so(pthread_mutex_init+0xe8) [0x7fc95d0d87a8]
/home/ondrej/Projects/bind9/lib/isc/.libs/libisc-9.19.6-dev.so(isc__task_create+0xbd) [0x7fc95cb98aee]
Mutex #1327168 (0x0x55ec3f8f9558) first referenced by:
/usr/local/lib/libmutrace.so(pthread_rwlock_init+0xca) [0x7fc95d0d942a]
/home/ondrej/Projects/bind9/lib/dns/.libs/libdns-9.19.6-dev.so(dns_zonemgr_create+0x218) [0x7fc95caa36d6]
Mutex #1325497 (0x0x55ec3f841648) first referenced by:
/usr/local/lib/libmutrace.so(pthread_mutex_init+0xe8) [0x7fc95d0d87a8]
/home/ondrej/Projects/bind9/lib/isc/.libs/libisc-9.19.6-dev.so(+0x438b4) [0x7fc95cb8c8b4]
Mutex #78281 (0x0x7fc95d0d1fa0) first referenced by:
/usr/local/lib/libmutrace.so(pthread_mutex_lock+0x4c) [0x7fc95d0d89bc]
/usr/lib/x86_64-linux-gnu/libuv.so.1(uv_mutex_lock+0x9) [0x7fc95c3aa689]
Mutex #1330013 (0x0x55ec3f9cf2c0) first referenced by:
/usr/local/lib/libmutrace.so(pthread_rwlock_init+0xca) [0x7fc95d0d942a]
/home/ondrej/Projects/bind9/lib/dns/.libs/libdns-9.19.6-dev.so(dns_rbtdb_create+0x497) [0x7fc95c9e5095]
Mutex #1325525 (0x0x55ec3f843f48) first referenced by:
/usr/local/lib/libmutrace.so(pthread_mutex_init+0xe8) [0x7fc95d0d87a8]
/home/ondrej/Projects/bind9/lib/isc/.libs/libisc-9.19.6-dev.so(+0x411fe) [0x7fc95cb8a1fe]
mutrace: Showing 10 mutexes in order of (write) contention count:
Mutex # Changed Locked tot.Time[ms] avg.Time[ms] max.Time[ms] Cont. tot.Cont[ms] avg.Cont[ms] max.Cont[ms] Flags
1326836 959 3544 16.228 0.005 0.055 829 93.809 0.113 2.528 Mx.a-.
1332002 10563 942412 784.830 0.001 0.183 330 0.379 0.001 0.021 Mx.a-.
1329994 7845 141856 7.790 0.000 0.079 185 1.978 0.011 0.130 W!...r
(read) 989396 57.085 0.000 0.169 254 5.736 0.023 0.112 ||||||
1332378 4593 48592 1590.456 0.033 45.106 165 56.996 0.345 7.611 Wx...r
(read) 1 39.878 39.878 39.878 0 0.000 0.000 0.000 ||||||
1332379 23798 36940 37.882 0.001 0.101 108 0.210 0.002 0.081 Mx.a-.
1327168 133 314 24.442 0.078 0.276 96 8.554 0.089 0.534 Wx...r
(read) 2 1.013 0.507 0.760 0 0.000 0.000 0.000 ||||||
1325497 2107 1018717 828.111 0.001 0.207 65 0.115 0.002 0.015 Mx.a-.
78281 19 21 1.274 0.061 0.186 15 2.693 0.180 0.251 M-.--.
1330013 2650 41181 52.259 0.001 0.104 11 0.044 0.004 0.007 Wx...r
(read) 103766 96.121 0.001 0.130 8 0.099 0.012 0.049 ||||||
1325525 26482 30675 31.489 0.001 0.092 11 0.033 0.003 0.010 Mx.a-.
... ... ... ... ... ... ... ... ... ... ||||||
/|||||
Object: M = Mutex, W = RWLock, I = isc_rwlock /||||
State: x = dead, ! = inconsistent /|||
Use: R = used in realtime thread /||
Mutex Type: r = RECURSIVE, e = ERRORCHECK, a = ADAPTIVE /|
Mutex Protocol: i = INHERIT, p = PROTECT /
RWLock Kind: r = PREFER_READER, w = PREFER_WRITER, W = PREFER_WRITER_NONREC
mutrace: Note that rwlocks are shown as two lines: write locks then read locks.
mutrace: Note that the flags column R is only valid in --track-rt mode!
mutrace: 1 condition variables used.
mutrace: No condition variable contended according to filtering parameters.
mutrace: Total runtime is 185346.476 ms.
mutrace: Results for SMP with 8 processors.
mutrace: WARNING: 139919 inconsistent mutex uses detected. Results might not be reliable.
mutrace: Fix your program first!
mutrace: WARNING: 98 internal hash collisions detected. Results might not be as reliable as they could be.
mutrace: Try to increase --hash-size=, which is currently at 400000009.
mutrace: WARNING: 533 internal mutex contention detected. Results might not be reliable as they could be.
mutrace: Try to increase --hash-size=, which is currently at 400000009.
```https://gitlab.isc.org/isc-projects/bind9/-/issues/3558Batched UPDATE processing2022-09-23T08:52:56ZTony FinchBatched UPDATE processingThere's a cool thing that can happen when you have a queue whose consumer can work in batches: as the load increases, the batch size increases, so the per-batch overhead is amortized over more queue entries and the system's overall effic...There's a cool thing that can happen when you have a queue whose consumer can work in batches: as the load increases, the batch size increases, so the per-batch overhead is amortized over more queue entries and the system's overall efficiency goes up.
This occurs in the DNS for UPDATE processing: UPDATEs are serialized, so new UPDATEs must be queued while an UPDATE is in progress. It then makes sense when an UPDATE is completed to process all pending UPDATEs in a single transaction, so the server has a chance of catching up with its backlog.Not plannedTony FinchTony Finchhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3556journal performance improvements - limit fsync()2023-04-06T17:09:34ZPetr Špačekpspacek@isc.orgjournal performance improvements - limit fsync()Version: 9.19.5-dev (Development Release) <id:b13d973>
Observation:
In a trivial test on a laptop the UPDATE performance on a single zone is about 1/300 of the query performance on the same zone. Using `tmpfs` for journal makes it as fa...Version: 9.19.5-dev (Development Release) <id:b13d973>
Observation:
In a trivial test on a laptop the UPDATE performance on a single zone is about 1/300 of the query performance on the same zone. Using `tmpfs` for journal makes it as fast as ~ 1/8 of the query performance on a 4 core machine, with multi-threading 8 virtual cores. (That kind of makes sense because of the serialization?)
From a quick look I guess it is because `fsync()` syscall is a real performance drag.
This issue is for discussing possibilities to improve UPDATE performance.
In the code I can see two relatively radical but simple opportunities for optimization:
- [ ] Make `fsync()` use for journal files configurable. IIUC `fsync()` guards only against hardware failure/kernel crash but it should not matter from perspective of a crashing process. Once the data are in kernel buffer after `write()` the process can crash but the filesystem will retain the content anyway. A fsync-knob would allow users to trade performance vs. journal resiliency against system crash (but again, not `named` crash).
- It might even make sense for for secondary zones which can be re-transferred at will.
- It is not unheard-of, see https://www.knot-dns.cz/docs/latest/singlehtml/#journal-db-mode.
- [ ] Even when `fsync()` is required/configured, I think that in `dns_journal_compact()` it should be good enough to call `fsync()` once at the very end. There should be zero risk until the new journal is renamed to the original name, so there is no need to `fsync()` it before that point. That should improve performance too.
Side-note about `fsync()`: It is a trap because on journaled filesystems it can force _all other_ transactions in filesystem journal to be flushed to disk before it even starts flushing the intended file, so it might be even slower than we would like.https://gitlab.isc.org/isc-projects/bind9/-/issues/3552Potential method to DoS the statistics channel and prevent BIND from exiting ...2023-05-23T12:31:12ZCathy AlmondPotential method to DoS the statistics channel and prevent BIND from exiting on 9.16As reported to us in [Support Ticket #21309](https://support.isc.org/Ticket/Display.html?id=21309)
The submitter writes:
This ticket is being reported against BIND 9.16.23; these issues were found during code review after the (Septembe...As reported to us in [Support Ticket #21309](https://support.isc.org/Ticket/Display.html?id=21309)
The submitter writes:
This ticket is being reported against BIND 9.16.23; these issues were found during code review after the (September) CVE announcement.
The implementation in lib/isc/httpd.c appears to be implementing the "Connection" header from HTTP/1.0 form of persistent connections. In
HTTP/1.1, persistent connections are the default. But this seems to have an effect in the lib/isc/httpd.c code only when a HTTP request has been received on the connection and the appropriate flags have been set.
isc_httpdmgr_create() accepts a timermgr argument, but appears to do nothing with it except save it into a context. There are no timeouts. So
a client can open a connection and keep it open perpetually (I've checked that this can happen). There is no quota. So if this socket is open to the public internet, a question is whether this be exploited to cause, e.g., fd exhaustion or even OOM condition. (SO_KEEPALIVE is not
set either.)
It appears that named limits the number of open accepted connections for the statistics-channel socket.
ISC BIND 9.16.23-S1's named is run with the following config:
```
options {
listen-on port 5300 { 127.0.0.1; };
// ...
};
statistics-channels {
inet * port 5302 allow { localhost; };
};
```
named logged the following message on startup:
```
21-Sep-2022 21:05:43.287 using up to 21000 sockets
```
A simple Python client program was run after setting the fd soft-limit of the client program environment:
```
$ ulimit -H -n
524288
$ ulimit -S -n 512000
$ ulimit -S -n
512000
$ cat statsconnect.py
import socket
import time
sockets = []
for i in range(0, 480000):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 5302))
sockets.append(s)
time.sleep(3600)
$ python statsconnect.py
^CTraceback (most recent call last):
File "/home/myself/statsconnect.py", line 7, in <module>
s.connect(('127.0.0.1', 5302))
KeyboardInterrupt
$
```
(It was interrupted by SIGINT keyboard break.)
named logged the following message upon running statsconnect.py:
```
21-Sep-2022 21:13:46.799 accept: file descriptor exceeds limit (21000/21000)
```
After that, named still responds to DNS queries:
```
$ dig +tcp +short @127.0.0.1 -p 5300 . soa
a.root-servers.net. nstld.verisign-grs.com. 2022091300 1800 900 604800 86400
$
```
But it does not accept any more connections on the statistics-channels socket (indefinitely, even after the statsconnect.py process has exited
and over 60 seconds have passed which should also factor in any wait period if the kernel automatically collects the client's fds):
```
$ telnet 127.0.0.1 5302
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection timed out
$
```
So at least DoS of the statistics-channel appears possible.
And named does not terminate when attempting to stop it:
```
21-Sep-2022 21:13:46.799 accept: file descriptor exceeds limit (21000/21000)
21-Sep-2022 21:29:23.296 no longer listening on 127.0.0.1#5300
21-Sep-2022 21:29:23.298 shutting down
21-Sep-2022 21:29:23.298 stopping statistics channel on 0.0.0.0#5302
```
and blocks there indefinitely.
pstack dumps this:
```
Thread 2 (Thread 0x7fd3f1893640 (LWP 431172) "isc-net-0000"):
#0 0x00007fd3f1e627b0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fd3f1e5b6b2 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007fd3f245ee6f in isc_socket_cancel (sock=0x7fd3df675178, task=0x7fd3df670190, how=how@entry=15) at socket.c:4975
#3 0x00007fd3f2423f57 in isc_httpdmgr_shutdown (httpdmgrp=<optimized out>) at httpd.c:1094
#4 0x000000000044ee8c in shutdown_listener (listener=0x7fd3cdf41010) at statschannel.c:3773
#5 named_statschannels_shutdown (server=server@entry=0x7fd3df65f010) at statschannel.c:4156
#6 0x000000000043c505 in shutdown_server (task=<optimized out>, event=<optimized out>) at ./server.c:10144
#7 0x00007fd3f244c355 in task_run (task=0x7fd3df66a010) at task.c:857
#8 isc_task_run (task=0x7fd3df66a010) at task.c:950
#9 0x00007fd3f2436e49 in isc__nm_async_task (worker=0xc93790, ev0=0x7fd3df673970) at netmgr.c:879
#10 process_netievent (worker=worker@entry=0xc93790, ievent=0x7fd3df673970) at netmgr.c:958
#11 0x00007fd3f2436fc5 in process_queue (worker=worker@entry=0xc93790, type=type@entry=NETIEVENT_TASK) at netmgr.c:1027
#12 0x00007fd3f2437773 in process_all_queues (worker=0xc93790) at netmgr.c:798
#13 async_cb (handle=0xc93af0) at netmgr.c:827
#14 0x00007fd3f220fc0d in uv.async_io.part () from /lib64/libuv.so.1
#15 0x00007fd3f222ba84 in uv.io_poll.part () from /lib64/libuv.so.1
#16 0x00007fd3f2215630 in uv_run () from /lib64/libuv.so.1
#17 0x00007fd3f2437077 in nm_thread (worker0=0xc93790) at netmgr.c:733
#18 0x00007fd3f244e6d6 in isc__trampoline_run (arg=0xc7fc40) at trampoline.c:196
#19 0x00007fd3f1e592a5 in start_thread () from /lib64/libpthread.so.0
#20 0x00007fd3f1d81323 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fd3f1a17480 (LWP 431171) "named"):
#0 0x00007fd3f1d48ac5 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fd3f1d4dce7 in nanosleep () from /lib64/libc.so.6
#2 0x00007fd3f1d79669 in usleep () from /lib64/libc.so.6
#3 0x00007fd3f244cd2a in isc__taskmgr_destroy (managerp=0x47ba28 <named_g_taskmgr>) at task.c:1107
#4 0x00007fd3f242b381 in isc_managers_destroy (netmgrp=0x47ba08 <named_g_nm>, taskmgrp=0x47ba28 <named_g_taskmgr>) at managers.c:90
#5 0x000000000041772d in destroy_managers () at ./main.c:967
#6 cleanup () at ./main.c:1341
#7 main (argc=<optimized out>, argv=<optimized out>) at ./main.c:1613
```
There are 98 other threads [it is a 32 processors machine].
As the named resolver's upstream communications still use the socketmgr, could it also be impacted by this, as that 21000 is a socketmgr limit of fds?https://gitlab.isc.org/isc-projects/bind9/-/issues/3533Describe security policies on various components to the ARM2022-09-18T09:46:42ZOndřej SurýDescribe security policies on various components to the ARMWe were discussing how we should treat the security vulnerabilities in BIND 9 components and this is probably something that should go into the ARM along with recommendations.
E.g.:
* authoritative query processing (PR:N)
* recursive qu...We were discussing how we should treat the security vulnerabilities in BIND 9 components and this is probably something that should go into the ARM along with recommendations.
E.g.:
* authoritative query processing (PR:N)
* recursive query processing (PR:L)
* zone transfers
* control channel
I mean, I think we could expand and merge this document: https://gitlab.isc.org/isc-projects/bind9/-/wikis/CVSS-Scoring-Guidelines to the ARM.https://gitlab.isc.org/isc-projects/bind9/-/issues/3516Log root priming failures at severity NOTICE2024-03-27T13:34:12ZCathy AlmondLog root priming failures at severity NOTICEResolver root priming is logged at level INFO.
From BIND 9.18 and newer, root priming also logs the outcome (success or failure).
With our interest in #2744 continuing, it becomes useful to know if this condition (logged from from prim...Resolver root priming is logged at level INFO.
From BIND 9.18 and newer, root priming also logs the outcome (success or failure).
With our interest in #2744 continuing, it becomes useful to know if this condition (logged from from prime_done()) occurred because root priming failed, or because it succeeded but before we ran checkhints, the newly primed roots were no longer in cache any more (or perhaps there had been a problem writing them there?).
However, much much stuff is logged at level INFO in category Resolver and it's not uncommon for administrators to set the logging level higher for this category so that they're not overwhelmed by it.
Since logging of priming is (quite reasonably) logged at INFO because it's a routine operation, we would like to request that logging of the failure to prime, be logged at NOTICE since this exceptional.
This will also make it easier to catch this logged message on a busy resolver (for it is the busy ones that most often encounter #2744 )
See [Support ticket #20909](https://support.isc.org/Ticket/Display.html?id=20909) for an example of why we need this.https://gitlab.isc.org/isc-projects/bind9/-/issues/3482Clarify ACLs and their interaction in the ARM2022-10-06T10:22:09ZPetr Špačekpspacek@isc.orgClarify ACLs and their interaction in the ARMThings I noticed in the ARM which I think are in need of clarification:
- allow-query*/allow-recursion* descriptions are just confusing.
- Sections [8.1.4. Address Match Lists](https://bind9.readthedocs.io/en/v9_19_3/reference.html#addr...Things I noticed in the ARM which I think are in need of clarification:
- allow-query*/allow-recursion* descriptions are just confusing.
- Sections [8.1.4. Address Match Lists](https://bind9.readthedocs.io/en/v9_19_3/reference.html#address-match-lists) and [7.1. Access Control Lists](https://bind9.readthedocs.io/en/v9_19_3/chapter7.html#access-control-lists) overlap and really should be merged
## Summary how I believe ACLs work
**Proceed with caution!**
Different sets of ACLs affect different queries, depending on whether the server has data in cache or if the data are in an authoritative zone on the server. Caveats:
- `type mirror` zone still counts as cache (cache ACLs apply, I think)
- `type static-stub` zones are not queriable without recursion desired bit anyway
With this in mind, I _believe_ BIND checks this:
- blackhole acl checks client address and drops packets on the floor - highest priority
- for queries into authoritative zones check ONLY:
- allow-query
- allow-query-on
- for queries into cache
- for data present in the cache check:
- recursion (yes, even if no recursion is happening yet)
- allow-query-cache
- allow-query-cache-on
- _(no allow-query(-on) ACL here!)_
- for data NOT present in the cache **additionally** check (before a fetch is started):
- allow-recursion
- allow-recursion-on
- I.e. a query which triggers a fetch from elsewhere must match all the allow-query-cache(-on) and allow-recursion(-on)
Fun fact: Data stay in cache even if `recursion yes;` is changed to `recursion no;`. Subsequent `rndc reconfig` will create configuration with data in cache but without any means to access it. Turning it back on will provide access to the old cache content.
## Text problems/suggestions
- allow-query-cache(-on) does not "effectively control recursion". It control access to cache data WITHOUT doing recursion: I.e. queries allowed by these ACLs can get content of the cache but not necessarily trigger recursion for things which are missing in the cache. (equivalent of `dig +norecurse` queries)
- allow-recursion(-on) control what queries do trigger recursion for data not available in cache
- allow-query - also covers updates (must intersect with allow-update?)
Relevant log message:
```
update 'example.com/IN' denied due to allow-query
```
(Implementation wise it makes sense because of prerequisites in the update messages, but it is not mentioned in the ARM. Basically it is impossible to make write-only client - no big deal, but better to mention that the ACLs must intersect.)https://gitlab.isc.org/isc-projects/bind9/-/issues/3480unsupported optional libraries2022-08-05T11:42:43ZPeter Daviesunsupported optional librariesBased on the experiences of a user that experienced problems compiling the
BIND man pages with an older version of Sphinx.
There are a number of optional libraries that BIND can be compiled with and it
isn't always clear which v...Based on the experiences of a user that experienced problems compiling the
BIND man pages with an older version of Sphinx.
There are a number of optional libraries that BIND can be compiled with and it
isn't always clear which versions current BIND releases support.
In so far as the configure script doesn't check the versions of these we might
want to consider adding a note to the ARM, if one does not already exist.
I found the following:
sphinx, json-c, zlib, lmdb, libmaxminddb, libfstrm, libprotobuf-c, libidn2,
readline, libedit
There may be morehttps://gitlab.isc.org/isc-projects/bind9/-/issues/3470Implement RESINFO type2022-07-26T21:55:40ZMark AndrewsImplement RESINFO typehttps://datatracker.ietf.org/doc/html/draft-reddy-add-resolver-info-05
The draft still needs work and I've recommended that *ws be treated as
zero length ws when encoding to wire.
A type code has yet to be assigned.
We should also rej...https://datatracker.ietf.org/doc/html/draft-reddy-add-resolver-info-05
The draft still needs work and I've recommended that *ws be treated as
zero length ws when encoding to wire.
A type code has yet to be assigned.
We should also reject invalid json when parsing the master file.https://gitlab.isc.org/isc-projects/bind9/-/issues/3455Zone statistics, esp for tracking transfers, update status2023-03-27T14:30:40ZGreg ChoulesZone statistics, esp for tracking transfers, update statusIt is important to know how your server is performing! Complex and large authoritative server setups - many secondaries, zones and data records - have a lot of plates spinning, or balls in the air (tasks running) and understanding how ea...It is important to know how your server is performing! Complex and large authoritative server setups - many secondaries, zones and data records - have a lot of plates spinning, or balls in the air (tasks running) and understanding how each of these tasks is behaving will help operators to know what is 'normal' and what is not.
This issue is to request additional counters in BIND, to provide metrics on various objects and activities. Here are some initial thoughts on what those might be:
1. Total current no. of zones. This is the sum of numbers of zones of all types (see below).
2. Current no. of primary zones, added by zone statements or dynamically using 'addzone'.
3. Current no. of secondary zones..
4. Current no. of forward zones..
5. Current no. of stub zones..
6. Current no. of static-stub zones..
7. Current no. of catalog zones (catz)
8. Current no. of rpz zones
9. No. of SOA queries currently in progress - similar in concept to the current no. of recursive clients. (this might exist already?
10. Histogram of times to resolve SOA zone refresh queries - similar to the query RTT buckets in `named.stats`.
- 0-0.999ms
- 1-4.999ms
- 5-9.999ms
- 10-19.999ms
- 20-49.999ms
- 50-99.999ms
- 100-499.99ms
- 500ms-infinity
11. No. of AXFRs currently in progress
12. No. of IXFRs currently in progress
13. Histogram of times XFRs have been in progress (the bucket sizes are a guess, not based on empirical data)
- 0-0.999s
- 1-4.999s
- 5-9.999s
- 10-19.999s
- 20-49.999s
- 50-99.999s
- 100-499.99s
- 500s-infinityNot plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3442Remove the "max-zone-ttl" option (on options/zone level)2024-03-01T04:29:28ZMichał KępieńRemove the "max-zone-ttl" option (on options/zone level)The `max-zone-ttl` option should now be configured as part of
`dnssec-policy`. The option with the same name on `options` and `zone`
levels should be removed.
!6542 deprecated that sort of use. This issue serves as a reminder to
ultim...The `max-zone-ttl` option should now be configured as part of
`dnssec-policy`. The option with the same name on `options` and `zone`
levels should be removed.
!6542 deprecated that sort of use. This issue serves as a reminder to
ultimately remove the relevant code altogether.
The due date for this issue has been set to an arbitrary date that is
presumed to fall within the BIND 9.21 development cycle.
See #2918, !6542Not plannedEvan HuntEvan Hunt2024-08-01https://gitlab.isc.org/isc-projects/bind9/-/issues/3435Develop doc helper for upgrades across many versions2022-07-04T16:27:52ZPetr Špačekpspacek@isc.orgDevelop doc helper for upgrades across many versions(This request describes a pipe dream. Of course there are various intermediate levels which are also useful even without full coverage.)
### Description
Use-case: People upgrading from arbitrarily old version want to know what changed:...(This request describes a pipe dream. Of course there are various intermediate levels which are also useful even without full coverage.)
### Description
Use-case: People upgrading from arbitrarily old version want to know what changed:
- Configuration grammar
- Defaults
- Behavior
We have people upgrading from _ancient_ versions, and going through all the release notes is pain for everyone involved, especially if something changed repeated in meanwhile.
### Request
Ultimately, an ability to "diff" grammar, defaults, and preferably also notes about changes between versions (= unicorns and rainbow).
To do that, we need:
- [X] programmatic access to grammar - available in doc/misc/parsegrammar.py
- [ ] programmatic access to defaults - parser producing machine-readable output is missing
- [ ] rst directive `.. versionchanged:: versionnumber` in the ARM describing when we changed behavior and how ([Sphinx docs](https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-versionchanged)
With that in place we should be able to pick two versions and generate "diff" in terms of added/removed/deprecated configuration statements, changed defaults, and notes pointing to feature changes.
### Links / referencesNot planned