BIND issueshttps://gitlab.isc.org/isc-projects/bind9/-/issues2023-11-02T16:58:15Zhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1878Improve mirror zone implementation by using an iterator for the zone validati...2023-11-02T16:58:15ZCathy AlmondImprove mirror zone implementation by using an iterator for the zone validation stepTying loosely into [Support ticket #16268](https://support.isc.org/Ticket/Display.html?id=16268)
We were never able to replicate problems with client resolution during mirror zone refresh (using IXFR), and neither were we able to replic...Tying loosely into [Support ticket #16268](https://support.isc.org/Ticket/Display.html?id=16268)
We were never able to replicate problems with client resolution during mirror zone refresh (using IXFR), and neither were we able to replicated 'slowness' of updating the zone itself. The suspicion is now that the reported issues were due to `something else` and what we were seeing was symptom not cause.
However, along the way (and in #1802 and #1803) what was re-exposed is that the validation step for mirror zone updates doesn't take place within an iterator, so it's anti-social, in that it doesn't relinquish the CPU/thread it's working on until it's finished. This is documented as a known feature of the mirror zone implementation, and most of the time it really should not matter (it doesn't take long to validate the entire root zone, which is what the mirror zone implementation was designed for).
This issue ticket is a placeholder to note that we considered this something that we'd like to do, although it's not burningly urgent in the bigger picture of Things That Need To Be Done.
(We also uncovered that the validation step takes place against the entire zone, for each increment being applied (increment = bundled set of changes between SOA start and end RRs, not each individual change), so is potentially inefficient when pulling IXFRs rather than AXFRs from the root servers - but this has to be balanced against the rate of flux of the mirror zone (low for the root zone), so it's probably not worth tackling this too. )Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1871RNDC command to dump stats2023-05-31T11:52:38ZVicky Riskvicky@isc.orgRNDC command to dump statsWe have a request, from a resolver operator, to be able to issue an rndc command to dump a file of statistics in some format that can be imported and processed by Prometheus. I think the use case is, the node might be in a remote pop wit...We have a request, from a resolver operator, to be able to issue an rndc command to dump a file of statistics in some format that can be imported and processed by Prometheus. I think the use case is, the node might be in a remote pop with limited connectivity and setting up on-going streaming may be difficult for some reason, or perhaps there isn't the access for that.
entered on behalf of cmosher@quad9.netBIND 9.19.xhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1864Commit 54fe75b9b76eba92efd0fc1cded4a0ac0adc0ba9 introduced a regression in 9.112021-10-05T12:17:40ZBrian ConryCommit 54fe75b9b76eba92efd0fc1cded4a0ac0adc0ba9 introduced a regression in 9.11This commit causes a problem in 9.11 when using `query-source` with `port 53`.
After a reconfig or reload the server will stop receiving data.
I've confirmed that 9.14 and later versions are not affected.
I've also confirmed that both ...This commit causes a problem in 9.11 when using `query-source` with `port 53`.
After a reconfig or reload the server will stop receiving data.
I've confirmed that 9.14 and later versions are not affected.
I've also confirmed that both 9.11 and 9.11-S are affected.
This was discovered from a customer ticket, though the customer has since removed the query source port configuration.Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1863dnssec-verify specify a current time2024-02-14T14:44:27ZPeter Daviesdnssec-verify specify a current time### Description
dnssec-verify specify a current time:
If we generate a signed zone via dnssec-signzone with an inception time in the future, then we have a way to fully verify it (including signature validity time checks) with dnssec-ve...### Description
dnssec-verify specify a current time:
If we generate a signed zone via dnssec-signzone with an inception time in the future, then we have a way to fully verify it (including signature validity time checks) with dnssec-verify before hand.
### Request
A command line parameter that allowed you to specify a current time to be used by dnssec-verify.
### Links / references
[GL #743 ](https://gitlab.isc.org/isc-projects/bind9/-/issues/743)
[RT #16466](https://support.isc.org/Ticket/Display.html?id=16466)Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1837[ISC-support #14369] Considering changing ECS caching strategy or negative data2024-03-13T13:17:48ZBrian Conry[ISC-support #14369] Considering changing ECS caching strategy or negative dataCurrently our ECS implementation caches all negative responses at the global scope.
A customer has requested that we store them at the learned scope.
RFC 7871 (Informational only) section 7.4, paragraph 4 notes that the original spec w...Currently our ECS implementation caches all negative responses at the global scope.
A customer has requested that we store them at the learned scope.
RFC 7871 (Informational only) section 7.4, paragraph 4 notes that the original spec was ambiguous and hints that the interpretation of scoping negative answers is more likely to be used in future protocol specifications (if any) than caching them globally.
> This issue is expected to be revisited in a future revision of the protocol, possibly blessing the mixing of positive and negative answers. There are implications for cache data structures that developers should consider when writing new ECS code.Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1836Return Extended EDNS Errors (EDE)2023-11-02T16:58:15ZMark AndrewsReturn Extended EDNS Errors (EDE)This is a Summary issue. Please open individual issues for individual code points.
There are 25 defined EDE codes. Some will be possible to return easily, others not so. See [RFC 8914](https://datatracker.ietf.org/doc/html/rfc8914) for...This is a Summary issue. Please open individual issues for individual code points.
There are 25 defined EDE codes. Some will be possible to return easily, others not so. See [RFC 8914](https://datatracker.ietf.org/doc/html/rfc8914) for explanation of the error codes.
* [ ] 0 - Other
* [ ] 1 - Unsupported DNSKEY Algorithm #2715
* [ ] 2 - Unsupported DS Digest Type #2715
* [x] 3 - Stale Answer #2267 (9.18.3, 9.19.1)
* [x] 4 - Forged Answer #3410 (9.19.5)
* [ ] 5 - DNSSEC Indeterminate #2715
* [ ] 6 - DNSSEC Bogus #2715
* [ ] 7 - Signature Expired #2715
* [ ] 8 - Signature Not Yet Valid #2715
* [ ] 9 - DNSKEY Missing #2715
* [ ] 10 - RRSIGs Missing #2715
* [ ] 11 - No Zone Key Bit Set #2715
* [ ] 12 - NSEC Missing #2715
* [ ] 13 - Cached Error
* [ ] 14 - Not Ready
* [x] 15 - Blocked #3410 (9.19.5)
* [x] 16 - Censored #3410 (9.19.5)
* [x] 17 - Filtered #3410 (9.19.5)
* [x] 18 - Prohibited (9.17.21), #3410 (9.19.5)
* [x] 19 - Stale NXDOMAIN Answer #2267 (9.18.3, 9.19.1)
* [ ] 20 - Not Authoritative
* [ ] 21 - Not Supported
* [ ] 22 - No Reachable Authority #2268
* [ ] 23 - Network Error
* [ ] 24 - Invalid DataNot plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1831Feature request: Separate NXDOMAIN cache with its own max-ncache-size2024-03-01T10:04:57ZCathy AlmondFeature request: Separate NXDOMAIN cache with its own max-ncache-sizeThis relates to PRSD DDoS attacks, and the effect on participating resolvers when the domain under onslaught is able to keep responding and does not die or rate-limit the resolvers.
The scenario is one in which a very large number of un...This relates to PRSD DDoS attacks, and the effect on participating resolvers when the domain under onslaught is able to keep responding and does not die or rate-limit the resolvers.
The scenario is one in which a very large number of unique names are being queried, the objective being to bypass cached NXDOMAINs in resolvers and to force every name to become a query to the authoritative servers for the domain (or hosting provider) that is being attacked.
Typically, the target servers will either die, or will commence rate-limiting their perceived attackers. In the case of a resolver, this will result in a large number of recursive queries being backlogged while they wait for the server responses that never arrive.
BIND uses fetch-limits to mitigate the non-responding servers scenario.
But in the situation where the servers never die or never rate-limit, the outcome is rather different. Resolvers that can cope with the increase in traffic (which usually isn't actually that much), instead see a rapid increase in memory consumption (and decrease in cache hits!) due to the NXDOMAIN responses that are received and then cached (never to be used again).
One mitigation for resolver operators has been to reduce max-ncache-ttl to silly small values - but the effectiveness of this depends on the structure of the cache nodes and how often opportunistic cache cleaning hits these nodes.
Yes, overmem (LRU-based logic) cache-cleaning will help with this, but for many, it is going to be at the expense of 'positive' cache content, and regular clients will start to suffer with more cache-misses, as well as cache churn increasing as negative and positive cache content keeps being 'swapped'.
Mark suggested keeping negative answers in a separate cache, where they could have their own max-ncache-size and churn all by themselves, without affecting main cache.
This sounds like A Good Idea - but one that we've never get got around to, as part of ongoing DDoS mitigation work.
(Also tagging this as 'Customer' since I can find many a customer ticket where customers have been bitten by this when one specific and well-known DNS hosting company have been under attack, and their servers never falter in sending back NXDOMAIN responses to their 'attackers').Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1793failed query to a `forward only` forwarder increments `serverquota` counter (...2023-11-02T16:58:14ZCathy Almondfailed query to a `forward only` forwarder increments `serverquota` counter (spilled due to server quota)As observed in [Support ticket #16297](https://support.isc.org/Ticket/Display.html?id=16297)
I was inspecting the stats output and was very surprised to see this:
` 13779 spilled due to server quota`
The server in questi...As observed in [Support ticket #16297](https://support.isc.org/Ticket/Display.html?id=16297)
I was inspecting the stats output and was very surprised to see this:
` 13779 spilled due to server quota`
The server in question does not have `fetches-per-server` configured, so this defaults to zero (unlimited). But yet...
Looking at the code - I suspect there's a failure mode that drops through the 'out' block in fctx_getaddresses() without resetting all_spilled (which starts at 'true').
```c
static isc_result_t
fctx_getaddresses(fetchctx_t *fctx, bool badcache) {
dns_rdata_t rdata = DNS_RDATA_INIT;
isc_result_t result;
dns_resolver_t *res;
isc_stdtime_t now;
unsigned int stdoptions = 0;
dns_forwarder_t *fwd;
dns_adbaddrinfo_t *ai;
bool all_bad;
dns_rdata_ns_t ns;
bool need_alternate = false;
bool all_spilled = true;
```
...
```c
/*
* If all of the addresses found were over the
* fetches-per-server quota, return the configured
* response.
*/
if (all_spilled) {
result = res->quotaresp[dns_quotatype_server];
inc_stats(res, dns_resstatscounter_serverquota);
}
```
This is a server that is using global forwarding, so we skip case 'normal_nses', which is where 'all_spilled' is normally reset from true to false during processing:
```c
if (fctx->fwdpolicy == dns_fwdpolicy_only)
goto out;
```
So I'm guessing that what's been 'counted' and then reported here, is failures in getting responses back from any of the global forwarders (which tallies quite nicely with the problem I'm investigating - even though this wasn't a counter I was expecting to see in the stats!).
The assumption seems to be if it's a failure for any other reason than fetch-limits, that something will reset the 'all_spilled' flag - it would appear that assumption is flawed for some configurations and situations. Could someone have a look at this please - it should be an easy one to fix.
I note that this has also been noticed before on bind-users:
https://lists.isc.org/pipermail/bind-users/2016-June/097011.html
I observed this in 9.11.15-S1, but the code path looks the same still on master.
Requested changes:
- [ ] fix serverquota counter
- [ ] add a new counter for specifically for situation when all forwarders have failedNot plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1784RFC9103: DNS Zone Transfer over TLS (XoT)2022-08-01T11:52:34ZPeter DaviesRFC9103: DNS Zone Transfer over TLS (XoT)### Description
The [RFC9103](https://datatracker.ietf.org/doc/html/rfc9103) describes the use of TLS to encrypt zones transfers in order to provide confidentiality, known as XFR-over-TLS (XoT). The standard has been adopted by the DPRI...### Description
The [RFC9103](https://datatracker.ietf.org/doc/html/rfc9103) describes the use of TLS to encrypt zones transfers in order to provide confidentiality, known as XFR-over-TLS (XoT). The standard has been adopted by the DPRIVE WG.
### Feature Request
The feature request is for BIND to support XFR-over-TLS as described in the above RFC. This will obviously be dependent on DoT (RFC7858) being implemented in BIND. The specific aspects of the XoT implementation that are desired are:
- [x] * Support for both AXFR and IXFR
- [x] * XoT requires `dot` ALPN token to be negotiated (See: #2794)
- [x] * XoT requires TLSv1.3 or higher (See: #2795, and related #2796)
- [x] * Support for XFR-over-TLS both when BIND is acting as a primary and a secondary
- [X] * XFR-over-TLS (XoT): Primaries need to be able to restrict XFR to just TLS (#2776)
- [ ] * Related: Replace `tcp-only` with a more generic option (#2992)
- [X] * Support for authentication of TLS connections via X.509 certificates (Strict TLS and Mutual TLS)
- Related MR: !5600
- [X] * A TLS contexts cache needs to be implemented for contexts reuse and fast retrieval of the data associated with contexts (like CA intermediates chain): #3067, !5672
- [x] * Add remote TLS certificate verification support, implement Strict and Mutual TLS authentication (#3163)
- [ ] * Optimisation of TCP/TLS connections such that persistent connections can be re-used for multiple IXFRs for the same zone, and also IXFRs for different zones.
- [X] Client TLS session resumption support: !6274
### Related issues/bugs
- [x] * #2450 - Follow-up from "Draft: Resolve "XoT xfrin""
- See !5602 which addresses the most important points from the issue
- [x] * #2884 - Sometimes dig aborts on an AXFR query over TLS
- [X] * #2986 - TLS not working on the client-side (dig/named)
- [X] * #3004 - dig and named crash when receiving XFR over TLS
See RT [#16298](https://support.isc.org/Ticket/Display.html?id=16298)BIND 9.19.xArtem BoldarievArtem Boldarievhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1776BIND 9.16 and cache node locks for name cleaning vs. 'the thundering herd'2021-10-05T12:07:29ZCathy AlmondBIND 9.16 and cache node locks for name cleaning vs. 'the thundering herd'From [Support ticket #16212](https://support.isc.org/Ticket/Display.html?id=16212)
During investigations of intermittent 'brownouts' - periods in which named seemingly stops actioning client queries for a short period, and then resumes ...From [Support ticket #16212](https://support.isc.org/Ticket/Display.html?id=16212)
During investigations of intermittent 'brownouts' - periods in which named seemingly stops actioning client queries for a short period, and then resumes processing a second or two later (yes, delays of seconds not ms from this) we 'caught' one interesting scenario on BIND 9.16 in which it appeared that the vast majority of the active threads (netmgr and taskmgr both - so both client queries being answered from cache, AND client queries for which recursion had just taken place) were competing for the same cache node lock.
The pstack output demonstrating the problem was automatically triggered by monitoring for anomalies in inbound versus outbound network traffic.
The symptoms when this issue occurs are that:
* Outbound client-facing traffic rates plummet (well below the proportion that you would expect to see if it was only cache-misses not being serviced
* Recursive query rates plummet too
* CPU use increases - but in user space not in system space
* Recursive clients backlog increases (and may hit the limit)
* Fetchlimits may be triggered (we suspect this, and its predecessor are symptom not cause however, although triggering fetchlimits will exacerbate the situation, both from the client perspective, and as increased traffic rates as clients retry/re-send.
What we saw in the pstacks was that the majority netmgr threads (these answer directly from cache) were attempting to get a write lock on the node - for example:
```
Thread 74 (Thread 0x7f3ff366e700 (LWP 11713)):
#0 isc_rwlock_lock (rwl=rwl@entry=0x7f3f59523980, type=type@entry=isc_rwlocktype_write) at rwlock.c:57
#1 0x000000000051d826 in decrement_reference (rbtdb=rbtdb@entry=0x7f3fc6457010, node=node@entry=0x7f3eace34510, least_serial=least_serial@entry=0, nlock=nlock@entry=isc_rwlocktype_read, tlock=tlock@entry=isc_rwlocktype_none, pruning=pruning@entry=false) at rbtdb.c:2040
#2 0x00000000005215bf in detachnode (db=0x7f3fc6457010, targetp=targetp@entry=0x7f3ff366da88) at rbtdb.c:5352
#3 0x00000000005217be in rdataset_disassociate (rdataset=<optimized out>) at rbtdb.c:8691
#4 0x00000000005657e8 in dns_rdataset_disassociate (rdataset=rdataset@entry=0x7f3fad30cf28) at rdataset.c:111
#5 0x00000000004ebb21 in msgresetnames (first_section=0, msg=0x7f3fad2e1a50, msg@entry=0x7f3fad30b5f0) at message.c:438
#6 msgreset (msg=msg@entry=0x7f3fad2e1a50, everything=everything@entry=false) at message.c:524
#7 0x00000000004ec95a in dns_message_reset (msg=0x7f3fad2e1a50, intent=intent@entry=1) at message.c:760
#8 0x00000000004797ba in ns_client_endrequest (client=0x7f3fae5b8550) at client.c:229
#9 ns__client_reset_cb (client0=0x7f3fae5b8550) at client.c:1586
#10 0x0000000000632989 in isc_nmhandle_unref (handle=handle@entry=0x7f3fae5b83e0) at netmgr.c:1158
#11 0x0000000000632c30 in isc__nm_uvreq_put (req0=req0@entry=0x7f3ff366dbb8, sock=<optimized out>) at netmgr.c:1291
#12 0x00000000006357c4 in udp_send_cb (req=<optimized out>, status=<optimized out>) at udp.c:465
#13 0x00007f3ff5375153 in uv__udp_run_completed () from /lib64/libuv.so.1
#14 0x00007f3ff53754d3 in uv__udp_io () from /lib64/libuv.so.1
#15 0x00007f3ff5367c43 in uv_run () from /lib64/libuv.so.1
#16 0x0000000000632fda in nm_thread (worker0=0x138e3e0) at netmgr.c:481
#17 0x00007f3ff4f39e65 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f3ff484488d in clone () from /lib64/libc.so.6
```
A handful of threads are attempting to get a read lock on the same node - for example:
```
Thread 59 (Thread 0x7f3feab0e700 (LWP 11734)):
#0 0x00007f3ff4f3d144 in pthread_rwlock_rdlock () from /lib64/libpthread.so.0
#1 0x000000000063cc6e in isc_rwlock_lock (rwl=0x7f3f59523980, type=type@entry=isc_rwlocktype_read) at rwlock.c:48
#2 0x00000000005129c6 in rdataset_getownercase (rdataset=<optimized out>, name=0x7f3feaaffde0) at rbtdb.c:9770
#3 0x000000000056620a in towiresorted (rdataset=rdataset@entry=0x7f3ec42dee70, owner_name=owner_name@entry=0x7f3ec42dd0a0, cctx=<optimized out>, target=<optimized out>, order=<optimized out>, order_arg=order_arg@entry=0x7f3ec42b8718, partial=true, options=1, countp=0x7f3feab005dc, state=<optimized out>) at rdataset.c:444
#4 0x0000000000566e3f in dns_rdataset_towirepartial (rdataset=rdataset@entry=0x7f3ec42dee70, owner_name=owner_name@entry=0x7f3ec42dd0a0, cctx=<optimized out>, target=<optimized out>, order=<optimized out>, order_arg=order_arg@entry=0x7f3ec42b8718, options=<optimized out>, options@entry=1, countp=<optimized out>, countp@entry=0x7f3feab005dc, state=<optimized out>, state@entry=0x0) at rdataset.c:565
#5 0x00000000004ecc71 in dns_message_rendersection (msg=0x7f3ec42b8550, sectionid=sectionid@entry=1, options=options@entry=6) at message.c:2086
#6 0x00000000004780f3 in ns_client_send (client=client@entry=0x7f3ec5d4b510) at client.c:555
#7 0x0000000000485b7c in query_send (client=0x7f3ec5d4b510) at query.c:552
#8 0x000000000048de23 in ns_query_done (qctx=qctx@entry=0x7f3feab09a70) at query.c:10921
#9 0x000000000048f76d in query_respond (qctx=0x7f3feab09a70) at query.c:7414
#10 query_prepresponse (qctx=qctx@entry=0x7f3feab09a70) at query.c:9913
#11 0x000000000049181c in query_gotanswer (qctx=qctx@entry=0x7f3feab09a70, res=res@entry=0) at query.c:6836
#12 0x0000000000493a22 in query_lookup (qctx=qctx@entry=0x7f3feab09a70) at query.c:5617
#13 0x00000000004950f6 in query_zone_delegation (qctx=0x7f3feab09a70) at query.c:8003
#14 query_delegation (qctx=qctx@entry=0x7f3feab09a70) at query.c:8031
#15 0x0000000000491a1a in query_gotanswer (qctx=qctx@entry=0x7f3feab09a70, res=res@entry=65565) at query.c:6842
#16 0x0000000000493a22 in query_lookup (qctx=qctx@entry=0x7f3feab09a70) at query.c:5617
#17 0x0000000000494036 in ns__query_start (qctx=qctx@entry=0x7f3feab09a70) at query.c:5493
#18 0x000000000048de05 in ns_query_done (qctx=qctx@entry=0x7f3feab09a70) at query.c:10853
#19 0x0000000000492420 in query_dname (qctx=<optimized out>) at query.c:9806
#20 query_gotanswer (qctx=qctx@entry=0x7f3feab09a70, res=res@entry=65568) at query.c:6872
#21 0x0000000000493a22 in query_lookup (qctx=qctx@entry=0x7f3feab09a70) at query.c:5617
#22 0x00000000004950f6 in query_zone_delegation (qctx=0x7f3feab09a70) at query.c:8003
#23 query_delegation (qctx=qctx@entry=0x7f3feab09a70) at query.c:8031
#24 0x0000000000491a1a in query_gotanswer (qctx=qctx@entry=0x7f3feab09a70, res=res@entry=65565) at query.c:6842
#25 0x0000000000493a22 in query_lookup (qctx=qctx@entry=0x7f3feab09a70) at query.c:5617
#26 0x0000000000494036 in ns__query_start (qctx=qctx@entry=0x7f3feab09a70) at query.c:5493
#27 0x000000000048de05 in ns_query_done (qctx=qctx@entry=0x7f3feab09a70) at query.c:10853
#28 0x0000000000492420 in query_dname (qctx=<optimized out>) at query.c:9806
#29 query_gotanswer (qctx=qctx@entry=0x7f3feab09a70, res=res@entry=65568) at query.c:6872
#30 0x0000000000493a22 in query_lookup (qctx=qctx@entry=0x7f3feab09a70) at query.c:5617
#31 0x00000000004950f6 in query_zone_delegation (qctx=0x7f3feab09a70) at query.c:8003
#32 query_delegation (qctx=qctx@entry=0x7f3feab09a70) at query.c:8031
#33 0x0000000000491a1a in query_gotanswer (qctx=qctx@entry=0x7f3feab09a70, res=res@entry=65565) at query.c:6842
#34 0x0000000000493a22 in query_lookup (qctx=qctx@entry=0x7f3feab09a70) at query.c:5617
#35 0x0000000000494036 in ns__query_start (qctx=qctx@entry=0x7f3feab09a70) at query.c:5493
#36 0x0000000000494b26 in query_setup (client=client@entry=0x7f3ec5d4b510, qtype=<optimized out>) at query.c:5217
#37 0x0000000000497056 in ns_query_start (client=client@entry=0x7f3ec5d4b510) at query.c:11318
#38 0x000000000047b101 in ns__client_request (handle=<optimized out>, region=<optimized out>, arg=<optimized out>) at client.c:2209
#39 0x0000000000635462 in udp_recv_cb (handle=<optimized out>, nrecv=48, buf=0x7f3feab0ab00, addr=<optimized out>, flags=<optimized out>) at udp.c:329
#40 0x00007f3ff53755db in uv__udp_io () from /lib64/libuv.so.1
#41 0x00007f3ff53779c8 in uv__io_poll () from /lib64/libuv.so.1
#42 0x00007f3ff5367c70 in uv_run () from /lib64/libuv.so.1
#43 0x0000000000632fda in nm_thread (worker0=0x13926e8) at netmgr.c:481
#44 0x00007f3ff4f39e65 in start_thread () from /lib64/libpthread.so.0
#45 0x00007f3ff484488d in clone () from /lib64/libc.so.6
```
Meanwhile, the threads run by taskmgr (this bunch would have recursed) were attempting to get write locks (unsurprisingly, although depending on the node and the client query, I guess it's also possible that one might want to get a read lock):
Here's a writer:
```
Thread 50 (Thread 0x7f3fe587b700 (LWP 11746)):
#0 isc_rwlock_lock (rwl=rwl@entry=0x7f3f59523980, type=type@entry=isc_rwlocktype_write) at rwlock.c:57
#1 0x000000000051d826 in decrement_reference (rbtdb=rbtdb@entry=0x7f3fc6457010, node=node@entry=0x7f3eace34510, least_serial=least_serial@entry=0, nlock=nlock@entry=isc_rwlocktype_read, tlock=tlock@entry=isc_rwlocktype_none, pruning=pruning@entry=false) at rbtdb.c:2040
#2 0x00000000005215bf in detachnode (db=0x7f3fc6457010, targetp=0x7f3fe587acc0) at rbtdb.c:5352
#3 0x00000000004bdd83 in dns_db_detachnode (db=<optimized out>, nodep=nodep@entry=0x7f3fe587acc0) at db.c:588
#4 0x00000000004804cb in qctx_clean (qctx=qctx@entry=0x7f3fe587a830) at query.c:5097
#5 0x000000000048db5a in ns_query_done (qctx=qctx@entry=0x7f3fe587a830) at query.c:10834
#6 0x000000000048f76d in query_respond (qctx=0x7f3fe587a830) at query.c:7414
#7 query_prepresponse (qctx=qctx@entry=0x7f3fe587a830) at query.c:9913
#8 0x000000000049181c in query_gotanswer (qctx=qctx@entry=0x7f3fe587a830, res=res@entry=0) at query.c:6836
#9 0x0000000000496870 in query_resume (qctx=0x7f3fe587a830) at query.c:6134
#10 fetch_callback (task=<optimized out>, event=0x7f3ead5c9c18) at query.c:5716
#11 0x000000000064007a in dispatch (threadid=<optimized out>, manager=<optimized out>) at task.c:1152
#12 run (queuep=<optimized out>) at task.c:1344
#13 0x00007f3ff4f39e65 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f3ff484488d in clone () from /lib64/libc.so.6
```
In this particular instance, every single one of the legacy i/o-handler threads was twiddling its thumbs (sitting on epoll_wait() ) - which is probably not too surprising, if no taskmgr workers are sending out queries to auth servers?
Doing stats on this particular capture (74 threads - 24x netmgr, 24x taskmgr, 24x legacy i/o plus 1 each main and the timer thread), we have:
33 instances of isc_rwlock_lock (rwl=rwl@entry=0x7f3f59523980
31 instances of rbtdb=rbtdb@entry=0x7f3fc6457010
30 instances of node=node@entry=0x7f3eace34510
It might be that it's possible to prove from the pstack output that this is a series of different names all attached to the same node, versus a single name that is expiring that all of the threads are attempting to clean-up simultaneously.
Either way, the locking is not working well in this situation - there's a lot of spinning in user space it would appear.
Hypotheses being tendered currently include:
* This scenario has always potentially existed, but using pthread-rwlocks amplifies it considerably
* Could this be a case where prefetching (enabled with default settings in this example) hits a surprise edge case?
* Is it possible we're seeing the after-effects of another delay which has resulted in late client query-response processing for something that has a very short TTL in cache?
* Is this a scenario where a client comes along and queries near-simultaneously (and probably quite innocently) for a lot of similar names under the same domain/apex very close to the time where they would all be naturally expiring from cache?
* Could it be that TTL=0 handling has broken in 9.16 with the introduction of netmgr (noting that TTL=0 responses from auth servers would be expected to be available solely to the clients that recursed and waited for the fetch completion - not to anyone who came along after the fetch had populated cache for the waiting client request to be fulfilled - this should all be in taskmgr and none of it in netmgr)?
* Do we perhaps have too many threads running (detected CPUs = 24)?BIND 9.19.xOndřej SurýOndřej Surýhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1773Consider including all compile flags in "named -V" output2023-11-02T16:58:14ZMichał KępieńConsider including all compile flags in "named -V" outputThe following discussion from !985 should be addressed:
- [ ] @michal started a [discussion](https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/985#note_125108): (+2 comments)
> Nit: I think this would be better written as...The following discussion from !985 should be addressed:
- [ ] @michal started a [discussion](https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/985#note_125108): (+2 comments)
> Nit: I think this would be better written as:
>
> echo "CFLAGS: $STD_CFLAGS $CFLAGS"
>
> because that is the order that Automake uses in the `Makefile`s it
> generates with `AM_CFLAGS` set to `$(STD_CFLAGS)`.
>
> Also, something for the future perhaps, but it would be nice to see
> *all* (effective) `CFLAGS`/`CPPFLAGS`/`LDFLAGS` in `named -V` output,
> not just those overridden using environment variables set when
> `./configure` is run.Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1771Refactor how we load librpz.so2023-11-02T16:58:14ZOndřej SurýRefactor how we load librpz.soCurrently, there are three ways how `librpz.so` could be linked into BIND 9. In BIND 9.17, the `dlopen()` is mandatory (via libltdl), so this needs little bit of refactoring.Currently, there are three ways how `librpz.so` could be linked into BIND 9. In BIND 9.17, the `dlopen()` is mandatory (via libltdl), so this needs little bit of refactoring.Not plannedOndřej SurýOndřej Surýhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1756Derive maximum zone TTL from zone contents2023-07-26T08:54:49ZMatthijs Mekkingmatthijs@isc.orgDerive maximum zone TTL from zone contentsThe dnssec-policy system should look into the Zone as the default method of finding the "maximum zone TTL".
Currently `max-zone-ttl` defaults to 24 h. This silently breaks RRs with larger TTLs if rollover is happening. This is rare, but...The dnssec-policy system should look into the Zone as the default method of finding the "maximum zone TTL".
Currently `max-zone-ttl` defaults to 24 h. This silently breaks RRs with larger TTLs if rollover is happening. This is rare, but still a bug.
TTL caps for caches
-------------------
* BIND v9_19_0 default for 7 days
* Knot Resolver 5.5.0: 6 days
Knot DNS 3.1 (authoritative) computes the value from zone data already.Not plannedMatthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/1746Clarifications in ARM about catalog zones2023-11-02T17:00:02ZDan MahoneyClarifications in ARM about catalog zonesAt the bottom of the catalog zones syntax in the 9.14 ARM, the following example is listed for a catalog zone:
```
masters.5960775ba382e7a4e09263fc06e7c00569b6a05c.zones.catalog.example. IN A 192.0.2.2
label.masters.5960775ba382e7a4e092...At the bottom of the catalog zones syntax in the 9.14 ARM, the following example is listed for a catalog zone:
```
masters.5960775ba382e7a4e09263fc06e7c00569b6a05c.zones.catalog.example. IN A 192.0.2.2
label.masters.5960775ba382e7a4e09263fc06e7c00569b6a05c.zones.catalog.example. IN AAAA 2001:db8::2
label.masters.5960775ba382e7a4e09263fc06e7c00569b6a05c.zones.catalog.example. IN TXT "tsig_key"
allow-query.5960775ba382e7a4e09263fc06e7c00569b6a05c.zones.catalog.example. IN APL 1:10.0.0.0/24
```
This is a sort of uncommon usage which is basically like:
```
zone "hash" {
masters { 192.0.2.2; 2001:db8::2 key tsig_key; };
allow-query 10.0.0.0/24;
}
```
It's odd that a zone would be configured with one master with a TSIG key, and one without. I think the intent was to show the flexibility here, but if you want to show this it would be better to specify a second zone example that includes the TSIG key.
For the various examples given in this file, it would also be helpful to show the equivalent named.conf clause, to help understanding of a new concept.Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1739Restore the GSS.framework (and KRB5.framework) on macOS2023-11-02T16:51:56ZOndřej SurýRestore the GSS.framework (and KRB5.framework) on macOSThe !985 has refactored GSSAPI usage and temporarily dropped use of GSS.framework (using gssapi and krb5 libraries instead of frameworks). Use of GSS.framework on macOS needs to be restored.The !985 has refactored GSSAPI usage and temporarily dropped use of GSS.framework (using gssapi and krb5 libraries instead of frameworks). Use of GSS.framework on macOS needs to be restored.Not plannedOndřej SurýOndřej Surýhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1733Additional parameter to minimal-responses2023-11-06T11:08:42ZPeter DaviesAdditional parameter to minimal-responses### Description
Description:
In order to minimise the data returned from a forwarder to a forwardee when the forwarder has "minimal-responses" enabled; including delegations, negative responses etc.
This is because, in a typical setup, t...### Description
Description:
In order to minimise the data returned from a forwarder to a forwardee when the forwarder has "minimal-responses" enabled; including delegations, negative responses etc.
This is because, in a typical setup, the forwardee will be configured to "forward only" and so has no use for additional or authority data.
Suggestions on how this might be enabled are:
1) If the query received by the forwarder has RD=1 (from the forwardee), additional/authority data is NOT returned, even for delegations and negative responses.
Note: The danger with this is there might be a use case where the forwardee needs some or all of this data, even if RD=1
or
2) An additional parameter for "minimal-responses", something like "always", which if configured on the forwarder would cause it to never return additional/authority data to the forwardee under any circumstances.
Note: I think this would involve changing its parameter type from boolean to integer(?) in order to introduce a third option.
see RT [#16200](https://support.isc.org/Ticket/Display.html?id=16200)Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1732"rndc loadkeys" and "rndc sign" provide no feedback on missing or unreadable ...2023-11-02T16:51:56ZMichael Graff"rndc loadkeys" and "rndc sign" provide no feedback on missing or unreadable keysWhen using inline-signing, issuing a "rndc loadkeys" where there are no keys, the key directory specified in the config is unreadable, or the key files themselves are unreadable due to permission issues, no error is logged.
This made de...When using inline-signing, issuing a "rndc loadkeys" where there are no keys, the key directory specified in the config is unreadable, or the key files themselves are unreadable due to permission issues, no error is logged.
This made debugging fairly difficult.Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1692Use include-what-you-use to trim down included header files2023-11-02T16:51:55ZOndřej SurýUse include-what-you-use to trim down included header files[include-what-you-use](https://include-what-you-use.org/) is a great tool that can be used to trim down the header files that we include in every file.
> "Include what you use" means this: for every symbol (type, function variable, or m...[include-what-you-use](https://include-what-you-use.org/) is a great tool that can be used to trim down the header files that we include in every file.
> "Include what you use" means this: for every symbol (type, function variable, or macro) that you use in foo.cc, either foo.cc or foo.h should #include a .h file that exports the declaration of that symbol. The include-what-you-use tool is a program that can be built with the clang libraries in order to analyze #includes of source files to find include-what-you-use violations, and suggest fixes for them.
>
> The main goal of include-what-you-use is to remove superfluous #includes. It does this both by figuring out what #includes are not actually needed for this file (for both .cc and .h files), and replacing #includes with forward-declares when possible.
Why this is useful?
Excerpt from [Why Include What You Use?](https://github.com/include-what-you-use/include-what-you-use/blob/master/docs/WhyIWYU.md)
* Faster Compiles
* Fewer Recompiles
* Allow Refactoring
* Self-documentation
* Dependency CuttingNot plannedOndřej SurýOndřej Surýhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1690BIND signer should enforce RFC6840 Section 5.112023-11-02T16:51:55ZOndřej SurýBIND signer should enforce RFC6840 Section 5.11There are couple issues I found when investigating #1689:
1. RSASHA256 CSK (SEP bit set), publish-only ED25519 ZSK
```
Verifying the zone using the following algorithms: RSASHA512.
Zone fully signed:
Algorithm: RSASHA512: KSKs: 1 active...There are couple issues I found when investigating #1689:
1. RSASHA256 CSK (SEP bit set), publish-only ED25519 ZSK
```
Verifying the zone using the following algorithms: RSASHA512.
Zone fully signed:
Algorithm: RSASHA512: KSKs: 1 active, 0 stand-by, 0 revoked
ZSKs: 0 active, 0 stand-by, 0 revoked
Algorithm: ED25519: KSKs: 0 active, 0 stand-by, 0 revoked
ZSKs: 0 active, 1 stand-by, 0 revoked
```
2. RSASHA256 CSK (SEP bit set), ED25519 ZSK (treated as CSK)
```
Verifying the zone using the following algorithms: RSASHA512 ED25519.
Zone fully signed:
Algorithm: RSASHA512: KSKs: 1 active, 0 stand-by, 0 revoked
ZSKs: 0 active, 0 stand-by, 0 revoked
Algorithm: ED25519: KSKs: 0 active, 0 stand-by, 0 revoked
ZSKs: 1 active, 0 stand-by, 0 revoked
```
3. Single ZSK key (no SEP bit) is treated as CSK
```
Verifying the zone using the following algorithms: ED25519.
Zone fully signed:
Algorithm: ED25519: KSKs: 0 active, 0 stand-by, 0 revoked
ZSKs: 1 active, 0 stand-by, 0 revoked
```
All of these scenarios should be rejected both by `dnssec-signzone` and inline signing.Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/1687It's not clear in the ARM what options resolver-nonbackoff-tries and resolver...2023-10-31T14:23:16ZCathy AlmondIt's not clear in the ARM what options resolver-nonbackoff-tries and resolver-retry-interval actually doThey've been added to the descriptions (but not the grammar) in BIND 9.11 ARM, then they're added to the grammar in newer versions of BIND.
I understand that they're now knobs that allow you to changed the default settings for these val...They've been added to the descriptions (but not the grammar) in BIND 9.11 ARM, then they're added to the grammar in newer versions of BIND.
I understand that they're now knobs that allow you to changed the default settings for these values (that have been around in BIND9 for a long time).
But the definition of them in the ARM is just the names of them expanded into words - there's no explanation of how they're used by resolvers in context. For example, I know that named will deal with multiple nameservers for a domain and will have to decide which one to query first (shortest SRTT), how long to wait for that one to respond, how soon to retry if it doesn't, and when to start querying the next server in the list...
An explanation of how this all works would be very helpful - in the ARM or in the KB. And then alongside that:
- why did we make it possible now to tweak these values?
- what could be the costs/benefits to changing the defaults?
- why specifically were these knobs added as part of the work done to introduce Serve Stale?
If it would be generally not useful and potentially detrimental to BIND performance to change the values, then perhaps we should have left these options as undocumented (or unavailable unless named was built with specific non-default options).
Question arose from Support Ticket [RT16171](https://support.isc.org/Ticket/Display.html?id=16171)BIND 9.19.x