ISC Open Source Projects issueshttps://gitlab.isc.org/groups/isc-projects/-/issues2023-09-08T14:05:18Zhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4249compile test binaries/libraries during make2023-09-08T14:05:18ZTom Krizekcompile test binaries/libraries during makeCurrently, test files in `bin/tests/system` which need to be compiled are not part of the typical `make` invocation. Instead, they're compiled when `make check` is invoked.
While this worked well with the legacy test runner, it makes th...Currently, test files in `bin/tests/system` which need to be compiled are not part of the typical `make` invocation. Instead, they're compiled when `make check` is invoked.
While this worked well with the legacy test runner, it makes things needlessly complicated for the pytest runner, and prevents use-cases such as running out-of-tree tests easily.
Compile the required test files as `noinst_` instead of `check_` to ensure they're compiled and available after `make` invocation.
Related #4246, #3810September 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)Tom KrizekTom Krizekhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4247Autosign resigning too fast after dnssec-settime calls2024-03-21T15:29:09ZMark AndrewsAutosign resigning too fast after dnssec-settime callsJob [#3570991](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3570991) failed for ce1db0017ee498c3c9e0d632b83113a344d25dda:
I've seen autosign fail in the ZSK roll test several times and it appears to be because dnssec-signzone is bei...Job [#3570991](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3570991) failed for ce1db0017ee498c3c9e0d632b83113a344d25dda:
I've seen autosign fail in the ZSK roll test several times and it appears to be because dnssec-signzone is being called in the same second as dnssec-settime. dnssec-signzone uses the phase `is set and is in the past` as the determinant of when things change. Should this be made `is set and is now or in the past`? Which behaviour is consistent with named's behaviour? Which behaviour is less error prone? If we don't change the behaviour we need to add sleeps to ensure the tests succeed.
```
-S This option enables smart signing, which instructs dnssec-signzone to search the key repository for keys that match the zone
being signed, and to include them in the zone if appropriate.
When a key is found, its timing metadata is examined to determine how it should be used, according to the following rules.
Each successive rule takes priority over the prior ones:
If no timing metadata has been set for the key, the key is published in the zone and used to sign the zone.
If the key's publication date is set and is in the past, the key is published in the zone.
If the key's activation date is set and is in the past, the key is published (regardless of publication date) and used to
sign the zone.
If the key's revocation date is set and is in the past, and the key is published, then the key is revoked, and the revoked
key is used to sign the zone.
If either the key's unpublication or deletion date is set and in the past, the key is NOT published or used to sign the
zone, regardless of any other metadata.
If the key's sync publication date is set and is in the past, synchronization records (type CDS and/or CDNSKEY) are
created.
If the key's sync deletion date is set and is in the past, synchronization records (type CDS and/or CDNSKEY) are removed.
```
```
2023-08-08 00:59:52 INFO:autosign I:autosign_tmp_x_6wkr8z:preparing ZSK roll
2023-08-08 00:59:52 INFO:autosign I:autosign_tmp_x_6wkr8z:ns1 A zone reload and thaw was started.
2023-08-08 00:59:52 INFO:autosign I:autosign_tmp_x_6wkr8z:ns1 Check the logs to see the result.
2023-08-08 00:59:56 INFO:autosign I:autosign_tmp_x_6wkr8z:revoking key to duplicated key ID
2023-08-08 00:59:56 INFO:autosign dnssec-settime: warning: Permissions on the file ns2/Kbar.+013+59973.private have changed from 0644 to 0600 as a result of this operation.
2023-08-08 00:59:56 INFO:autosign I:autosign_tmp_x_6wkr8z:ns2 A zone reload and thaw was started.
2023-08-08 00:59:56 INFO:autosign I:autosign_tmp_x_6wkr8z:ns2 Check the logs to see the result.
2023-08-08 00:59:56 INFO:autosign I:autosign_tmp_x_6wkr8z:waiting for changes to take effect
2023-08-08 01:00:01 INFO:autosign I:autosign_tmp_x_6wkr8z:checking former standby key 5259 is now active (53)
2023-08-08 01:00:01 INFO:autosign I:autosign_tmp_x_6wkr8z:failed
```
```
echo_i "preparing ZSK roll"
starttime=$($PERL -e 'print time(), "\n";')
oldfile=$(cat active.key)
oldid=$(keyfile_to_key_id "$(cat active.key)")
newfile=$(cat standby.key)
newid=$(keyfile_to_key_id "$(cat standby.key)")
$SETTIME -K ns1 -I now -D now+25 $oldfile > settime.out.test$n.1 || ret=1
$SETTIME -K ns1 -i 0 -S $oldfile $newfile > settime.out.test$n.2 || ret=1
# note previous zone serial number
oldserial=$($DIG $DIGOPTS +short soa . @10.53.0.1 | awk '{print $3}')
($RNDCCMD 10.53.0.1 freeze . 2>&1 | sed 's/^/ns1 /' | cat_i) || ret=1
cp ns1/root.db.signed ns1/root.db.1
$SIGNER -S -o . -O full -K ns1 -f ns1/root.db.signed ns1/root.db.1 > signing.root.out$n 2>&1 || ret=1
($RNDCCMD 10.53.0.1 thaw . 2>&1 | sed 's/^/ns1 /' | cat_i) || ret=1
sleep 4
echo_i "revoking key to duplicated key ID"
$SETTIME -R now -K ns2 Kbar.+013+59973.key > settime.out.test$n.3 || ret=1
($RNDCCMD 10.53.0.2 freeze bar. 2>&1 | sed 's/^/ns2 /' | cat_i) || ret=1
cp ns2/bar.db.signed ns2/bar.db
$SIGNER -S -o bar. -O full -K ns2 ns2/bar.db > signing.bar.out$n 2>&1 || ret=1
($RNDCCMD 10.53.0.2 thaw bar. 2>&1 | sed 's/^/ns2 /' | cat_i) || ret=1
echo_i "waiting for changes to take effect"
sleep 5
echo_i "checking former standby key $newid is now active ($n)"
ret=0
$DIG $DIGOPTS dnskey . @10.53.0.1 > dig.out.ns1.test$n || ret=1
grep 'RRSIG.*'" $newid "'\. ' dig.out.ns1.test$n > /dev/null || ret=1
n=$((n + 1))
if [ $ret != 0 ]; then echo_i "failed"; fi
status=$((status + ret))
```
```
08-Aug-2023 00:59:52.469 allocate new control connection
08-Aug-2023 00:59:52.469 received control channel command 'freeze .'
08-Aug-2023 00:59:52.469 loop exclusive mode: starting
08-Aug-2023 00:59:52.469 loop exclusive mode: started
08-Aug-2023 00:59:52.469 zone_dump: zone ./IN: enter
08-Aug-2023 00:59:52.469 loop exclusive mode: ending
08-Aug-2023 00:59:52.469 loop exclusive mode: ended
08-Aug-2023 00:59:52.469 freezing zone './IN': success
08-Aug-2023 00:59:52.469 freeing control connection
08-Aug-2023 00:59:52.497 dump_done: zone ./IN: enter
08-Aug-2023 00:59:52.497 zone_journal_compact: zone ./IN: target journal size 7662
08-Aug-2023 00:59:52.497 zone ./IN: dns_journal_compact: success
08-Aug-2023 00:59:52.509 allocate new control connection
08-Aug-2023 00:59:52.509 received control channel command 'thaw .'
08-Aug-2023 00:59:52.509 loop exclusive mode: starting
08-Aug-2023 00:59:52.509 loop exclusive mode: started
08-Aug-2023 00:59:52.509 zone ./IN: starting load
08-Aug-2023 00:59:52.509 zone_startload: zone ./IN: enter
08-Aug-2023 00:59:52.509 loop exclusive mode: ending
08-Aug-2023 00:59:52.509 loop exclusive mode: ended
08-Aug-2023 00:59:52.509 thawing zone './IN': success
08-Aug-2023 00:59:52.509 freeing control connection
08-Aug-2023 00:59:52.509 zone_loaddone: zone ./IN: enter
08-Aug-2023 00:59:52.509 zone ./IN: number of nodes in database: 5
08-Aug-2023 00:59:52.509 zone ./IN: loaded; checking validity
08-Aug-2023 00:59:52.509 dns_zone_verifydb: zone ./IN: enter
08-Aug-2023 00:59:52.509 zone ./IN: zone serial (2000042101) unchanged. zone may fail to transfer to secondaries.
08-Aug-2023 00:59:52.509 zone ./IN: replacing zone database
08-Aug-2023 00:59:52.509 calling free_rbtdb(.)
08-Aug-2023 00:59:52.509 done free_rbtdb(.)
08-Aug-2023 00:59:52.509 zone ./IN: loaded serial 2000042101 (DNSSEC signed)
08-Aug-2023 00:59:52.509 zone_postload: zone ./IN: done
08-Aug-2023 00:59:52.509 zone__settimer: zone ./IN: enter
```April 2024 (9.16.50, 9.16.50-S1, 9.18.26, 9.18.26-S1, 9.19.23)https://gitlab.isc.org/isc-projects/bind9/-/issues/4246use pytest runner to run system tests for out-of-tree tests2023-08-21T14:19:40ZTom Krizekuse pytest runner to run system tests for out-of-tree testsImplement support to run system tests for out of tree builds with the pytest system test runner.
This is one of the last uses of the legacy runner in our CI (along with OpenBSD and CentOS7 on ~"v9.18").
Related #3810Implement support to run system tests for out of tree builds with the pytest system test runner.
This is one of the last uses of the legacy runner in our CI (along with OpenBSD and CentOS7 on ~"v9.18").
Related #3810September 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)Tom KrizekTom Krizekhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4245Incorrect return values in rpz's addr and drop functions2023-08-31T15:03:41ZMark AndrewsIncorrect return values in rpz's addr and drop functions`addr` and `drop` should `return 0` after calling `setret` rather than `return 1` as the error has already been logged. This is fallout from the `set -e` changes.`addr` and `drop` should `return 0` after calling `setret` rather than `return 1` as the error has already been logged. This is fallout from the `set -e` changes.September 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)https://gitlab.isc.org/isc-projects/bind9/-/issues/4243_wait_for_stats errors not detected in ixfr system test2023-08-08T00:43:48ZMark Andrews_wait_for_stats errors not detected in ixfr system testSeptember 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)Mark AndrewsMark Andrewshttps://gitlab.isc.org/isc-projects/stork/-/issues/1125Update AUTHORS file2023-10-02T10:17:54ZSlawek FigielUpdate AUTHORS fileThe issue was found by @marcin during [1.12 sanity checks](https://gitlab.isc.org/isc-projects/stork/-/issues/1122#note_393096).
The AUTHORS file may require an update. For example: there are new features, like hooks, that Slawek implem...The issue was found by @marcin during [1.12 sanity checks](https://gitlab.isc.org/isc-projects/stork/-/issues/1122#note_393096).
The AUTHORS file may require an update. For example: there are new features, like hooks, that Slawek implemented but aren't listed under his name. There can be more.1.13Tomek MrugalskiTomek Mrugalskihttps://gitlab.isc.org/isc-projects/bind9/-/issues/4242[CVE-2023-4236] named may terminate unexpectedly under high DNS-over-TLS quer...2023-10-18T15:59:10ZCathy Almond[CVE-2023-4236] named may terminate unexpectedly under high DNS-over-TLS query load| Quick Links | :link: |
| ------------------------ | ------------------------------------ |
| Incident Manager: | @pspacek |
| Deputy Incident Manager: | @gre...| Quick Links | :link: |
| ------------------------ | ------------------------------------ |
| Incident Manager: | @pspacek |
| Deputy Incident Manager: | @greg |
| Public Disclosure Date: | 2023-09-20 |
| CVSS Score: | [7.5][cvss_score] |
| Security Advisory: | isc-private/printing-press!66 |
| Mattermost Channel: | [CVE-2023-4236][mattermost_url] |
| Support Ticket: | Salesforce case \#1086 |
| Release Checklist: | #4289 |
| Post-mortem Etherpad: | [postmortem-2023-09][postmortem_url] |
[cvss_score]: https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H&version=3.1
[mattermost_url]: https://mattermost.isc.org/isc/channels/cve-2023-4236-crash-after-tls-connection-termination
[postmortem_url]: https://pad.isc.org/p/postmortem-2023-09
:bulb: **Click [here][checklist_explanations] (internal resource) for general information about the security incident handling process.**
[checklist_explanations]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations
### Earlier Than T-5
- [x] [:link:][step_deputy] **(IM)** Pick a Deputy Incident Manager
- [x] [:link:][step_respond] **(IM)** Respond to the bug reporter
- [x] [:link:][step_etherpad] **(IM)** Create an Etherpad for post-mortem
- [x] [:link:][step_public_mrs] **(SwEng)** Ensure there are no public merge requests which inadvertently disclose the issue
- [x] [:link:][step_assign_cve_id] **(IM)** [Assign a CVE identifier](#note_394286)
- [x] [:link:][step_note_cve_info] **(SwEng)** Update this issue with the assigned CVE identifier and the CVSS score: [MM](https://mattermost.isc.org/isc/pl/6bq1nynyaff68drzmt8bq8stda)
- [x] [:link:][step_versions_affected] **(SwEng)** Determine the range of product [versions affected](https://gitlab.isc.org/isc-projects/bind9/-/issues/4242#note_393519) (including the Subscription Edition)
- [X] [:link:][step_workarounds] **(SwEng)** Determine whether workarounds for the problem exist
- :no_entry_sign: ~~**(SwEng)** If necessary, coordinate with other parties~~
- [x] [:link:][step_earliest] **(Support)** Prepare and send out "earliest" notifications
- [x] [:link:][step_advisory_mr] **(Support)** [Create a merge request for the Security Advisory and include all readily available information in it](isc-private/printing-press!66)
- :no_entry_sign: ~~**(SwEng)** Prepare a private merge request containing a system test reproducing the problem~~
- :no_entry_sign: ~~**(SwEng)** Notify Support when a reproducer is ready~~
- [x] [:link:][step_code_analysis] **(SwEng)** Prepare a detailed explanation of the code flow triggering the problem: see [here](https://gitlab.isc.org/isc-private/bind9/-/merge_requests/565)
- [x] [:link:][step_fix_mr] **(SwEng)** Prepare a private merge request with the [fix](https://gitlab.isc.org/isc-private/bind9/-/merge_requests/565)
- [x] [:link:][step_review_fix] **(SwEng)** Ensure the merge request with the fix is reviewed and has no outstanding discussions
- [x] [:link:][step_review_docs] **(Support)** [Review the documentation changes introduced by the merge request with the fix](https://gitlab.isc.org/isc-private/bind9/-/merge_requests/565#note_400226)
- :no_entry_sign: ~~**(SwEng)** Prepare backports of the merge request addressing the problem for all affected (and still maintained) branches of a given product~~
- [x] [:link:][step_finish_advisory] **(Support)** Finish preparing the Security Advisory
- [x] [:link:][step_meta_issue] **(QA)** [Create (or update) the private issue containing links to fixes & reproducers for all CVEs fixed in a given release cycle](isc-private/bind9#70)
- [x] [:link:][step_changes] **(QA)** (BIND 9 only) Reserve a block of `CHANGES` placeholders once the complete set of vulnerabilities fixed in a given release cycle is determined
- [x] [:link:][step_merge_fixes] **(QA)** Merge the CVE fixes in CVE identifier order
- [x] [:link:][step_patches] **(QA)** Prepare a standalone patch for the last stable release of each affected (and still maintained) product branch
- [x] [:link:][step_asn_releases] **(QA)** Prepare ASN releases (as outlined in the Release Checklist)
### At T-5
- [x] [:link:][step_send_asn] **(Support)** Send ASN to eligible customers
- [x] [:link:][step_preannouncement] **(Support)** [(BIND 9 only) Send a pre-announcement email to the *bind-announce* mailing list to alert users that the upcoming release will include security fixes](isc-private/printing-press!67)
### At T-4
- [x] [:link:][step_verify_asn] **(Support)** [Verify that all ASN-eligible customers have received the notification email](https://mattermost.isc.org/isc/pl/384wo14bmprm3yhfcbyjrmdz5r)
### At T-1
- [x] [:link:][step_check_customers] **(Support)** [Verify that any new or reinstated customers have received the notification email](https://mattermost.isc.org/isc/pl/384wo14bmprm3yhfcbyjrmdz5r)
- [x] [:link:][step_packager_emails] **(First IM)** [Send notifications to OS packagers](isc-private/printing-press!68)
### On the Day of Public Disclosure
- [x] [:link:][step_clearance] **(IM)** [Grant Support clearance to proceed with public release](https://mattermost.isc.org/isc/pl/8jq9z43tx78xbyb8g6di44dpnh)
- [x] [:link:][step_publish] **(Support)** Publish the releases (as outlined in the release checklist)
- [x] [:link:][step_matrix] **(Support)** (BIND 9 only) Add the new CVEs to the vulnerability matrix in the Knowledge Base
- [x] [:link:][step_publish_advisory] **(Support)** [Bump Document Version for the Security Advisory and publish it in the Knowledge Base](https://kb.isc.org/docs/cve-2023-4236)
- [x] [:link:][step_notifications] **(First IM)** [Send notification emails to third parties](isc-private/printing-press!70)
- [x] [:link:][step_mitre] **(First IM)** [Advise MITRE about the disclosed CVEs](https://gitlab.isc.org/isc-private/printing-press/-/merge_requests/66#note_403632)
- [x] [:link:][step_merge_advisory] **(First IM)** Merge the Security Advisory merge request
- [x] [:link:][step_embargo_end] **(IM)** Inform original reporter (if external) that the security disclosure process is complete
- [x] [:link:][step_customers] **(Support)** Inform customers a fix has been released
### After Public Disclosure
- :no_entry_sign: [:link:][step_postmortem] **(First IM)** Organize post-mortem meeting and make sure it happens
- [x] [:link:][step_tickets] **(Support)** Close support tickets
- :no_entry_sign: [:link:][step_regression] **(QA)** Merge a regression test reproducing the bug into all affected (and still maintained) branches
[step_deputy]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#pick-a-deputy-incident-manager
[step_respond]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#respond-to-the-bug-reporter
[step_etherpad]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-an-etherpad-for-post-mortem
[step_public_mrs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-there-are-no-public-merge-requests-which-inadvertently-disclose-the-issue
[step_assign_cve_id]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#assign-a-cve-identifier
[step_note_cve_info]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-this-issue-with-the-assigned-cve-identifier-and-the-cvss-score
[step_versions_affected]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-the-range-of-product-versions-affected-including-the-subscription-edition
[step_workarounds]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-whether-workarounds-for-the-problem-exist
[step_coordinate]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#if-necessary-coordinate-with-other-parties
[step_earliest]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-and-send-out-earliest-notifications
[step_advisory_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-a-merge-request-for-the-security-advisory-and-include-all-readily-available-information-in-it
[step_reproducer_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-containing-a-system-test-reproducing-the-problem
[step_notify_support]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#notify-support-when-a-reproducer-is-ready
[step_code_analysis]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-detailed-explanation-of-the-code-flow-triggering-the-problem
[step_fix_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-with-the-fix
[step_review_fix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-the-merge-request-with-the-fix-is-reviewed-and-has-no-outstanding-discussions
[step_review_docs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#review-the-documentation-changes-introduced-by-the-merge-request-with-the-fix
[step_backports]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-backports-of-the-merge-request-addressing-the-problem-for-all-affected-and-still-maintained-branches-of-a-given-product
[step_finish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#finish-preparing-the-security-advisory
[step_meta_issue]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-or-update-the-private-issue-containing-links-to-fixes-reproducers-for-all-cves-fixed-in-a-given-release-cycle
[step_changes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-reserve-a-block-of-changes-placeholders-once-the-complete-set-of-vulnerabilities-fixed-in-a-given-release-cycle-is-determined
[step_merge_fixes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-cve-fixes-in-cve-identifier-order
[step_patches]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-standalone-patch-for-the-last-stable-release-of-each-affected-and-still-maintained-product-branch
[step_asn_releases]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-asn-releases-as-outlined-in-the-release-checklist
[step_send_asn]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-asn-to-eligible-customers
[step_preannouncement]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-send-a-pre-announcement-email-to-the-bind-announce-mailing-list-to-alert-users-that-the-upcoming-release-will-include-security-fixes
[step_verify_asn]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#verify-that-all-asn-eligible-customers-have-received-the-notification-email
[step_check_customers]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#verify-that-any-new-or-reinstated-customers-have-received-the-notification-email
[step_packager_emails]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notifications-to-os-packagers
[step_clearance]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#grant-support-clearance-to-proceed-with-public-release
[step_publish]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#publish-the-releases-as-outlined-in-the-release-checklist
[step_matrix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-add-the-new-cves-to-the-vulnerability-matrix-in-the-knowledge-base
[step_publish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bump-document-version-for-the-security-advisory-and-publish-it-in-the-knowledge-base
[step_notifications]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notification-emails-to-third-parties
[step_mitre]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#advise-mitre-about-the-disclosed-cves
[step_merge_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-security-advisory-merge-request
[step_embargo_end]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#inform-original-reporter-if-external-that-the-security-disclosure-process-is-complete
[step_customers]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#inform-customers-a-fix-has-been-released
[step_postmortem]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#organize-post-mortem-meeting-and-make-sure-it-happens
[step_tickets]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#close-support-tickets
[step_regression]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-a-regression-test-reproducing-the-bug-into-all-affected-and-still-maintained-branches
---
### Summary
From the customer:
I was running dnsperf in a test environment to try and find
parameters to get the highest QPS rate. Originally tested our
custom build on CentOS 7. After it started crashing I switched to stock
named on a Fedora 37 test VM to reproduce.
dnsperf example:
```
dnsperf -d /usr/share/dnsperf/queryfile-example-current -S 1 -s 192.168.122.239 -q 15000 -c 48 -T 32 -m dot
```
This or a similar command would often run without issues. While I
was testing I'd interrupt dnsperf if QPS was lower than a previous
run to tweak parameters. This would eventually cause named to crash.
### BIND version used
<will request this>
### Steps to reproduce
See above
### What is the current *bug* behavior?
`named` dies horribly: The assert as logged is:
The assert is
```
../../../lib/isc/netmgr/netmgr.c:2921: REQUIRE(((uvreq) != ((void *)0) && ((const isc__magic_t *)(uvreq))->magic == ((('N') << 24 | ('M') << 16 | ('U') << 8 | ('R'))))) failed, back trace
```
### What is the expected *correct* behavior?
named lives forever (or at least until an operator decides to stop and restart it)
### Relevant configuration files
see internal only note
### Relevant logs and/or screenshots
This is from the backtrace from the core dump that was produced:
Stack trace of thread 35031:
#0 0x00007f40cc2afe5c __pthread_kill_implementation (libc.so.6 + 0x8ce5c)
#1 0x00007f40cc25fa76 raise (libc.so.6 + 0x3ca76)
#2 0x00007f40cc2497fc abort (libc.so.6 + 0x267fc)
#3 0x0000561a0d6fb575 assertion_failed.cold (named + 0x1c575)
#4 0x00007f40cce39a50 isc_assertion_failed (libisc-9.18.16.so + 0x39a50)
#5 0x00007f40cce296d9 isc___nm_uvreq_put (libisc-9.18.16.so + 0x296d9)
#6 0x00007f40cce29d23 isc__nm_async_sendcb (libisc-9.18.16.so + 0x29d23)
#7 0x00007f40cce2da04 process_netievent (libisc-9.18.16.so + 0x2da04)
#8 0x00007f40cce2e057 process_queue (libisc-9.18.16.so + 0x2e057)
#9 0x00007f40cce2e277 async_cb (libisc-9.18.16.so + 0x2e277)
#10 0x00007f40ccd08e23 uv__async_io.part.0 (libuv.so.1 + 0xae23)
#11 0x00007f40ccd25dcb uv__io_poll (libuv.so.1 + 0x27dcb)
#12 0x00007f40ccd0e62f uv_run (libuv.so.1 + 0x1062f)
#13 0x00007f40cce2e704 nm_thread (libisc-9.18.16.so + 0x2e704)
#14 0x00007f40cce646e9 isc__trampoline_run (libisc-9.18.16.so + 0x646e9)
#15 0x00007f40cc2ae12d start_thread (libc.so.6 + 0x8b12d)
#16 0x00007f40cc32fbc0 __clone3 (libc.so.6 + 0x10cbc0)
### Possible fixesSeptember 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)Artem BoldarievArtem Boldarievhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4240dnstap system test fails on dnstap output file2024-03-28T10:20:18ZTom Krizekdnstap system test fails on dnstap output fileSystem test `dnstap` [started](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3552551) [to](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3553512) [fail](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3554027) [very](https://gitla...System test `dnstap` [started](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3552551) [to](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3553512) [fail](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3554027) [very](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3554808) [frequently](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3557127) on ~"v9.16" in the [system:gcc:oraclelinux7:amd64](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3560021) job (specifically on `oraclelinux7`):
```
I:dnstap:checking unix socket message counts
I:dnstap:dnstap output file smaller than expected
I:dnstap:failed
I:dnstap:checking UDP message counts
I:dnstap:ns4 0 expected 4
I:dnstap:failed
I:dnstap:checking TCP message counts
I:dnstap:checking AUTH_QUERY message counts
I:dnstap:checking AUTH_RESPONSE message counts
I:dnstap:checking CLIENT_QUERY message counts
I:dnstap:ns4 0 expected 1
I:dnstap:failed
I:dnstap:checking CLIENT_RESPONSE message counts
I:dnstap:ns4 0 expected 1
I:dnstap:failed
I:dnstap:checking RESOLVER_QUERY message counts
I:dnstap:checking RESOLVER_RESPONSE message counts
I:dnstap:checking UPDATE_QUERY message counts
I:dnstap:ns4 0 expected 1
I:dnstap:failed
I:dnstap:checking UPDATE_RESPONSE message counts
I:dnstap:ns4 0 expected 1
I:dnstap:failed
I:dnstap:checking reopened unix socket message counts
I:dnstap:failed
I:dnstap:checking UDP message counts
I:dnstap:checking TCP message counts
I:dnstap:checking AUTH_QUERY message counts
I:dnstap:checking AUTH_RESPONSE message counts
I:dnstap:checking CLIENT_QUERY message counts
I:dnstap:checking CLIENT_RESPONSE message counts
I:dnstap:checking RESOLVER_QUERY message counts
I:dnstap:checking RESOLVER_RESPONSE message counts
I:dnstap:checking UPDATE_QUERY message counts
I:dnstap:checking UPDATE_RESPONSE message counts
I:dnstap:checking large packet printing
I:dnstap:checking 'rndc -roll <value>' (no versions)
I:dnstap:checking 'rndc -roll <value>' (versions)
I:dnstap:exit status: 7
```
The good news is that this isn't cause by any code change, as the test used to [pass before](https://gitlab.isc.org/isc-projects/bind9/-/pipelines/143730) on the very same commit, and then [started to fail](https://gitlab.isc.org/isc-projects/bind9/-/pipelines/143854) afterwards since 2023-07-28.
However, it is quite puzzling why the test started to fail so often - the underlying `oraclelinux-7-amd64` image was last built on 2023-07-20, and the failure wasn't occurring at first, so it can't be caused by an image rebuild / dependencies either.
I wasn't able to reproduce the issue locally on ~"v9.16" using our `oraclelinux-7-amd` container image in over a hundred runs. I also didn't manage to reproduce it in my local environment on ~"v9.19" in over a hundred runs either.
Unfortunately, [increasing the timeout](https://gitlab.isc.org/isc-projects/bind9/-/commit/af9f675bcfd758efa86571c7528692679ab5fca9) for the dnstap file to be ready to 10s up from 5s [doesn't seem to help](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3561965).September 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)https://gitlab.isc.org/isc-projects/bind9/-/issues/4238The mkeys system test can update the root zone too fast2023-08-31T15:09:17ZMark AndrewsThe mkeys system test can update the root zone too fastThe keys system test can update the root zone too fast
causing reloads to fail as the modification time of root.db
has not changed. Add sleep 1 before signing.The keys system test can update the root zone too fast
causing reloads to fail as the modification time of root.db
has not changed. Add sleep 1 before signing.September 2023 (9.16.44, 9.16.44-S1, 9.18.19, 9.18.19-S1, 9.19.17)https://gitlab.isc.org/isc-projects/bind9/-/issues/4234[CVE-2023-4408] Parsing large DNS messages may cause excessive CPU load2024-03-28T12:11:21ZShoham Danino[CVE-2023-4408] Parsing large DNS messages may cause excessive CPU load| Quick Links | :link: |
| ------------------------ | ------------------------------------------------------------------------------ |
| Incident Manage...| Quick Links | :link: |
| ------------------------ | ------------------------------------------------------------------------------ |
| Incident Manager: | @matthijs |
| Deputy Incident Manager: | @michal |
| Public Disclosure Date: | 2024-02-13 |
| CVSS Score: | [7.5][cvss_score] |
| Security Advisory: | isc-private/printing-press!76 |
| Mattermost Channel: | [CVE-2023-4408: O(n²) complexity in DNS message parsing logic][mattermost_url] |
| Support Ticket: | N/A |
| Release Checklist: | #4515 & #4555 |
[cvss_score]: https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H&version=3.1
[mattermost_url]: https://mattermost.isc.org/isc/channels/cve-2023-4408
:bulb: **Click [here][checklist_explanations] (internal resource) for general information about the security incident handling process.**
[checklist_explanations]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations
### Earlier Than T-5
- [x] [:link:][step_deputy] **(IM)** Pick a Deputy Incident Manager
- [x] [:link:][step_respond] **(IM)** [Respond to the bug reporter](#note_394703)
- [x] [:link:][step_public_mrs] **(SwEng)** Ensure there are no public merge requests which inadvertently disclose the issue
- [x] [:link:][step_assign_cve_id] **(IM)** [Assign a CVE identifier](#note_396170)
- [x] [:link:][step_note_cve_info] **(SwEng)** Update this issue with the assigned CVE identifier and the CVSS score
- [x] [:link:][step_versions_affected] **(SwEng)** [Determine the range of product versions affected (including the Subscription Edition)](#note_396176)
- [x] [:link:][step_workarounds] **(SwEng)** [Determine whether workarounds for the problem exist](#note_396179)
- [X] [:link:][step_coordinate] **(SwEng)** [If necessary, coordinate with other parties](#note_397738)
- [x] [:link:][step_earliest_prepare] **(Support)** Prepare "earliest" notification text and hand it off to Marketing
- [x] [:link:][step_earliest_send] **(Marketing)** Update "earliest" notification document in SF portal and send bulk email to earliest customers
- [x] [:link:][step_advisory_mr] **(Support)** [Create a merge request for the Security Advisory and include all readily available information in it](isc-private/printing-press!76)
- [x] [:link:][step_reproducer_mr] ~~**(SwEng)** [Prepare a private merge request containing a system test reproducing the problem](#note_416143)~~
- [x] [:link:][step_notify_support] **(SwEng)** [Notify Support when a reproducer is ready](https://mattermost.isc.org/isc/pl/i5jke6nbhffizdduy4opewc56w)
- [x] [:link:][step_code_analysis] **(SwEng)** [Prepare a detailed explanation of the code flow triggering the problem](#note_416180)
- [x] [:link:][step_fix_mr] **(SwEng)** [Prepare a private merge request with the fix](isc-private/bind9!560)
- [x] [:link:][step_review_fix] **(SwEng)** Ensure the merge request with the fix is reviewed and has no outstanding discussions
- [x] [:link:][step_review_docs] **(Support)** [Review the documentation changes introduced by the merge request with the fix](#note_417455)
- [x] [:link:][step_backports] **(SwEng)** Prepare backports of the merge request addressing the problem [for](isc-private/bind9!585) [all](isc-private/bind9!586) [affected](isc-private/bind9!587) (and still maintained) branches of a given product
- [x] [:link:][step_finish_advisory] **(Support)** [Finish preparing the Security Advisory](https://mattermost.isc.org/isc/pl/76qusomtej88tjkqagiu1hfroy)
- [x] [:link:][step_meta_issue] **(QA)** [Create (or update) the private issue containing links to fixes & reproducers for all CVEs fixed in a given release cycle](#4486)
- [x] [:link:][step_changes] **(QA)** (BIND 9 only) [Reserve a block of `CHANGES` placeholders once the complete set of vulnerabilities fixed in a given release cycle is determined](!8625)
- [x] [:link:][step_merge_fixes] **(QA)** Merge the CVE fixes in CVE identifier order
- [x] [:link:][step_patches] **(QA)** Prepare a standalone patch for the last stable release of each affected (and still maintained) product branch
- [x] [:link:][step_asn_releases] **(QA)** Prepare ASN releases (as outlined in the Release Checklist)
### At T-5
- [x] [:link:][step_asn_documents] **(Marketing)** Update the text on the T-5 (from the Printing Press project) and "earliest" ASN documents in the SF portal
- [x] [:link:][step_asn_links] **(Marketing)** (BIND 9 only) Update the BIND -S information document in SF with download links to the new versions
- [x] [:link:][step_asn_send] **(Marketing)** Bulk email eligible customers to check the SF portal
- [x] [:link:][step_preannouncement] **(Marketing)** (BIND 9 only) Send a pre-announcement email to the *bind-announce* mailing list to alert users that the upcoming release will include security fixes
### At T-1
- [x] [:link:][step_packager_emails] **(First IM)** Send notifications to OS packagers
### On the Day of Public Disclosure
- [x] [:link:][step_clearance] **(IM)** [Grant QA & Marketing clearance to proceed with public release](https://mattermost.isc.org/isc/pl/hg9aypmxnjnjzc7ktwfo6xcpky)
- [x] [:link:][step_publish] **(QA/Marketing)** Publish the releases (as outlined in the release checklist)
- [x] [:link:][step_matrix] **(Support)** (BIND 9 only) Add the new CVEs to the vulnerability matrix in the Knowledge Base
- [x] [:link:][step_publish_advisory] **(Support)** Bump Document Version for the Security Advisory and publish it in the Knowledge Base
- [x] [:link:][step_notifications] **(First IM)** Send notification emails to third parties
- [x] [:link:][step_mitre] **(First IM)** Advise MITRE about the disclosed CVEs
- [x] [:link:][step_merge_advisory] **(First IM)** Merge the Security Advisory merge request
- [x] [:link:][step_embargo_end] **(IM)** [Inform original reporter (if external) that the security disclosure process is complete](#note_436560)
- [x] [:link:][step_asn_clear] **(Marketing)** Update the SF portal to clear the ASN
- [x] [:link:][step_customers] **(Marketing)** Email ASN recipients that the embargo is lifted
### After Public Disclosure
- [x] [:link:][step_regression] **(QA)** ~~[Merge a regression test reproducing the bug into all affected (and still maintained) branches](#note_416143)~~
[step_deputy]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#pick-a-deputy-incident-manager
[step_respond]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#respond-to-the-bug-reporter
[step_public_mrs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-there-are-no-public-merge-requests-which-inadvertently-disclose-the-issue
[step_assign_cve_id]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#assign-a-cve-identifier
[step_note_cve_info]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-this-issue-with-the-assigned-cve-identifier-and-the-cvss-score
[step_versions_affected]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-the-range-of-product-versions-affected-including-the-subscription-edition
[step_workarounds]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#determine-whether-workarounds-for-the-problem-exist
[step_coordinate]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#if-necessary-coordinate-with-other-parties
[step_earliest_prepare]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-earliest-notification-text-and-hand-it-off-to-marketing
[step_earliest_send]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-earliest-notification-document-in-sf-portal-and-send-bulk-email-to-earliest-customers
[step_advisory_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-a-merge-request-for-the-security-advisory-and-include-all-readily-available-information-in-it
[step_reproducer_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-containing-a-system-test-reproducing-the-problem
[step_notify_support]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#notify-support-when-a-reproducer-is-ready
[step_code_analysis]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-detailed-explanation-of-the-code-flow-triggering-the-problem
[step_fix_mr]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-private-merge-request-with-the-fix
[step_review_fix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#ensure-the-merge-request-with-the-fix-is-reviewed-and-has-no-outstanding-discussions
[step_review_docs]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#review-the-documentation-changes-introduced-by-the-merge-request-with-the-fix
[step_backports]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-backports-of-the-merge-request-addressing-the-problem-for-all-affected-and-still-maintained-branches-of-a-given-product
[step_finish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#finish-preparing-the-security-advisory
[step_meta_issue]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#create-or-update-the-private-issue-containing-links-to-fixes-reproducers-for-all-cves-fixed-in-a-given-release-cycle
[step_changes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-reserve-a-block-of-changes-placeholders-once-the-complete-set-of-vulnerabilities-fixed-in-a-given-release-cycle-is-determined
[step_merge_fixes]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-cve-fixes-in-cve-identifier-order
[step_patches]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-a-standalone-patch-for-the-last-stable-release-of-each-affected-and-still-maintained-product-branch
[step_asn_releases]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#prepare-asn-releases-as-outlined-in-the-release-checklist
[step_asn_documents]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-the-text-on-the-t-5-from-the-printing-press-project-and-earliest-asn-documents-in-the-sf-portal
[step_asn_links]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-update-the-bind-s-information-document-in-sf-with-download-links-to-the-new-versions
[step_asn_send]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bulk-email-eligible-customers-to-check-the-sf-portal
[step_preannouncement]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-send-a-pre-announcement-email-to-the-bind-announce-mailing-list-to-alert-users-that-the-upcoming-release-will-include-security-fixes
[step_packager_emails]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notifications-to-os-packagers
[step_clearance]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#grant-qa-marketing-clearance-to-proceed-with-public-release
[step_publish]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#publish-the-releases-as-outlined-in-the-release-checklist
[step_matrix]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bind-9-only-add-the-new-cves-to-the-vulnerability-matrix-in-the-knowledge-base
[step_publish_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#bump-document-version-for-the-security-advisory-and-publish-it-in-the-knowledge-base
[step_notifications]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#send-notification-emails-to-third-parties
[step_mitre]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#advise-mitre-about-the-disclosed-cves
[step_merge_advisory]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-the-security-advisory-merge-request
[step_embargo_end]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#inform-original-reporter-if-external-that-the-security-disclosure-process-is-complete
[step_asn_clear]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#update-the-sf-portal-to-clear-the-asn
[step_customers]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#email-asn-recipients-that-the-embargo-is-lifted
[step_regression]: https://gitlab.isc.org/isc-private/isc-wiki/-/wikis/Security-Incident-Handling-Checklist-Explanations#merge-a-regression-test-reproducing-the-bug-into-all-affected-and-still-maintained-branches
---
Hi,
Continuing our research, regarding CVE 2023-2828 ([issue/4055](https://gitlab.isc.org/isc-projects/bind9/-/issues/4055)) we discovered yet another possible vulnerability in BIND9 resolvers where one malicious CNAME query can cause over 27,000,000 calls to the dns_name_equal function and over 1,000,000,000 CPU instructions.
In the getsection function in message.c file there is a check if a name is already present in the section before appending the new name to the section.
```
...
if (!dns_name_equal(dns_rootname, name) ||
sectionid != DNS_SECTION_ADDITIONAL ||
msg->opt != NULL)
{
DO_ERROR(DNS_R_FORMERR);
}
...
for (count = 0; count < msg->counts[sectionid]; count++) {
...
```
The function checks whether each name is already present in the section by sending all the previous names with the new name to _dns_name_equal_ function - which causes a quadratic function call.
For CNAME queries, the BIND resolver has a resolution limit of 17, so for each malicious CNAME query the resolver executes 17 queries to the authoritative name server.
I tested an answer with 1800 RRsets long CNAME chain, and as a result, the dns_name_equal function is called i-1 times for RRset i for i = n to n-17.
In the experiment, I observed 27,304,801 calls for _dns_name_equal_ function, and my calculation indeed predicts:
`∑ n^2 / 2`. n from 1800 to 1784 = 27,295,948
I used Valgrind and Kcachegrind for checking that. Here is the valgind file: [callgrind.out.cname_shoham1.shoham.fun](/uploads/09aef4edae0384e93d1a891781dcfc20/callgrind.out.cname_shoham1.shoham.fun)
You can see here the number of function calls and the CPU instructions:
![fig_callgrind.out.cname_shoham1.shoham.fun](/uploads/e878d48174b9a8ebf0a8549eb8391e04/fig_callgrind.out.cname_shoham1.shoham.fun.png)
And here is an example of the answer section I respond:
![fig_shoham1.shoham.fun_pcap](/uploads/f13a330dac95e9e63483635c97bf075d/fig_shoham1.shoham.fun_pcap.png)
How to reproduce the vulnerability and additional technical information:
For this experiment, I used my Azure environment:
I used 3 machines: a client, resolver and authoritative name server
All my machines are Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz x64 with 2 vCPUs 8 GiB RAM and Linux (Ubuntu 20.04) OS.
I used BIND 9.16.42 version with no configuration changes. Here is the config.log file: [config.log](/uploads/3e4d410a1d2ea1c4185b0faaf492f229/config.log)
and the conf.named.option file: [named.conf.options](/uploads/12e35dc45738b7b2a29b16c2092902db/named.conf.options)
In the authoritative, I have a CNAME chain with 1800 RRsets (from shoham1.shoham.fun to shoham1800.shoham.fun). Attaching the zonfile here: [shoham.fun.forward](/uploads/c4279fc7a7997435c0fedf8268195edc/shoham.fun.forward)
The client issued a single query using this command: dig shoham1.shoham.fun. @<my_resolver_ip>
To reproduce the attack:
Turn on the resolver with the Valgrind tool using the following command: valgrind --tool=callgrind named -g -c /etc/named.conf.
From the client, query shoham1.shoham.fun using this command: dig shoham1.shoham.fun. @<your_resolver_ip>
Close the Valgrind resolve (Ctrl c will work)
Open the Callgrind file using Qcachegrind: qcachegrind ./callgrind.out.<the resolver PID>
You can also create your authoritative with the zonefile attached or with any long CNAME chain, here is a script for that:
```
with open('zonfile.txt', 'w') as f:
for i in range(1, 1801):
if i % 1800 != 0:
print(f'shoham{i} 86400 IN CNAME shoham{i + 1}',file=f)
else:
print(f'shoham{i} 86400 IN A 1.1.1.1',file=f)
```
Please don't hesitate to ask any question or any additional information.
Thanks,
Shoham Danino, Anat Bremler-Barr, Yehuda Afek and Yuval ShavittJanuary 2024 (9.16.46, 9.16.46-S1, 9.18.22, 9.18.22-S1, 9.19.20) (❗RECALLED❗)Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/4229nextpart failed, set -e fallout?2023-08-02T08:27:07ZMark Andrewsnextpart failed, set -e fallout?Job [#3549936](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3549936) failed for c24edb125bc7eeb6ed4e86e336b27bab5d7092cd:Job [#3549936](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3549936) failed for c24edb125bc7eeb6ed4e86e336b27bab5d7092cd:August 2023 (9.16.43, 9.16.43-S1, 9.18.18, 9.18.18-S1, 9.19.16)https://gitlab.isc.org/isc-projects/bind9/-/issues/4226dig help message https-plain-get vs http-plain-get2023-08-02T08:24:45ZOli Schacherdig help message https-plain-get vs http-plain-get### Summary
`dig -h` says:
```
+[no]https-plain-get (Use GET instead of default POST method while using plain HTTP)
```
but the option is actually called `+[no]http-plain-get`, without the s
### BIND version used
`DiG 9.18.17`...### Summary
`dig -h` says:
```
+[no]https-plain-get (Use GET instead of default POST method while using plain HTTP)
```
but the option is actually called `+[no]http-plain-get`, without the s
### BIND version used
`DiG 9.18.17`
### Steps to reproduce
run `dig -h`
or for the more adventurous:
```
dig @dns.switch.ch isc.org +https-plain-get
Invalid option: +https-plain-get
[....]
```
### What is the current *bug* behavior?
dig -h claims the option name is `+[no]https-plain-get`
### What is the expected *correct* behavior?
`dig -h` should say the option name is `+[no]http-plain-get`
### Relevant configuration files
n/a
### Relevant logs and/or screenshots
n/a
### Possible fixes
change https://gitlab.isc.org/isc-projects/bind9/-/blob/6a6f2e58e9e4a2e5d30ebb6113e429712dc09b34/bin/dig/dig.c#L228August 2023 (9.16.43, 9.16.43-S1, 9.18.18, 9.18.18-S1, 9.19.16)Arаm SаrgsyаnArаm Sаrgsyаnhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4225SERVFAIL response to TKEY query2023-07-31T08:25:21ZTomas SimonaitisSERVFAIL response to TKEY query### Summary
(Summarize the bug encountered concisely.)
### BIND version used
9.16.42 (deb11u1)
9.18.16 (deb12u1~bpo11+1)
### Steps to reproduce
The example PCAP is attached.
### What is the current *bug* behavior?
The TKEY query f...### Summary
(Summarize the bug encountered concisely.)
### BIND version used
9.16.42 (deb11u1)
9.18.16 (deb12u1~bpo11+1)
### Steps to reproduce
The example PCAP is attached.
### What is the current *bug* behavior?
The TKEY query for domain which is not configured in the authoritative only DNS server
results in SERVFAIL response and log entry is:
9.18:
client @0x7fd9410d0368 XX.X.XXX.XX#59386 (2752-ms-7.986X-5052cb3.a4e0250c-2acc-11ee-4794-005056987d00): query failed (permission denied) for 2752-ms-7.986X-5052cb3.a4e0250c-2acc-11ee-4794-005056987d00/IN/TKEY at query.c:12326
9.16:
client @0x7fX13c24Xba0 XX.X.XXX.XX#51660 (2752-ms-7.1359-4fdedX.a4e0250c-2acX-11ee-4794-005056987d0X): query failed (permission denied) for 2752-ms-7.1359-4fdedX.a4e0250c-2acX-11ee-4794-005056987d0X/IN/TKEY at query.c:11891
### What is the expected *correct* behavior?
A REFUSED,FORMERR,NOTIMP response, but not a SERVFAIL.
### Relevant configuration files
```
acl "trusted" {
127.0.0.1/32;
192.0.2.65/32;
};
acl "probes" {
192.0.2.75/32;
212.224.66.61/32;
192.0.2.90/32;
};
acl "transfer-trusted" {
127.0.0.1/32;
192.0.2.65/32;
192.0.2.8/32;
};
logging {
channel "stats_log" {
null ;
};
channel "security_log" {
syslog "local1";
severity notice;
};
channel "query-errors-log" {
file "/var/log/dns/query-errors.log" versions 2 size 102400;
severity debug 1;
};
channel "rrl_log" {
syslog "local2";
severity notice;
};
channel "audit_log" {
syslog "local1";
severity info;
};
category "queries" {
"stats_log";
};
category "security" {
"security_log";
};
category "query-errors" {
"query-errors-log";
};
category "rate-limit" {
"rrl_log";
};
category "default" {
"audit_log";
};
category "general" {
"audit_log";
};
category "config" {
"audit_log";
};
category "resolver" {
"audit_log";
};
category "xfer-in" {
"audit_log";
};
category "xfer-out" {
"audit_log";
};
category "notify" {
"audit_log";
};
category "client" {
"audit_log";
};
category "network" {
"audit_log";
};
category "update" {
"audit_log";
};
category "lame-servers" {
"audit_log";
};
};
options {
blackhole {
10.0.0.0/8;
172.16.0.0/12;
192.168.0.0/16;
};
directory "/var/cache/bind";
hostname none;
interface-interval 0;
listen-on {
192.0.2.65/32;
192.0.2.60/32;
192.0.2.61/32;
};
listen-on-v6 {
"any";
};
serial-query-rate 20;
statistics-file "/var/cache/bind/named.stats";
version none;
auth-nxdomain no;
dnssec-validation no;
rate-limit {
errors-per-second 40;
ipv4-prefix-length 32;
ipv6-prefix-length 64;
max-table-size 30000;
responses-per-second 40;
slip 2;
window 60;
};
recursion no;
allow-query {
"trusted";
};
allow-transfer {
"transfer-trusted";
};
also-notify {
192.0.2.8;
};
notify explicit;
notify-source 192.0.2.60;
request-ixfr no;
transfer-source 192.0.2.65;
zone-statistics no;
};
zone "X" {
....
```
[tkey-servfail.pcap](/uploads/bbefe7dff2b23cdd875fc5e366b2ba91/tkey-servfail.pcap)August 2023 (9.16.43, 9.16.43-S1, 9.18.18, 9.18.18-S1, 9.19.16)Mark AndrewsMark Andrewshttps://gitlab.isc.org/isc-projects/bind9/-/issues/4215Add ISC_R_TIMEDOUT to the reasons to call dns_zonemgr_unreachableadd in xfrin2023-08-17T12:15:43ZMark AndrewsAdd ISC_R_TIMEDOUT to the reasons to call dns_zonemgr_unreachableadd in xfrinUse timeout as another reason to mark a primary as temporarily unreachable in xfrin.c. This should help the fall over to other primaries happen faster when one primary is not responding.Use timeout as another reason to mark a primary as temporarily unreachable in xfrin.c. This should help the fall over to other primaries happen faster when one primary is not responding.August 2023 (9.16.43, 9.16.43-S1, 9.18.18, 9.18.18-S1, 9.19.16)Mark AndrewsMark Andrewshttps://gitlab.isc.org/isc-projects/bind9/-/issues/4203run.gdb not found2023-07-28T11:53:56ZMark Andrewsrun.gdb not foundStack backtrace no produced because `run.gdb` was not found.
```
[New LWP 100128]
[New LWP 100256]
[New LWP 100257]
[New LWP 100323]
Core was generated by `/builds/isc-projects/bind9/bin/named/.libs/named -c /builds/isc-projects/bind9/b...Stack backtrace no produced because `run.gdb` was not found.
```
[New LWP 100128]
[New LWP 100256]
[New LWP 100257]
[New LWP 100323]
Core was generated by `/builds/isc-projects/bind9/bin/named/.libs/named -c /builds/isc-projects/bind9/b'.
Program terminated with signal SIGABRT, Aborted.
Sent by kill() from pid 2375 and user 1001.
#0 0x00000000002384f7 in control_recvmessage (result=ISC_R_SHUTTINGDOWN, arg=0x802659960, handle=<optimized out>) at controlconf.c:418
warning: Source file is more recent than executable.
418 if (conn->shuttingdown) {
[Current thread is 1 (LWP 100128)]
warning: run.gdb: No such file or directory
```August 2023 (9.16.43, 9.16.43-S1, 9.18.18, 9.18.18-S1, 9.19.16)https://gitlab.isc.org/isc-projects/bind9/-/issues/4201Excessive memory use when removing ZSK2023-10-09T13:44:15ZPeter DaviesExcessive memory use when removing ZSK
### Summary
BIND uses an excessive amount of memory when removing a ZSK with nsupdate from a very large zone file
### BIND version used
BIND 9.18.15
```
> named -V
BIND 9.18.16 (Extended Support Version) <id:8193c9b>
running on Linu...
### Summary
BIND uses an excessive amount of memory when removing a ZSK with nsupdate from a very large zone file
### BIND version used
BIND 9.18.15
```
> named -V
BIND 9.18.16 (Extended Support Version) <id:8193c9b>
running on Linux x86_64 4.18.0-477.15.1.el8_8.x86_64 #1 SMP Fri Jun 2 08:27:19 EDT 2023
built by make with '--prefix=/opt/bind-versions/bind-9.18.16' 'PKG_CONFIG_PATH=/opt/libuv-versions/libuv-1.44.2/lib/pkgconfig/'
compiled by GCC 8.5.0 20210514 (Red Hat 8.5.0-18)
compiled with OpenSSL version: OpenSSL 1.1.1k FIPS 25 Mar 2021
linked to OpenSSL version: OpenSSL 1.1.1k FIPS 25 Mar 2021
compiled with libuv version: 1.44.2
linked to libuv version: 1.44.2
compiled with libnghttp2 version: 1.33.0
linked to libnghttp2 version: 1.33.0
compiled with libxml2 version: 2.9.7
linked to libxml2 version: 20907
compiled with zlib version: 1.2.11
linked to zlib version: 1.2.11
threads support is enabled
DNSSEC algorithms: RSASHA1 NSEC3RSASHA1 RSASHA256 RSASHA512 ECDSAP256SHA256 ECDSAP384SHA384 ED25519 ED448
DS algorithms: SHA-1 SHA-256 SHA-384
HMAC algorithms: HMAC-MD5 HMAC-SHA1 HMAC-SHA224 HMAC-SHA256 HMAC-SHA384 HMAC-SHA512
TKEY mode 2 support (Diffie-Hellman): yes
TKEY mode 3 support (GSS-API): yes
default paths:
named configuration: /opt/bind-versions/bind-9.18.16/etc/named.conf
rndc configuration: /opt/bind-versions/bind-9.18.16/etc/rndc.conf
DNSSEC root key: /opt/bind-versions/bind-9.18.16/etc/bind.keys
nsupdate session key: /opt/bind-versions/bind-9.18.16/var/run/named/session.key
named PID file: /opt/bind-versions/bind-9.18.16/var/run/named/named.pid
named lock file: /opt/bind-versions/bind-9.18.16/var/run/named/named.lock
```
### What is the current *bug* behavior?
Using nsupate, manually remove a ZSK from a very large zone, BIND uses a very large amount of memory and get killed by the oom manager
### Relevant configuration files
See: [RT #22296](https://support.isc.org/Ticket/Display.html?id=22296)Not plannedMatthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/4200Repeated crashes from BIND 9.16.40-S1 and 9.16.42.S1 consistently from either...2023-08-02T10:18:37ZCathy AlmondRepeated crashes from BIND 9.16.40-S1 and 9.16.42.S1 consistently from either db.c:1263 or db.c:138### Summary
This organisation run 20 or so servers, they are all on BIND 9.16.40-S1 and 9.16.42.S1. The servers that run 9.16.42-S1 are recently upgraded. The servers running 9.16.40-S1 have been doing so for almost as long as this ve...### Summary
This organisation run 20 or so servers, they are all on BIND 9.16.40-S1 and 9.16.42.S1. The servers that run 9.16.42-S1 are recently upgraded. The servers running 9.16.40-S1 have been doing so for almost as long as this version has been available. The crashes started around June 29/30, so something has changed that is triggering them.
Here is a sample of the crashes as reported in their logs:
```
mpcold1/named.log:2023-07-02T11:43:09+00:00 mpcold1 named[28600]: 02-Jul-2023 11:43:09.241 general: critical: db.c:1263: REQUIRE((__b
uiltin_expect(!!((db) != ((void *)0)), 1) && __builtin_expect(!!(((const isc__magic_t *)(db))->magic == ((('D') << 24 | ('N') << 16 |
('S') << 8 | ('D')))), 1))) failed
mpcold1/named.log:2023-07-02T11:43:09+00:00 mpcold1 named[28600]: 02-Jul-2023 11:43:09.241 general: critical: exiting (due to asserti
on failure)
mpcold2/named.log:2023-07-05T14:50:13+00:00 mpcold2 named[2841]: 05-Jul-2023 14:50:13.161 general: critical: db.c:138: REQUIRE((__bui
ltin_expect(!!((source) != ((void *)0)), 1) && __builtin_expect(!!(((const isc__magic_t *)(source))->magic == ((('D') << 24 | ('N') <
< 16 | ('S') << 8 | ('D')))), 1))) failed
mpcold2/named.log:2023-07-05T14:50:13+00:00 mpcold2 named[2841]: 05-Jul-2023 14:50:13.161 general: critical: exiting (due to assertio
n failure)
mpcold3/named.log:2023-07-02T09:18:11+00:00 mpcold3 named[60441]: 02-Jul-2023 09:18:11.174 general: critical: db.c:1263: REQUIRE((__b
uiltin_expect(!!((db) != ((void *)0)), 1) && __builtin_expect(!!(((const isc__magic_t *)(db))->magic == ((('D') << 24 | ('N') << 16 |
('S') << 8 | ('D')))), 1))) failed
mpcold3/named.log:2023-07-02T09:18:11+00:00 mpcold3 named[60441]: 02-Jul-2023 09:18:11.174 general: critical: exiting (due to asserti
on failure)
mpcold3/named.log:2023-07-04T19:41:07+00:00 mpcold3 named[28125]: 04-Jul-2023 19:41:07.874 general: critical: db.c:1263: REQUIRE((__b
uiltin_expect(!!((db) != ((void *)0)), 1) && __builtin_expect(!!(((const isc__magic_t *)(db))->magic == ((('D') << 24 | ('N') << 16 |
('S') << 8 | ('D')))), 1))) failed
mpcold3/named.log:2023-07-04T19:41:07+00:00 mpcold3 named[28125]: 04-Jul-2023 19:41:07.874 general: critical: exiting (due to asserti
on failure)
mpcold4/named.log:2023-07-02T09:54:16+00:00 mpcold4 named[81687]: 02-Jul-2023 09:54:16.146 general: critical: db.c:1263: REQUIRE((__b
uiltin_expect(!!((db) != ((void *)0)), 1) && __builtin_expect(!!(((const isc__magic_t *)(db))->magic == ((('D') << 24 | ('N') << 16 |
('S') << 8 | ('D')))), 1))) failed
mpcold4/named.log:2023-07-02T09:54:16+00:00 mpcold4 named[81687]: 02-Jul-2023 09:54:16.146 general: critical: exiting (due to asserti
on failure)
mpcold4/named.log:2023-07-03T06:39:10+00:00 mpcold4 named[14845]: 03-Jul-2023 06:39:10.816 general: critical: db.c:138: REQUIRE((__bu
iltin_expect(!!((source) != ((void *)0)), 1) && __builtin_expect(!!(((const isc__magic_t *)(source))->magic == ((('D') << 24 | ('N')
<< 16 | ('S') << 8 | ('D')))), 1))) failed
mpcold4/named.log:2023-07-03T06:39:10+00:00 mpcold4 named[14845]: 03-Jul-2023 06:39:10.816 general: critical: exiting (due to asserti
on failure)
mpcold4/named.log:2023-07-04T11:12:28+00:00 mpcold4 named[71958]: 04-Jul-2023 11:12:28.531 general: critical: db.c:1263: REQUIRE((__b
uiltin_expect(!!((db) != ((void *)0)), 1) && __builtin_expect(!!(((const isc__magic_t *)(db))->magic == ((('D') << 24 | ('N') << 16 |
('S') << 8 | ('D')))), 1))) failed
```
### BIND version used
9.16.40-S1 and 9.16.42-S1 (I don't have named -V, but can get this). Self-built and RPMs created that they install on their suite of resolvers. Stripped binaries - we're working on this as we have a lot of core dumps ...
### Steps to reproduce
No reproduction, just wait...
### What is the current *bug* behavior?
BIND crashes.
### What is the expected *correct* behavior?
BIND doesn't crash.
### Relevant configuration files
Significantly, what we have here is resolvers that are acting as proxies in front of Intranet auth servers. The queries they receive had originally been fielded by load-balancers, that flip the RD bit and then send them on to one of two views (one is auth only, the other accepts recursion). We're interested in the recursive side of things, and the fact the query header RD bit has been flipped is interesting for the background picture, but shouldn't affect the scenario that we have a resolver repeatedly crashing. The important components of the options are:
```
options {
directory "/var/named";
//listen-on port 53 { _obscured_; _obscured_; _obscured_; _obscured_; };
listen-on port 53 { _obscured_; _obscured_; _obscured_; _obscured_; _obscured_; _obscured_; };
query-source address _obscured_;
query-source-v6 address _obscured_;
dnssec-enable yes;
dnssec-validation no;
//edns-udp-size 1024;
version "";
stale-cache-enable no;
allow-update { none ; };
allow-transfer { none; };
transfer-source _obscured_;
transfers-per-ns 1;
transfers-in 10;
recursive-clients 5000;
tcp-idle-timeout 30;
tcp-clients 750;
notify no;
send-cookie no;
require-server-cookie no;
ecs-forward {any;};
ecs-zones { obscured; };
fetches-per-zone 35 drop;
fetch-quota-params 100 .1 .3 .7;
fetches-per-server 50 drop;
rate-limit {
slip 2; // Every other response truncated
window 15; // Seconds to bucket
responses-per-second 100; // # of good responses per prefix-length/sec
referrals-per-second 80; // referral responses
nodata-per-second 25; // nodata responses
nxdomains-per-second 20; // nxdomain responses
errors-per-second 25; // error responses
all-per-second 500; // When we drop all
log-only no; // Debugging mode
qps-scale 5000; // x / 1000 * per-second = new drop limit
//exempt-clients { 127.0.0.1; _obscured_; _obscured_ !};
ipv4-prefix-length 24; // Define the IPv4 block size
ipv6-prefix-length 56; // Define the IPv6 block size
max-table-size 1200000; // 40 bytes * this number = max memory
min-table-size 500; // pre-allocate to speed startup
};
};
```
### Relevant logs and/or screenshots
See crashes logged above.
Significantly in the one I first looked at, just ahead of the crash was this:
> Jul 4 11:22:09 mppacd2 named[2884]: client @0x7f7eb03864e0
> _obscured_#44786 (_obscured_): view Rec-
> instance: recursive-clients soft limit exceeded (4901/4900/5000),
> aborting oldest query
> Jul 4 11:22:09 mppacd2 named[2884]: client @0x7f7e7c720f20
> _obscured-different_#62182 (_obscured-different_): view Rec-
> instance: no more recursive clients (5000/4900/5000): quota reached
However, exploring further - this is not consistently associated with the crashes - the quota is being reached with no crashes, and the crashes occur without quota reached being logged.
Note however that both fetch-limits and RRL are being triggered.
More data is available from [Support ticket #22339](https://support.isc.org/Ticket/Display.html?id=22339)
### Possible fixes
I checked the crash code locations - here are my musings
db.c:1263
```
isc_result_t
dns_db_getservestalerefresh(dns_db_t *db, uint32_t *interval) {
REQUIRE(DNS_DB_VALID(db));
REQUIRE((db->attributes & DNS_DBATTR_CACHE) != 0);
if (db->methods->getservestalerefresh != NULL) {
return ((db->methods->getservestalerefresh)(db, interval));
}
return (ISC_R_NOTIMPLEMENTED);
}
```
db.c:138
```
dns_db_attach(dns_db_t *source, dns_db_t **targetp) {
/*
* Attach *targetp to source.
*/
REQUIRE(DNS_DB_VALID(source));
REQUIRE(targetp != NULL && *targetp == NULL);
(source->methods->attach)(source, targetp);
ENSURE(*targetp == source);
}
```
The crash is for the exact same reason in each - this test fails:
` REQUIRE(DNS_DB_VALID(source));`
But why do we not have a valid DB pointer? `dns_db_attach()` is used all over the place - too many for me to do anything useful with without stack traces that show me the circumstances around the call. But there are only a handful of locations where we call `dns_db_getservestalerefresh()` so I took a meander...
The likeliest candidate is in query.c from query_lookup(). Note that we're getting the pointer to the db from the client. (Aside: also that we're getting the value of the length of time in which stale answers are directly returned from cache before attempting to refresh them (in case a previous attempt in doing so has failed), without first checking if we're going to use stale content at all, or even if we have stale cache enabled. So this is unnecessary, except that we then decide what to do with the value. But this lookup from cache DB should work anyway.)
I'm bothered that the pointer to the cachedb is dud. Might this be a race between looking in cache to construct query response (on a timeout) versus the same client being dropped simultaneously because of hitting recursive clients quota?
```
(void)dns_db_getservestalerefresh(qctx->client->view->cachedb,
&stale_refresh);
```
That's the call - the pointer to cachedb that is invalid has come from the client structure. Why is that 'bad'? Where did we get the pointer to the client struct from, and why is only this operation crashing if this is a race between 'do something with this client' and 'this client is being dropped because of some quota or something'?
Looking at the other locations, I doubt that the crash has come from server.c - that's just a call with a direct pointer to the cache DB in order to get the value to output in response to an rndc query to get the values.
Meanwhile, I see in cache.c that the call to dns_db_getservestalerefresh() has a wrapper:
dns_cache_getservestalerefresh()
But I think this is also not useful. I see it used in server.c whilst checking what its value is to determine whether or not it's OK for two views to share the same cache. And see what dns_cache_getservestalerefresh() does before it calls dns_db_getservestalerefresh() - calls REQUIRE(VALID_CACHE(cache)). I do not think we passed this way in the direction of the crash...August 2023 (9.16.43, 9.16.43-S1, 9.18.18, 9.18.18-S1, 9.19.16)https://gitlab.isc.org/isc-projects/bind9/-/issues/4184Minor documentation error in query log format2023-08-04T10:19:17ZVicky Riskvicky@isc.orgMinor documentation error in query log formatDocs bug.
Greg verified that this appears to be a correct report
----
> I'm trying to understand the Bind 9 query log format, and came across a potential mistake in the admin manual:
>
> https://bind9.readthedocs.io/en/latest/refere...Docs bug.
Greg verified that this appears to be a correct report
----
> I'm trying to understand the Bind 9 query log format, and came across a potential mistake in the admin manual:
>
> https://bind9.readthedocs.io/en/latest/reference.html#the-category-phrase
>
> under "queries", the manual states (emphasis mine);
>
> > The query log entry first reports a **client object identifier in @0x<hexadecimal-number> format**. Next, it reports the client’s IP address and port number, and the query name, class, and type.
>
> However, the example query log entry listed after this paragraph is missing this object identifier:
>
> > client 127.0.0.1#62536 (www.example.com): query: www.example.com IN AAAA +SE client ::1#62537 (www.example.net): query: www.example.net IN AAAA -SE
>
> Meanwhile, in my own Bind 9 query log, I *can* see this client object identifier. Therefore, I believe you are mistakenly using an old query log entry as an example. Can you please confirm if this is the case?July 2023 (9.18.17, 9.18.17-S1, 9.19.15)Michał KępieńMichał Kępieńhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4171catz crashed in dns_catz_dbupdate_callback2023-10-20T07:47:47ZOndřej Surýcatz crashed in dns_catz_dbupdate_callbackA [recent job](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3488427) failed with:
```
27-Jun-2023 06:41:51.371 ht.c:364: REQUIRE(((ht) != ((void *)0) && ((const isc__magic_t *)(ht))->magic == ((('H') << 24 | ('T') << 16 | ('a') << 8...A [recent job](https://gitlab.isc.org/isc-projects/bind9/-/jobs/3488427) failed with:
```
27-Jun-2023 06:41:51.371 ht.c:364: REQUIRE(((ht) != ((void *)0) && ((const isc__magic_t *)(ht))->magic == ((('H') << 24 | ('T') << 16 | ('a') << 8 | ('b'))))) failed
27-Jun-2023 06:41:51.371 /builds/isc-projects/bind9/lib/isc/.libs/libisc-9.19.15-dev.so(isc_backtrace_log+0x39) [0x7fb70ba2e0d8]
27-Jun-2023 06:41:51.371 /builds/isc-projects/bind9/bin/named/.libs/lt-named() [0x422644]
27-Jun-2023 06:41:51.371 /builds/isc-projects/bind9/lib/isc/.libs/libisc-9.19.15-dev.so(isc_assertion_failed+0xa) [0x7fb70ba2dc9d]
27-Jun-2023 06:41:51.371 /builds/isc-projects/bind9/lib/isc/.libs/libisc-9.19.15-dev.so(isc_ht_find+0x9d) [0x7fb70ba3633f]
27-Jun-2023 06:41:51.371 /builds/isc-projects/bind9/lib/dns/.libs/libdns-9.19.15-dev.so(dns_catz_dbupdate_callback+0x84) [0x7fb70b43fe1c]
27-Jun-2023 06:41:51.371 /builds/isc-projects/bind9/lib/dns/.libs/libdns-9.19.15-dev.so(dns_db_endload+0x46) [0x7fb70b44de68]
27-Jun-2023 06:41:51.371 /builds/isc-projects/bind9/lib/dns/.libs/libdns-9.19.15-dev.so(+0x1aa38b) [0x7fb70b5aa38b]
27-Jun-2023 06:41:51.371 /builds/isc-projects/bind9/lib/dns/.libs/libdns-9.19.15-dev.so(+0x8a3c6) [0x7fb70b48a3c6]
27-Jun-2023 06:41:51.371 /builds/isc-projects/bind9/lib/isc/.libs/libisc-9.19.15-dev.so(+0x56230) [0x7fb70ba56230]
27-Jun-2023 06:41:51.371 /lib64/libuv.so.1(uv__work_done+0xad) [0x7fb709425afd]
27-Jun-2023 06:41:51.371 /lib64/libuv.so.1(+0x132f1) [0x7fb70942a2f1]
27-Jun-2023 06:41:51.371 /lib64/libuv.so.1(uv__io_poll+0x4c5) [0x7fb70943bd15]
27-Jun-2023 06:41:51.371 /lib64/libuv.so.1(uv_run+0x114) [0x7fb70942aa74]
27-Jun-2023 06:41:51.371 /builds/isc-projects/bind9/lib/isc/.libs/libisc-9.19.15-dev.so(+0x4072c) [0x7fb70ba4072c]
27-Jun-2023 06:41:51.371 /builds/isc-projects/bind9/lib/isc/.libs/libisc-9.19.15-dev.so(+0x4fdbb) [0x7fb70ba4fdbb]
27-Jun-2023 06:41:51.371 /builds/isc-projects/bind9/lib/isc/.libs/libisc-9.19.15-dev.so(+0x4fde4) [0x7fb70ba4fde4]
27-Jun-2023 06:41:51.371 /lib64/libpthread.so.0(+0x81da) [0x7fb7082f81da]
27-Jun-2023 06:41:51.371 /lib64/libc.so.6(clone+0x43) [0x7fb7075cbe73]
27-Jun-2023 06:41:51.371 exiting (due to assertion failure)
```July 2023 (9.18.17, 9.18.17-S1, 9.19.15)Arаm SаrgsyаnArаm Sаrgsyаnhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4169Adding a META tag to the doc tree for Google Analytics/Bing2023-06-29T16:50:37ZDan MahoneyAdding a META tag to the doc tree for Google Analytics/BingThe only way we can track 404's on ReadTheDocs is with Google Search console (and the related BING tools, which validate using google, ie if you access to a thing in the google console, bing will also give you access to their tools with ...The only way we can track 404's on ReadTheDocs is with Google Search console (and the related BING tools, which validate using google, ie if you access to a thing in the google console, bing will also give you access to their tools with no extra work.
We cannot use the "upload an HTML file" to RTD, nor a TXT record, and the Google Analytics tag method isn't compatible. The META tag is pretty much the only option that works here.
Google Search Console wants us to add:
`<meta name="google-site-verification" content="0-DMrB_qDynsvXBKJhpsS5m8l5oVea8Qe2ojkudjtCY" />` to our default page (presently bind 9.18.16)
My reading of Sphinx implies that we need to add:
```
.. meta::
:google-site-verification: 0-DMrB_qDynsvXBKJhpsS5m8l5oVea8Qe2ojkudjtCY
```
to `doc/arm/index.rst` to cause this meta tag to be added. It's fine if this gets committed now and has to wait for the next release before it shows up in tree, i.e. for the release cut of 9.18.17July 2023 (9.18.17, 9.18.17-S1, 9.19.15)Michał KępieńMichał Kępień