resolver.c:7625: REQUIRE(event->ev_type == (((4) << 16) + 1)) assertion failure in BIND 9.16.49

Summary

I encountered an assertion failure (see logs below) on one of my production recursive nameservers, which I just upgraded earlier today from 9.16.48 to 9.16.49.

BIND version affected

[dmrz@resolver8-1 ~]$ /opt/isc/isc-bind/root/sbin/named -V
BIND 9.16.49 (Extended Support Version) <id:d5b3d64>
running on Linux x86_64 5.14.0-362.18.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jan 3 15:54:45 EST 2024
built by make with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/opt/isc/isc-bind/root/usr' '--exec-prefix=/opt/isc/isc-bind/root/usr' '--bindir=/opt/isc/isc-bind/root/usr/bin' '--sbindir=/opt/isc/isc-bind/root/usr/sbin' '--sysconfdir=/etc/opt/isc/scls/isc-bind' '--datadir=/opt/isc/isc-bind/root/usr/share' '--includedir=/opt/isc/isc-bind/root/usr/include' '--libdir=/opt/isc/isc-bind/root/usr/lib64' '--libexecdir=/opt/isc/isc-bind/root/usr/libexec' '--localstatedir=/var/opt/isc/scls/isc-bind' '--sharedstatedir=/var/opt/isc/scls/isc-bind/lib' '--mandir=/opt/isc/isc-bind/root/usr/share/man' '--infodir=/opt/isc/isc-bind/root/usr/share/info' '--enable-warn-error' '--disable-static' '--enable-dnstap' '--with-pic' '--with-gssapi' '--with-json-c' '--with-libtool' '--with-libxml2' '--without-lmdb' '--with-python' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CC=gcc' 'CFLAGS=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer' 'LDFLAGS=-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -L/opt/isc/isc-bind/root/usr/lib64' 'CPPFLAGS=' 'LT_SYS_LIBRARY_PATH=/usr/lib64' 'PKG_CONFIG_PATH=:/opt/isc/isc-bind/root/usr/lib64/pkgconfig:/opt/isc/isc-bind/root/usr/share/pkgconfig' 'SPHINX_BUILD=/builddir/build/BUILD/bind-9.16.49/sphinx/bin/sphinx-build'
compiled by GCC 11.4.1 20230605 (Red Hat 11.4.1-2)
compiled with OpenSSL version: OpenSSL 3.0.7 1 Nov 2022
linked to OpenSSL version: OpenSSL 3.0.7 1 Nov 2022
compiled with libuv version: 1.44.2
linked to libuv version: 1.44.2
compiled with libxml2 version: 2.9.13
linked to libxml2 version: 20913
compiled with json-c version: 0.14
linked to json-c version: 0.14
compiled with zlib version: 1.2.11
linked to zlib version: 1.2.11
compiled with protobuf-c version: 1.4.1
linked to protobuf-c version: 1.4.1
threads support is enabled
DNSSEC algorithms: RSASHA256 RSASHA512 ECDSAP256SHA256 ECDSAP384SHA384 ED25519 ED448
DS algorithms: SHA-1 SHA-256 SHA-384
HMAC algorithms: HMAC-MD5 HMAC-SHA1 HMAC-SHA224 HMAC-SHA256 HMAC-SHA384 HMAC-SHA512
TKEY mode 2 support (Diffie-Hellman): yes
TKEY mode 3 support (GSS-API): yes

default paths:
  named configuration:  /etc/opt/isc/scls/isc-bind/named.conf
  rndc configuration:   /etc/opt/isc/scls/isc-bind/rndc.conf
  DNSSEC root key:      /etc/opt/isc/scls/isc-bind/bind.keys
  nsupdate session key: /var/opt/isc/scls/isc-bind/run/named/session.key
  named PID file:       /var/opt/isc/scls/isc-bind/run/named/named.pid
  named lock file:      /var/opt/isc/scls/isc-bind/run/named/named.lock

Steps to reproduce

unknown, this occurred in production

Relevant configuration files

<snip>REDACTED</snip>

Relevant logs

named.log:

...
2024-03-28T20:24:06.233Z client: info: client @0x7fcc604ab500 128.174.180.7#50292 (ad.global.uillinois.edu): recursion loop detected
2024-03-28T20:24:06.235Z client: info: client @0x7fcc44259fc0 10.195.122.21#56767 (sgreport.wechat.com): recursion loop detected
2024-03-28T20:24:07.271Z general: critical: resolver.c:7625: REQUIRE(event->ev_type == (((4) << 16) + 1)) failed, back trace
2024-03-28T20:24:07.271Z general: critical: #0 0x5649b0f76664 in ??
2024-03-28T20:24:07.271Z general: critical: #1 0x7fcc6bccce2e in ??
2024-03-28T20:24:07.271Z general: critical: #2 0x7fcc6bb1ceea in ??
2024-03-28T20:24:07.271Z general: critical: #3 0x7fcc6bd0874f in ??
2024-03-28T20:24:07.271Z general: critical: #4 0x7fcc6bce8416 in ??
2024-03-28T20:24:07.271Z general: critical: #5 0x7fcc6bce8a09 in ??
2024-03-28T20:24:07.271Z general: critical: #6 0x7fcc6bce8be9 in ??
2024-03-28T20:24:07.271Z general: critical: #7 0x7fcc6b84ec9d in ??
2024-03-28T20:24:07.271Z general: critical: #8 0x7fcc6b86aaae in ??
2024-03-28T20:24:07.271Z general: critical: #9 0x7fcc6b854650 in ??
2024-03-28T20:24:07.271Z general: critical: #10 0x7fcc6bce901d in ??
2024-03-28T20:24:07.271Z general: critical: #11 0x7fcc6bd02a9e in ??
2024-03-28T20:24:07.271Z general: critical: #12 0x7fcc6b09f802 in ??
2024-03-28T20:24:07.271Z general: critical: #13 0x7fcc6b03f450 in ??
2024-03-28T20:24:07.271Z general: critical: exiting (due to assertion failure)