Bind 9.16.6 Assertion failure message.c:4733: REQUIRE(msg->state == (-1)) failed

Summary

Bind exits with a assertion failure.

Sep 3 07:20:14 r9 named[63118]: message.c:4733: REQUIRE(msg->state == (-1)) failed, back trace  
Sep 3 07:20:14 r9 named[63118]: #0 0x2ced70 in ??  
Sep 3 07:20:14 r9 named[63118]: #1 0x4b077a in ??  
Sep 3 07:20:14 r9 named[63118]: #2 0x38eea3 in ??  
Sep 3 07:20:14 r9 named[63118]: #3 0x404086 in ??  
Sep 3 07:20:14 r9 named[63118]: #4 0x4d5a3c in ??  
Sep 3 07:20:14 r9 named[63118]: #5 0x800a75736 in ??  
Sep 3 07:20:14 r9 named[63118]: exiting (due to assertion failure). 

BIND version used

$ named -V  
BIND 9.16.4 (Stable Release) <id:0849b42>  
running on FreeBSD amd64 12.1-RELEASE-p6 FreeBSD 12.1-RELEASE-p6 GENERIC  
built by make with '--disable-linux-caps' '--localstatedir=/var' '--sysconfdir=/usr/local/etc/namedb' '--with-dlopen=yes' '--with-libxml2' '--with-openssl=/usr' '--with-readline=-L/usr/local/lib -ledit' '--with-dlz-filesystem=yes' '--enable-dnstap' '--disable-fixed-rrset' '--disable-geoip' '--without-maxminddb' '--without-gssapi' '--with-libidn2=/usr/local' '--with-json-c' '--disable-largefile' '--with-lmdb=/usr/local' '--disable-native-pkcs11' '--without-python' '--disable-querytrace' 'STD_CDEFINES=-DDIG_SIGCHASE=1' '--enable-tcp-fastopen' '--with-tuning=large' '--disable-symtable' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/share/info/' '--build=amd64-portbld-freebsd12.1' 'build_alias=amd64-portbld-freebsd12.1' 'CC=cc' 'CFLAGS=-O2 -pipe -DLIBICONV_PLUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing ' 'LDFLAGS= -L/usr/local/lib -ljson-c -fstack-protector-strong ' 'LIBS=-L/usr/local/lib' 'CPPFLAGS=-DLIBICONV_PLUG -isystem /usr/local/include' 'CPP=cpp' 'PKG_CONFIG=pkgconf'  
compiled by CLANG 4.2.1 Compatible FreeBSD Clang 8.0.1 (tags/RELEASE_801/final 366581)  
compiled with OpenSSL version: OpenSSL 1.1.1d-freebsd 10 Sep 2019  
linked to OpenSSL version: OpenSSL 1.1.1d-freebsd 10 Sep 2019  
compiled with libxml2 version: 2.9.10  
linked to libxml2 version: 20910  
compiled with json-c version: 0.14  
linked to json-c version: 0.14  
compiled with zlib version: 1.2.11  
linked to zlib version: 1.2.11  
compiled with protobuf-c version: 1.3.2  
linked to protobuf-c version: 1.3.2  
threads support is enabled  
  
default paths:  
named configuration: /usr/local/etc/namedb/named.conf  
rndc configuration: /usr/local/etc/namedb/rndc.conf  
DNSSEC root key: /usr/local/etc/namedb/bind.keys  
nsupdate session key: /var/run/named/session.key  
named PID file: /var/run/named/pid  
named lock file: /var/run/named/named.lock  

What is the current bug behavior?

Bind exits

Relevant configuration files

See RT #1776

Relevant logs and/or screenshots

Sep 3 07:20:14 r9 named[63118]: message.c:4733: REQUIRE(msg->state == (-1)) failed, back trace  
Sep 3 07:20:14 r9 named[63118]: #0 0x2ced70 in ??  
Sep 3 07:20:14 r9 named[63118]: #1 0x4b077a in ??  
Sep 3 07:20:14 r9 named[63118]: #2 0x38eea3 in ??  
Sep 3 07:20:14 r9 named[63118]: #3 0x404086 in ??  
Sep 3 07:20:14 r9 named[63118]: #4 0x4d5a3c in ??  
Sep 3 07:20:14 r9 named[63118]: #5 0x800a75736 in ??  
Sep 3 07:20:14 r9 named[63118]: exiting (due to assertion failure). 

Comment:

• The failing assertion is REQUIRE(msg->state == DNS_SECTION_ANY);.
  I do not think we have seen similar crashes in the past few months.

• Assuming this is a resolver without any secondary zones configured,
  there is just one call to dns_message_setclass() that could have
  triggered the crash: lib/dns/resolver.c:7589. It would be
  interesting to see what state fctx->rmessage is in at the time of
  the crash.

• Some ThreadSanitizer fixes were merged since 9.16.4, but I would
  personally be surprised if any of them actually addressed this
  problem.

• Maybe this is something related to prefetch? Nothing should be
  touching fctx->rmessage->state until a response to a query comes
  in... unless multiple queries are involved?