rbtdb.c asssertion failure -- REQUIRE(isc_refcount_current(&rbtdb->node_locks[i].references) == 0) failed

Summary

A customer has reported hitting a REQUIRE assertion failure in rbtdb.c

BIND version used

BIND 9.16.8 running on FreeBSD 12.1

Their "named -V" info:

BIND 9.16.8 (Stable Release) <id:539f9f0>
running on FreeBSD amd64 12.1-RELEASE-p10 FreeBSD 12.1-RELEASE-p10 GENERIC
built by make with '--disable-linux-caps' '--localstatedir=/var' '--sysconfdir=/usr/local/etc/namedb' '--with-dlopen=yes' '--with-libxml2' '--with-openssl=/usr' '--with-readline=-L/usr/local/lib -ledit' '--with-dlz-filesystem=yes' '--enable-dnstap' '--disable-fixed-rrset' '--disable-geoip' '--without-maxminddb' '--without-gssapi' '--with-libidn2=/usr/local' '--with-json-c' '--disable-largefile' '--with-lmdb=/usr/local' '--disable-native-pkcs11' '--without-python' '--disable-querytrace' '--enable-tcp-fastopen' '--enable-developer' '--enable-symtable' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/share/info/' '--build=amd64-portbld-freebsd12.1' 'build_alias=amd64-portbld-freebsd12.1' 'CC=cc' 'CFLAGS=-pipe -DLIBICONV_PLUG -g -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing ' 'LDFLAGS= -L/usr/local/lib -ljson-c -fstack-protector-strong ' 'LIBS=-L/usr/local/lib' 'CPPFLAGS=-DLIBICONV_PLUG -isystem /usr/local/include' 'CPP=cpp' 'PKG_CONFIG=pkgconf'
compiled by CLANG 4.2.1 Compatible FreeBSD Clang 8.0.1 (tags/RELEASE_801/final 366581)
compiled with OpenSSL version: OpenSSL 1.1.1d-freebsd 10 Sep 2019
linked to OpenSSL version: OpenSSL 1.1.1d-freebsd 10 Sep 2019
compiled with libuv version: 1.40.0
linked to libuv version: 1.40.0
compiled with libxml2 version: 2.9.10
linked to libxml2 version: 20910
compiled with json-c version: 0.15
linked to json-c version: 0.15
compiled with zlib version: 1.2.11
linked to zlib version: 1.2.11
compiled with protobuf-c version: 1.3.2
linked to protobuf-c version: 1.3.2
threads support is enabled

default paths:
named configuration: /usr/local/etc/namedb/named.conf
rndc configuration: /usr/local/etc/namedb/rndc.conf
DNSSEC root key: /usr/local/etc/namedb/bind.keys
nsupdate session key: /var/run/named/session.key
named PID file: /var/run/named/pid
named lock file: /var/run/named/named.lock

Steps to reproduce

Currently unknown. Until we make a determination whether this is deliberately triggerable, I am starting this ticket as confidential.

What is the current bug behavior?

These are the lines that were logged when the customer's server exited:

Nov 29 06:59:00 res named[28212]: rbtdb.c:1146: REQUIRE(isc_refcount_current(&rbtdb->node_locks[i].references) == 0) failed, back trace
Nov 29 06:59:00 res named[28212]: #0 0x300be9 in assertion_failed()+0x59
Nov 29 06:59:00 res named[28212]: #1 0x67dd18 in isc_assertion_failed()+0x38
Nov 29 06:59:00 res named[28212]: #2 0x49207c in free_rbtdb()+0xcdc
Nov 29 06:59:00 res named[28212]: #3 0x4b16de in free_rbtdb_callback()+0x2e
Nov 29 06:59:00 res named[28212]: #4 0x6ca483 in dispatch()+0xc63
Nov 29 06:59:00 res named[28212]: #5 0x6c5ed1 in run()+0x41
Nov 29 06:59:00 res named[28212]: #6 0x800c76736 in _fini()+0x80058d1ca
Nov 29 06:59:00 res named[28212]: exiting (due to assertion failure)

What else have we got to go on?

Support ticket #17349 has two responses on it which contain links, one to a core file generated by the crash, the second to a package containing their named binary. They are linked from their ftp server, so I omit the URLs here, authorized devs please visit the support ticket for that info or request that Support copy it to a more convenient location if you prefer.