Again: BIND | rbtdb.c:2162: INSIST with bind with 9.11.20 (see #1718)

Summary

Same problem as #1718 (closed), but this time after several hours on two of our three slave servers at the same time:

Sudden crash of the named process.

BIND version used

BIND 9.11.20 (Extended Support Version) <id:f3d1d66>
running on Linux x86_64 3.16.0-10-amd64 #1 SMP Debian 3.16.81-1 (2020-01-17)
built by make with '--prefix=/usr' '--mandir=/usr/share/man' '--libdir=/usr/lib/x86_64-linux-gnu' '--infodir=/usr/share/info' '--sysconfdir=/etc/bind' '--with-python=python3' '--localstatedir=/' '--enable-threads' '--enable-largefile' '--with-libtool' '--enable-shared' '--enable-static' '--with-openssl=/usr' '--with-gssapi=/usr' '--with-gnu-ld' '--enable-ipv6' '--enable-filter-aaaa' '--with-make-clean'
compiled by GCC 4.9.2
compiled with OpenSSL version: OpenSSL 1.0.1t  3 May 2016
linked to OpenSSL version: OpenSSL 1.0.1t  3 May 2016
compiled with libxml2 version: 2.9.1
linked to libxml2 version: 20901
compiled with libjson-c version: 0.11.99
linked to libjson-c version: 0.11.99
compiled with zlib version: 1.2.8
linked to zlib version: 1.2.8
threads support is enabled

default paths:
  named configuration:  /etc/bind/named.conf
  rndc configuration:   /etc/bind/rndc.conf
  DNSSEC root key:      /etc/bind/bind.keys
  nsupdate session key: //run/named/session.key
  named PID file:       //run/named/named.pid
  named lock file:      //run/named/named.lock

Steps to reproduce

# Created bind as usual:
VERSION=9.11.20
wget -O bind-$(VERSION).tar.gz https://downloads.isc.org/isc/bind9/$(VERSION)/bind-$(VERSION).tar.gz
wget -O bind-$(VERSION).tar.gz.sha512.asc https://downloads.isc.org/isc/bind9/$(VERSION) /bind-$(VERSION).tar.gz.sha512.asc
gpg --verify bind-$(VERSION).tar.gz.sha512.asc bind-$(VERSION).tar.gz
tar -zxf bind-$(VERSION).tar.gz
bind-$(VERSION)
./configure --prefix=/usr \
                --mandir=\$${prefix}/share/man \
                --libdir=\$${prefix}/lib/$(DEB_HOST_MULTIARCH) \
                --infodir=\$${prefix}/share/info \
                --sysconfdir=/etc/bind \
                --with-python=python3 \
                --localstatedir=/ \
                --enable-threads \
                --enable-largefile \
                --with-libtool \
                --enable-shared \
                --enable-static \
                --with-openssl=/usr \
                --with-gssapi=/usr \
                --with-gnu-ld \
                --enable-ipv6 \
                --enable-filter-aaaa
make && make install

What is the current bug behavior?

After several hours, bind crashes with the following message in general.log:

23-Jun-2020 14:25:01.251 general: rbtdb.c:2162: INSIST(((unsigned int)(isc_atomic_xadd(&(&(node)->references)->refs, 0))) == 0 && node->data == ((void *)0)) failed, back trace
23-Jun-2020 14:25:01.251 general: #0 0x43fe6d in ??
23-Jun-2020 14:25:01.251 general: #1 0x7f382e8ffcfa in ??
23-Jun-2020 14:25:01.251 general: #2 0x7f382fbc73ed in ??
23-Jun-2020 14:25:01.251 general: #3 0x7f382fbc776c in ??
23-Jun-2020 14:25:01.251 general: #4 0x7f382e929a17 in ??
23-Jun-2020 14:25:01.251 general: #5 0x7f382daaa064 in ??
23-Jun-2020 14:25:01.251 general: #6 0x7f382d47862d in ??
23-Jun-2020 14:25:01.251 general: exiting (due to assertion failure)

The same (at the same time!) happens on the other dns-slave server:

23-Jun-2020 14:25:01.594 general: rbtdb.c:2162: INSIST(((unsigned int)(isc_atomic_xadd(&(&(node)->references)->refs, 0))) == 0 && node->data == ((void *)0)) failed, back trace
23-Jun-2020 14:25:01.594 general: #0 0x43fe6d in ??
23-Jun-2020 14:25:01.594 general: #1 0x7f19743eacfa in ??
23-Jun-2020 14:25:01.594 general: #2 0x7f19756b23ed in ??
23-Jun-2020 14:25:01.594 general: #3 0x7f19756b276c in ??
23-Jun-2020 14:25:01.594 general: #4 0x7f1974414a17 in ??
23-Jun-2020 14:25:01.594 general: #5 0x7f1973595064 in ??
23-Jun-2020 14:25:01.594 general: #6 0x7f1972f6362d in ??
23-Jun-2020 14:25:01.594 general: exiting (due to assertion failure)

What is the expected correct behavior?

No crash.

Relevant configuration files

named.conf:

include "/etc/bind/named.conf.local"; // only ACLs, logging and statistic channels
include "/etc/bind/named.conf.options"; // look down

include "/etc/bind/bind.keys";

include "/etc/bind/named.conf.namedboot";
include "/etc/bind/tsig.key";

named.options:

options {
        directory "/var/cache/bind";
        pid-file "/var/run/named/named.pid";

        auth-nxdomain no;    # conform to RFC1035
        listen-on-v6 { ::1; ********;  };
        listen-on { 127.0.0.1; *********;  };

        allow-query {  any;  }; 
        allow-transfer {  ******;  };
        recursion no;
        version "0";
        dnssec-enable yes;
        dnssec-validation yes;

        tcp-clients 1500;

        rate-limit {
                responses-per-second 50;
        };
};

controls {
        inet 127.0.0.1 allow { 127.0.0.1; ::1; };
};

Relevant logs and/or screenshots

general.log:

23-Jun-2020 14:25:01.251 general: rbtdb.c:2162: INSIST(((unsigned int)(isc_atomic_xadd(&(&(node)->references)->refs, 0))) == 0 && node->data == ((void *)0)) failed, back trace
23-Jun-2020 14:25:01.251 general: #0 0x43fe6d in ??
23-Jun-2020 14:25:01.251 general: #1 0x7f382e8ffcfa in ??
23-Jun-2020 14:25:01.251 general: #2 0x7f382fbc73ed in ??
23-Jun-2020 14:25:01.251 general: #3 0x7f382fbc776c in ??
23-Jun-2020 14:25:01.251 general: #4 0x7f382e929a17 in ??
23-Jun-2020 14:25:01.251 general: #5 0x7f382daaa064 in ??
23-Jun-2020 14:25:01.251 general: #6 0x7f382d47862d in ??
23-Jun-2020 14:25:01.251 general: exiting (due to assertion failure)

Possible fixes