Assert on validator.c:3205: INSIST(__v > 0) during resolver benchmark

Summary

First sketch. I will supply more details next week, hopefully.

Crash in DNSSEC validator. Exact cause unknown.

BIND version used

• Affects v9.19: v9.19.13 Started with 6ba2579c. Worked on commit da0f154b.

# /var/opt/bind9/bin/named/.libs/named -V  
BIND 9.19.13 (Development Release) <id:66a3c6b>
running on Linux x86_64 6.2.12-1-ec2 #1 SMP PREEMPT_DYNAMIC Mon, 24 Apr 2023 20:44:00 +0000
built by make with  '--without-python' '--disable-linux-caps' '--prefix=/usr' 'CC=' 'CFLAGS=' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS='
compiled by GCC 11.3.0
compiled with OpenSSL version: OpenSSL 3.0.2 15 Mar 2022
linked to OpenSSL version: OpenSSL 3.0.2 15 Mar 2022
compiled with libuv version: 1.43.0
linked to libuv version: 1.43.0
compiled with liburcu version: 0.13.1
compiled with libnghttp2 version: 1.43.0
linked to libnghttp2 version: 1.43.0
compiled with json-c version: 0.15
linked to json-c version: 0.15
compiled with zlib version: 1.2.11
linked to zlib version: 1.2.11
threads support is enabled
DNSSEC algorithms: RSASHA1 NSEC3RSASHA1 RSASHA256 RSASHA512 ECDSAP256SHA256 ECDSAP384SHA384 ED25519 ED448
DS algorithms: SHA-1 SHA-256 SHA-384
HMAC algorithms: HMAC-MD5 HMAC-SHA1 HMAC-SHA224 HMAC-SHA256 HMAC-SHA384 HMAC-SHA512
TKEY mode 2 support (Diffie-Hellman): no
TKEY mode 3 support (GSS-API): no

default paths:
  named configuration:  /usr/etc/named.conf
  rndc configuration:   /usr/etc/rndc.conf
  nsupdate session key: /usr/var/run/named/session.key
  named PID file:       /usr/var/run/named/named.pid
  named lock file:      /usr/var/run/named/named.lock

Did not reproduce on v9.18 or v9.16 😌

Steps to reproduce

• Run https://gitlab.isc.org/isc-projects/bind9-shotgun-ci/-/pipelines/new for 5 minutes with load factor 10x. Dataset telco-eu-2023-03.

What is the current bug behavior?

Crashes in the last minute.

What is the expected correct behavior?

Well, no crash.

Relevant logs and/or screenshots

Asserts on

validator.c:3205: INSIST(__v > 0)

Backtrace - best I was able to get so far:

(No debugging symbols found in /var/opt/bind9/bin/named/.libs/named)
...
Core was generated by `/var/opt/bind9/bin/named/.libs/named -c /etc/bind9/named.conf -g -n 16 -S 10000'.
...
(gdb) bt
#0  __GI_abort () at ./stdlib/abort.c:107
#1  0x0000560012ed77a4 in assertion_failed ()
#2  0x00007fd80e8e40c2 in isc_assertion_failed () from /var/opt/bind9/lib/isc/.libs/libisc-9.19.13.so
#3  0x00007fd80e7befc4 in dns_validator_unref () from /var/opt/bind9/lib/dns/.libs/libdns-9.19.13.so
#4  0x00007fd80e7bf0a7 in dns_validator_detach () from /var/opt/bind9/lib/dns/.libs/libdns-9.19.13.so
#5  0x00007fd80e7b73fb in validator_done_cb () from /var/opt/bind9/lib/dns/.libs/libdns-9.19.13.so
#6  0x00007fd80e8e4616 in isc.async_cb () from /var/opt/bind9/lib/isc/.libs/libisc-9.19.13.so
#7  0x00007fd80e0881ed in ?? () from /lib/x86_64-linux-gnu/libuv.so.1
#8  0x00007fd80e0a411e in ?? () from /lib/x86_64-linux-gnu/libuv.so.1
#9  0x00007fd80e08dc88 in uv_run () from /lib/x86_64-linux-gnu/libuv.so.1
#10 0x00007fd80e900a2f in loop_thread () from /var/opt/bind9/lib/isc/.libs/libisc-9.19.13.so
#11 0x00007fd80e9175d1 in thread_body () from /var/opt/bind9/lib/isc/.libs/libisc-9.19.13.so
#12 0x00007fd80e917619 in thread_run () from /var/opt/bind9/lib/isc/.libs/libisc-9.19.13.so
#13 0x00007fd80de90b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#14 0x00007fd80df22a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81