Assertion failure in fetches-per-zone

Summary

The fix in !9115 (merged) introduce a regression when all servers configure with fetches-per-zone to non-zero value would soon crash with assertion failure - if QMIN or forwarder table is used (basically anything that uses fcount_incr(..., true);

BIND version affected

• 9.21.0-unreleased acfa5b28f918667a03602f3d779d0e478324a0e7
• 9.20.1-unreleased 9a0c59c89a9d5c1a3c3e119c7578ec1109d1989b

Steps to reproduce

1. Use following patch to change server default config:

diff --git a/bin/named/config.c b/bin/named/config.c
index 3ad700a887..b1d6170bf6 100644
--- a/bin/named/config.c
+++ b/bin/named/config.c
@@ -159,8 +159,8 @@ options {\n\
 #endif /* ifdef HAVE_DNSTAP */
                 "\
     fetch-quota-params 100 0.1 0.3 0.7;\n\
-    fetches-per-server 0;\n\
-    fetches-per-zone 0;\n\
+    fetches-per-server 400;\n\
+    fetches-per-zone 40;\n\
     lame-ttl 0;\n"
 #ifdef HAVE_LMDB
                 "    lmdb-mapsize 32M;\n"

3. Run system tests
4. Observe system tests crashing one after another.

What is the current bug behavior?

Server crashes with:

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1  0x00007f2c860fde8f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  0x00007f2c860aefb2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007f2c86099472 in __GI_abort () at ./stdlib/abort.c:79
#4  0x0000560a967c326d in assertion_failed (file=<optimized out>, line=1531, type=isc_assertiontype_insist, cond=0x7f2c86bee41a "counter->count > 0") at main.c:234
#5  0x00007f2c871324cd in isc_assertion_failed (file=file@entry=0x7f2c86bee40f "resolver.c", line=line@entry=1531, type=type@entry=isc_assertiontype_insist, cond=cond@entry=0x7f2c86bee41a "counter->count > 0")
    at assertions.c:48
#6  0x00007f2c86b37e1f in fcount_decr (fctx=0x7f2c7cd11c00) at resolver.c:1531
#7  0x00007f2c86b426e5 in fctx_destroy (fctx=0x7f2c7cd11c00) at resolver.c:4337
#8  0x00007f2c86b42871 in fetchctx_unref (ptr=0x1c8bc) at resolver.c:6974
#9  0x00007f2c86b428b0 in fetchctx_detach (ptrp=0x1c8bc) at resolver.c:6974
#10 0x00007f2c86b4659c in dns_resolver_destroyfetch (fetchp=<optimized out>) at resolver.c:10626
#11 0x00007f2c86b48f8a in resume_qmin (arg=<optimized out>) at resolver.c:4168
#12 0x00007f2c871327f6 in isc__async_cb (handle=<optimized out>) at async.c:111
#13 0x00007f2c86d1c09d in ?? () from /lib/x86_64-linux-gnu/libuv.so.1
#14 0x00007f2c86d2fe6c in ?? () from /lib/x86_64-linux-gnu/libuv.so.1
#15 0x00007f2c86d1c9e4 in uv_run () from /lib/x86_64-linux-gnu/libuv.so.1
#16 0x00007f2c87145ba4 in loop_thread (arg=arg@entry=0x7f2c83508900) at loop.c:288
#17 0x00007f2c87156f34 in thread_body (wrap=wrap@entry=0x7f2c834c07c0) at thread.c:85
#18 0x00007f2c87156f5d in thread_run (wrap=0x7f2c834c07c0) at thread.c:100
#19 0x00007f2c860fc134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#20 0x00007f2c8617c7dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

What is the expected correct behavior?

Server not crashing.

Relevant configuration files

options { fetches-per-zone 40; };

Relevant logs