"fetchlimit" system test fails intermittently

The fetchlimit system test strikes again...

https://gitlab.isc.org/isc-private/bind9/-/jobs/2755343

The bit that caused this particular failure is:

I:fetchlimit:checking lame server clients are dropped below the hard limit
I:fetchlimit:clients: 345
I:fetchlimit:clients: 216
I:fetchlimit:clients: 316
I:fetchlimit:clients: 88
I:fetchlimit:clients: 320
I:fetchlimit:5 successful valid queries (expected 5)
I:fetchlimit:0 SERVFAIL responses (expected 0)
I:fetchlimit:clients count exceeded 400 on 1 trials (expected 0)
I:fetchlimit:failed

The messages logged here are confusing - the client counts never exceeded 400; instead, it dropped below 100 on 1 "trial", which is something that the test checks since !6308 (merged) has been merged; unfortunately, the logging output has not been adjusted accordingly.

This failure happened in a TSAN-enabled job, where named performance in "heavy" tests is affected quite significantly, so I do not think this is something worth taking a closer look at, but I still felt like opening an issue to track this problem.

Click to expand/collapse full test output

S:fetchlimit:2022-09-09T14:22:50+0000
T:fetchlimit:1:A
A:fetchlimit:System test fetchlimit
I:fetchlimit:PORTRANGE:8100 - 8199
I:fetchlimit:starting servers
I:fetchlimit:checking recursing clients are dropped at the per-server limit
I:fetchlimit:clients: 20
I:fetchlimit:clients: 40
I:fetchlimit:clients: 60
I:fetchlimit:clients: 80
I:fetchlimit:clients: 100
I:fetchlimit:clients: 120
I:fetchlimit:clients: 140
I:fetchlimit:clients: 160
I:fetchlimit:clients: 180
I:fetchlimit:clients: 200
I:fetchlimit:clients: 197
I:fetchlimit:clients: 180
I:fetchlimit:clients: 156
I:fetchlimit:clients: 132
I:fetchlimit:clients: 132
I:fetchlimit:clients: 121
I:fetchlimit:clients: 121
I:fetchlimit:clients: 121
I:fetchlimit:clients: 121
I:fetchlimit:clients: 122
I:fetchlimit:dumping ADB data
I:fetchlimit:atr 1.00 quota 114
I:fetchlimit:checking servfail statistics
I:fetchlimit:checking lame server recovery
I:fetchlimit:clients: 91
I:fetchlimit:clients: 12
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:dumping ADB data
I:fetchlimit:atr 0.44 quota 95
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:clients: 0
I:fetchlimit:dumping ADB data
I:fetchlimit:atr 0.04 quota 101
I:fetchlimit:checking lame server clients are dropped at the per-domain limit
I:fetchlimit:clients: 40
I:fetchlimit:clients: 40
I:fetchlimit:clients: 40
I:fetchlimit:clients: 40
I:fetchlimit:clients: 40
I:fetchlimit:5 successful valid queries, 0 SERVFAIL
I:fetchlimit:checking drop statistics
I:fetchlimit:checking lame server clients are dropped below the hard limit
I:fetchlimit:clients: 345
I:fetchlimit:clients: 216
I:fetchlimit:clients: 316
I:fetchlimit:clients: 88
I:fetchlimit:clients: 320
I:fetchlimit:5 successful valid queries (expected 5)
I:fetchlimit:0 SERVFAIL responses (expected 0)
I:fetchlimit:clients count exceeded 400 on 1 trials (expected 0)
I:fetchlimit:failed
I:fetchlimit:failed
I:fetchlimit:checking drop statistics
I:fetchlimit:failed
I:fetchlimit:exit status: 2
I:fetchlimit:stopping servers
R:fetchlimit:FAIL
E:fetchlimit:2022-09-09T14:24:04+0000