'fetchlimit' system test failed

Job #3197549 failed for b737d280:

2 sub issues:

• I:fetchlimit:checking lame server clients are dropped below the hard limit
• I:fetchlimit:checking drop statistics

I:fetchlimit:checking lame server clients are dropped below the hard limit
I:fetchlimit:clients: 350
I:fetchlimit:clients: 91
I:fetchlimit:clients: 300
I:fetchlimit:clients: 196
I:fetchlimit:clients: 292
I:fetchlimit:5 successful valid queries (expected 5)
I:fetchlimit:0 SERVFAIL responses (expected 0)
I:fetchlimit:clients count exceeded 400 on 1 trials (expected 0)
I:fetchlimit:failed
I:fetchlimit:failed
I:fetchlimit:checking drop statistics
I:fetchlimit:failed
I:fetchlimit:exit status: 2
I:fetchlimit:stopping servers
R:fetchlimit:FAIL
E:fetchlimit:2023-02-28T02:15:48+0000
FAIL fetchlimit (exit status: 1)

I:fetchlimit:checking lame server clients are dropped below the hard limit is a false positive as stat 100 400 fails if clients in less than 100 as well as greater than 400.

I:fetchlimit:clients: 91

stat() {
    clients=`$RNDCCMD status | grep "recursive clients" |
            sed 's;.*: \([^/][^/]*\)/.*;\1;'`
    echo_i "clients: $clients"
    [ "$clients" = "" ] && return 1
    [ "$clients" -ge $1 ] || return 1
    [ "$clients" -le $2 ] || return 1
    return 0
}

I:fetchlimit:checking drop statistics failure appears to be a race in the sub test as the tests waits for the existence of ns3/named.stats then performs the grep which may occur before the sought after line is written. The sought after line was in ns3/named.stats.

[ant-7149:tests/system/fetchlimit] marka% more ns3/named.stats | grep dropped
                1360 queries dropped
                1073 queries dropped due to recursive client limit
[ant-7149:tests/system/fetchlimit] marka%