system test `serve-stale` doesn't get refreshed answer when expected

Job #3214712 failed with:

I:serve-stale:check stale data.example TXT was refreshed (stale-answer-client-timeout 0 stale-refresh-time 4) (183)
I:serve-stale:failed

The test was expecting a fresh answer, but it got a stale one instead:

; <<>> DiG 9.18.13-S1 <<>> +time +tries -p 14092 @10.53.0.3 data.example TXT
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62349
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 8c969d3b5a853b08010000006405d40e4781607af7f2ae49 (good)
; EDE: 3 (Stale Answer): (stale data prioritized over lookup)
;; QUESTION SECTION:
;data.example.			IN	TXT

;; ANSWER SECTION:
data.example.		3	IN	TXT	"A text record with a 2 second ttl"

;; Query time: 23 msec
;; SERVER: 10.53.0.3#14092(10.53.0.3) (UDP)
;; WHEN: Mon Mar 06 11:52:46 UTC 2023
;; MSG SIZE  rcvd: 155

Saw this in v9.18, but the exact same test is present in other branches, so I assume they're affected as well.