Resolver under load retransmits even though the authoritative replies promptly
Summary
With enough traffic going to and from BIND 9.18 running as a resolver, some packets from the authoritative (sent well before the 800ms retransmit deadline in packet capture seem) to get stuck somewhere in BIND.
BIND version
$ ./named -V
BIND 9.18.10 (Stable Release) <id:7011eaf>
Steps to reproduce
- Start an authoritative server serving this zone on
1.0.0.100
(Knot was used in testing):
. 86400 IN SOA j.root-servers.net. nstld.verisign-grs.com. 2019072500 1800 900 604800 86400
*. 3600 IN A 1.1.1.1
. 86400 IN NS j.root-servers.net.
- Start BIND 9.18 with this config:
options {
listen-on {1.0.0.1; };
listen-on-v6 { };
directory "/tmp/maze-one_server-5y3fes_c/resolver";
recursion yes;
query-source address 1.0.0.1;
dnssec-validation no;
empty-zones-enable yes;
};
zone "." {
type hint;
file "/tmp/maze-one_server-5y3fes_c/resolver/root.hints";
}
with hints file:
. 3600000 NS J.ROOT-SERVERS.NET.
J.ROOT-SERVERS.NET. 3600000 A 1.0.0.100
-
Optional: set up constant delay on authoritative server's answers with
netem
(done in the example below (delay is set to 10ms), but not necessary). -
Set up traffic capture.
-
Run
dnsperf
as a client to create traffic:
dnsperf -d q -Q 5000 -q 10000 -t 10
with the file q
that contains the lines 0. A
to 9999. A
.
- Inspect the resulting PCAP with Wireshark and filter for long delays on responses to client. This can be achieved with the filter
dns.time > 0.08
.
Details (current behavior)
This was run as an comparison between 9.16.36
, 9.18.10
and 35e2842
(marked as main
in the text and images bellow).
There is only one authoritative server in this case and its RTT is set to 10 ms. This was run on a 2-core/4-thread VM with Knot as the authoritative.
The resolution of culprit queries look like this when examined packet by packet:
t = 0 ms, client asks the resolver
t = 0+ε ms, resolver asks the authoritative
t = 10 ms, authoritative sends response
t = 800 ms, resolver asks the authoritative again
t = 810 ms, authoritative answers again
t = 810+ε ms, resolver finally answers
The delayed responses also appear in chunks pointing to them maybe being stuck in some kind of queue (looking and you libuv
) this can be shown by graphing the response times to client with a sliding window average:
Note, that there are no peaks with 9.16.
I also attach the pcap of the run with 9.18 for reference.
Expected behavior
There should be no such retransmits.