D2 can enounters write errors sending to DNS
During tests for #3295 (closed) which used a single bind9 DNS server I discoverd that under enough load for a long enough period of time D2 begins to encounter errors writing to the DNS server:
2024-03-26 15:00:22.965 ERROR [kea-dhcp-ddns.asiodns/4711.140284993357696] ASIODNS_SEND_DATA error 101 sending data using UDP to 175.16.1.1(53)
Error code 101 is Network Unreachable.
When this occurs, excuting "nsupdate" from the command line of D2's host produces similar result:
tmark@ubuserver:kea $ nsupdate -v $HOME/n.txt
; Communication with 175.16.1.1#53 failed: operation canceled
could not reach any name server
I do not, at this time, know the root cause of these write failures. Excuting an nsupdate from another VM is successful.
In addition to the write failures themselves, I believe there is an error in how IOFetch handles these errors. It appears that the fetch is allowed to proceed to executing a read after the send has failed, rather that calling IOFetch::stop() as is done for read TIME_OUTs. There seems little point executing a read that will only timeout. This has the affect of causing D2 to do 3 send(fail)/read(timeout) cycles.
Bottom line is there are two questions:
- What causes the Network Unreachable errors?
- Should IOFetch() fail on send failures (perhaps by callng stop()) rather than proceed to read after failed send?
The testing referred to above was done with:
-
Centos 7 VM running perfdhcp: perfdhcp -4 -r 20 -R 10000000 -p 600 -l enp0s10 -W 5000000
-
Centos 7 VM running BIND9 9.11.2:
version: BIND 9.11.2 <id:0a2b929>
running on cthird: Linux x86_64 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017
boot time: Tue, 26 Mar 2024 13:00:27 GMT
last configured: Tue, 26 Mar 2024 13:00:27 GMT
configuration file: /opt/bind9/bind-9.11.2/local/servers/42620/named.conf
CPUs found: 4
worker threads: 4
UDP listeners per interface: 3
number of zones: 6 (0 automatic)
debug level: 100
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is ON
recursive clients: 0/900/1000
tcp clients: 0/150
server is up and running
```lsb_release -a
3. Ubuntu VM 22.02 running kea-dhcp4 and kea-dhcp-ddns, configs attached:
[kea-d2.conf](/uploads/1ec60a23a57c33f03ee4621d6a15c048/kea-d2.conf)
[kea-ddns4.conf](/uploads/7300d3f06194bd7b2310daaed278d722/kea-ddns4.conf)