CPU get abnormally high and thread get stucked
sometimes my DNS server CPU rise to 800% , which is normally 100% , here is the rndc status and pstack output
rndc status version: BIND 9.16.4 (Stable Release) id:0849b42 (i don't know) running on bjht17876: Linux x86_64 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 boot time: Wed, 01 Mar 2023 14:02:22 GMT last configured: Mon, 06 Mar 2023 03:45:20 GMT configuration file: /etc/opt/isc/isc-bind/named.conf CPUs found: 64 worker threads: 64 UDP listeners per interface: 64 number of zones: 109 (97 automatic) debug level: 0 xfers running: 0 xfers deferred: 0 soa queries in progress: 0 query logging is OFF recursive clients: 6/102300/102400 tcp clients: 0/150 TCP high-water: 14 server is up and running
seems there are some locks , but I can not find them, named.run log and rndc recursing seems all ok :
cat named.recursing ; ; Recursing Queries ; ; client 10.37.42.69#15676: id 29223 'msgrecv.f5.ynhtbank.com/AAAA/IN' for msgrecv.ynhtbank.com requesttime 1678073230 ; client 10.35.22.18#29524: id 51632 'ns2.dnslog.cn/AAAA/IN' requesttime 1678073234 ; client 10.35.64.3#51697: id 4066 'olearning-sgp-replay.myoppo.com.cdn.gl102.com/AAAA/IN' for olearning-sgp-replay.myoppo.com requesttime 1678073237 ; client 10.35.22.18#51079: id 8557 '52m7hp.dnslog.cn/AAAA/IN' requesttime 1678073242 ; client 10.34.172.48#39630: id 24172 '143.219.241.192.in-addr.arpa/PTR/IN' requesttime 1678073244 ; client 10.35.96.6#48324: id 11819 'apiserver-test-sg01-baymax-op-lc-e4eda5fda71edd38.elb.ap-southeast-1.amazonaws.com/A/IN' for apiserver-test-sg01.baymax.oppo.local requesttime 1678073244 ; ; Active fetch domains [view: _default] ; msgrecv.f5.ynhtbank.com.: 1 active (0 spilled, 1 allowed) elb.ap-southeast-1.amazonaws.com.: 1 active (0 spilled, 1 allowed) cdn.gl102.com.: 1 active (0 spilled, 1 allowed) 219.241.192.in-addr.arpa.: 1 active (0 spilled, 1 allowed) dnslog.cn.: 4 active (0 spilled, 7 allowed) ; ; Active fetch domains [view: _bind] ; ; Dump complete
06-Mar-2023 12:04:08.491 network unreachable resolving 'web.congrong-inc.com/AAAA/IN': 2001:502:7094::30#53 06-Mar-2023 12:04:08.491 network unreachable resolving 'web.congrong-inc.com/AAAA/IN': 2001:502:8cc::30#53 06-Mar-2023 12:04:08.720 network unreachable resolving 'xhn-wap.hinews.cn.iname.damddos.com/A/IN': 240e:980:0:1600::118:254#53 06-Mar-2023 12:04:08.720 network unreachable resolving 'xhn-wap.hinews.cn.iname.damddos.com/A/IN': 240e:980:0:1600::116:253#53 06-Mar-2023 12:04:08.822 timed out resolving 'ht-44-29-188.internal.zeku.com/A/IN': 10.123.35.100#53 06-Mar-2023 12:04:09.936 network unreachable resolving 'downloads.sourceforge.net/A/IN': 2600:180a:4001::1#53 06-Mar-2023 12:04:10.075 timed out resolving 'ht-44-29-188.internal.zeku.com/A/IN': 10.123.35.101#53 06-Mar-2023 12:04:10.879 network unreachable resolving 'bj02-train-prod002-node-10-38-32-169/A/IN': 2001:dc3::35#53 06-Mar-2023 12:04:11.043 network unreachable resolving 'h5.yuyouma.com/A/IN': 2402:4e00:1020:1264:0:9136:29bc:87f9#53 06-Mar-2023 12:04:11.043 network unreachable resolving 'h5.yuyouma.com/A/IN': 2402:4e00:1430:1102:0:9136:2b30:e554#53 06-Mar-2023 12:04:11.644 network unreachable resolving 'version.gitlab.com/A/IN': 2a06:98c1:50::ac40:239d#53 06-Mar-2023 12:04:11.792 DNS format error from 8.143.241.249#53 resolving m-gsdk.snssdk.com.queniurc.com/AAAA for 10.37.42.81#54716: Name w.cdngslb.com (SOA) not subdomain of zone queniurc.com -- invalid response 06-Mar-2023 12:04:11.792 FORMERR resolving 'm-gsdk.snssdk.com.queniurc.com/AAAA/IN': 8.143.241.249#53 06-Mar-2023 12:04:11.809 network unreachable resolving '188.80.222.60.adsl-pool.sx.cn/AAAA/IN': 2803:f800:50::6ca2:c14b#53 06-Mar-2023 12:04:11.809 network unreachable resolving '188.80.222.60.adsl-pool.sx.cn/AAAA/IN': 2a06:98c1:50::ac40:20f7#53 06-Mar-2023 12:04:12.441 network unreachable resolving 'jyys.easepictures.art/A/IN': 2001:67c:13cc::1:49#53 06-Mar-2023 12:04:12.574 timed out resolving 'ht-44-24-119.internal.zeku.com/AAAA/IN': 10.123.35.100#53 06-Mar-2023 12:04:13.019 network unreachable resolving 'packagecloud.io/A/IN': 2600:9000:5301:3800::1#53 06-Mar-2023 12:04:13.232 network unreachable resolving 'cdl-lb-1356093980.us-east-1.elb.amazonaws.com/AAAA/IN': 2600:9000:5303:a600::1#53 06-Mar-2023 12:04:13.340 network unreachable resolving 'gt-push.gltxy.xyz/AAAA/IN': 2600:1480:b800::40#53