BIND 9.11.19 dead lock
Summary
Seems that BIND has a deadlock. and I don't find any related race condition issues, maybe it is a new issue?
BIND version used
BIND 9.11.19-RedHat-9.11.10-20200601113814.alios7 (Extended Support Version) <id:905ec64>
running on Linux x86_64 3.10.0-327.ali2012.alios7.x86_64 #1 SMP Mon Oct 9 14:09:14 CST 2017
built by make with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-libtool' '--localstatedir=/var' '--enable-threads' '--enable-epoll' '--with-tuning=large' '--enable-ipv6' '--with-pic' '--disable-static' '--disable-openssl-version-check' '--with-python=/home/tops/bin/python2.7' '--with-python-install-dir=/home/tops' '--with-docbook-xsl=/usr/share/sgml/docbook/xsl-stylesheets' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS= -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro ' 'CPPFLAGS= -DDIG_SIGCHASE' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'
compiled by GCC 4.8.5 20150623 (Red Hat 4.8.5-4)
compiled with OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
linked to OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
compiled with libxml2 version: 2.9.1
linked to libxml2 version: 20901
compiled with zlib version: 1.2.7
linked to zlib version: 1.2.7
threads support is enabled
default paths:
named configuration: /etc/named.conf
rndc configuration: /etc/rndc.conf
DNSSEC root key: /etc/bind.keys
nsupdate session key: /var/run/named/session.key
named PID file: /var/run/named/named.pid
named lock file: /var/run/named/named.lockSteps to reproduce
Not clear.
What is the current bug behavior?
BIND is hung.
What is the expected correct behavior?
(What you should see instead.)
Relevant configuration files
logging {
channel "default_debug" {
file "data/named.run";
severity dynamic;
};
};
options {
bindkeys-file "/etc/named.iscdlv.key";
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
listen-on port 53 {
127.0.0.1/32;
};
listen-on-v6 port 53 {
::1/128;
};
memstatistics-file "/var/named/data/named_mem_stats.txt";
statistics-file "/var/named/data/named_stats.txt";
dnssec-enable yes;
dnssec-lookaside auto;
dnssec-validation yes;
recursion yes;
allow-query {
"localhost";
};
};
zone "." IN {
type hint;
file "named.ca";
};
zone "localhost.localdomain" IN {
type master;
file "named.localhost";
allow-update {
"none";
};
};
zone "localhost" IN {
type master;
file "named.localhost";
allow-update {
"none";
};
};
zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa" IN {
type master;
file "named.loopback";
allow-update {
"none";
};
};
zone "1.0.0.127.in-addr.arpa" IN {
type master;
file "named.loopback";
allow-update {
"none";
};
};
zone "0.in-addr.arpa" IN {
type master;
file "named.empty";
allow-update {
"none";
};
};Relevant logs and/or screenshots
Pstack result as followed:
Thread 19 (Thread 0x7f8a3d2fa700 (LWP 20008)):
#0 0x00007f8a3ee2f995 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3fafd4af in dispatch (manager=0x7f8a41514eb0) at task.c:1103
#2 run (uap=0x7f8a41514eb0) at task.c:1331
#3 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 18 (Thread 0x7f8a3caf9700 (LWP 20009)):
#0 0x00007f8a3ee2f995 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3fafd4af in dispatch (manager=0x7f8a41514eb0) at task.c:1103
#2 run (uap=0x7f8a41514eb0) at task.c:1331
#3 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 17 (Thread 0x7f8a3c2f8700 (LWP 20010)):
#0 0x00007f8a3ee3251d in __lll_lock_wait () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3ee2de51 in _L_lock_1022 () from /usr/lib64/libpthread.so.0
#2 0x00007f8a3ee2ddf2 in pthread_mutex_lock () from /usr/lib64/libpthread.so.0
#3 0x00007f8a40e20cd5 in empty_bucket (res=0x7f87d17a14b8) at resolver.c:8910
#4 0x00007f8a40e26c51 in fctx_doshutdown (task=<optimized out>, event=<optimized out>) at resolver.c:4144
#5 0x00007f8a3fafd6bb in dispatch (manager=0x7f8a41514eb0) at task.c:1157
#6 run (uap=0x7f8a41514eb0) at task.c:1331
#7 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#8 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 16 (Thread 0x7f8a3baf7700 (LWP 20011)):
#0 0x00007f8a3ee2f995 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3fafd4af in dispatch (manager=0x7f8a41514eb0) at task.c:1103
#2 run (uap=0x7f8a41514eb0) at task.c:1331
#3 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 15 (Thread 0x7f8a3b2f6700 (LWP 20012)):
#0 0x00007f8a3ee2f995 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3fafd4af in dispatch (manager=0x7f8a41514eb0) at task.c:1103
#2 run (uap=0x7f8a41514eb0) at task.c:1331
#3 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 14 (Thread 0x7f8a3aaf5700 (LWP 20013)):
#0 0x00007f8a3ee2f995 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3fafd4af in dispatch (manager=0x7f8a41514eb0) at task.c:1103
#2 run (uap=0x7f8a41514eb0) at task.c:1331
#3 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 13 (Thread 0x7f8a3a2f4700 (LWP 20014)):
#0 0x00007f8a3ee2f995 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3fafd4af in dispatch (manager=0x7f8a41514eb0) at task.c:1103
#2 run (uap=0x7f8a41514eb0) at task.c:1331
#3 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 12 (Thread 0x7f8a39af3700 (LWP 20015)):
#0 0x00007f8a3ee2f995 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3fafd4af in dispatch (manager=0x7f8a41514eb0) at task.c:1103
#2 run (uap=0x7f8a41514eb0) at task.c:1331
#3 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 11 (Thread 0x7f8a392f2700 (LWP 20016)):
#0 0x00007f8a3ee2f995 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3fafcc33 in isc__task_beginexclusive (task0=<optimized out>) at task.c:1757
#2 0x000000000046850b in load_configuration ()
#3 0x000000000046c1d5 in loadconfig ()
#4 0x000000000046c42e in ns_server_reconfigcommand ()
#5 0x00000000004373bd in ns_control_docommand ()
#6 0x000000000043a483 in control_recvmessage ()
#7 0x00007f8a3fafd6bb in dispatch (manager=0x7f8a41514eb0) at task.c:1157
#8 run (uap=0x7f8a41514eb0) at task.c:1331
#9 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#10 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 10 (Thread 0x7f8a38af1700 (LWP 20017)):
#0 0x00007f8a3ee2f995 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3fafd4af in dispatch (manager=0x7f8a41514eb0) at task.c:1103
#2 run (uap=0x7f8a41514eb0) at task.c:1331
#3 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 9 (Thread 0x7f8a382f0700 (LWP 20018)):
#0 0x00007f8a3ee2f995 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3fafd4af in dispatch (manager=0x7f8a41514eb0) at task.c:1103
#2 run (uap=0x7f8a41514eb0) at task.c:1331
#3 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 8 (Thread 0x7f8a37aef700 (LWP 20019)):
#0 0x00007f8a3ee3251d in __lll_lock_wait () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3ee2de51 in _L_lock_1022 () from /usr/lib64/libpthread.so.0
#2 0x00007f8a3ee2ddf2 in pthread_mutex_lock () from /usr/lib64/libpthread.so.0
#3 0x00007f8a40e253ee in dns_resolver_shutdown (res=0x7f87d17a14b8) at resolver.c:9399
#4 0x00007f8a40e650c9 in view_flushanddetach (viewp=<optimized out>, flush=<optimized out>) at view.c:601
#5 0x000000000042e387 in exit_check ()
#6 0x0000000000443310 in prefetch_done ()
#7 0x00007f8a3fafd6bb in dispatch (manager=0x7f8a41514eb0) at task.c:1157
#8 run (uap=0x7f8a41514eb0) at task.c:1331
#9 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#10 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 7 (Thread 0x7f8a372ee700 (LWP 20020)):
#0 0x00007f8a3ee2f995 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3fafd4af in dispatch (manager=0x7f8a41514eb0) at task.c:1103
#2 run (uap=0x7f8a41514eb0) at task.c:1331
#3 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 6 (Thread 0x7f8a36aed700 (LWP 20021)):
#0 0x00007f8a3ee3251d in __lll_lock_wait () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3ee2de51 in _L_lock_1022 () from /usr/lib64/libpthread.so.0
#2 0x00007f8a3ee2ddf2 in pthread_mutex_lock () from /usr/lib64/libpthread.so.0
#3 0x00007f8a40e20cd5 in empty_bucket (res=0x7f87d17a14b8) at resolver.c:8910
#4 0x00007f8a40d35d9a in fetch_callback (task=<optimized out>, ev=0x7f85a1732568) at adb.c:3868
#5 0x00007f8a3fafd6bb in dispatch (manager=0x7f8a41514eb0) at task.c:1157
#6 run (uap=0x7f8a41514eb0) at task.c:1331
#7 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#8 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 5 (Thread 0x7f8a362ec700 (LWP 20022)):
#0 0x00007f8a3ee3251d in __lll_lock_wait () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3ee2de51 in _L_lock_1022 () from /usr/lib64/libpthread.so.0
#2 0x00007f8a3ee2ddf2 in pthread_mutex_lock () from /usr/lib64/libpthread.so.0
#3 0x00007f8a40e62106 in dns_view_findzonecut2 (view=0x7f88f1f49240, name=name@entry=0x7f851fd3cd08, fname=fname@entry=0x7f8a362e9070, now=now@entry=0, options=0, use_hints=use_hints@entry=true, use_cache=use_cache@entry=true, rdataset=rdataset@entry=0x7f89103158e8, sigrdataset=sigrdataset@entry=0x0) at view.c:1295
#4 0x00007f8a40e625e8 in dns_view_findzonecut (view=<optimized out>, name=name@entry=0x7f851fd3cd08, fname=fname@entry=0x7f8a362e9070, now=now@entry=0, options=<optimized out>, use_hints=use_hints@entry=true, rdataset=rdataset@entry=0x7f89103158e8, sigrdataset=sigrdataset@entry=0x0) at view.c:1256
#5 0x00007f8a40e23b62 in fctx_create (res=res@entry=0x7f87d17a14b8, name=name@entry=0x7f851fd3cd08, type=type@entry=1, domain=0x7f8a362e9070, domain@entry=0x0, nameservers=nameservers@entry=0x0, client=client@entry=0x0, options=options@entry=32, bucketnum=bucketnum@entry=400, depth=depth@entry=2, qc=qc@entry=0xf7d4918, fctxp=fctxp@entry=0x7f8a362e9f98) at resolver.c:4454
#6 0x00007f8a40e25dd8 in dns_resolver_createfetch3 (res=<optimized out>, name=name@entry=0x7f851fd3cd08, type=type@entry=1, domain=domain@entry=0x0, nameservers=nameservers@entry=0x0, forwarders=forwarders@entry=0x0, client=client@entry=0x0, id=id@entry=0, options=options@entry=32, depth=depth@entry=2, qc=qc@entry=0xf7d4918, task=0x7f8930a85ae8, action=action@entry=0x7f8a40d35c90 <fetch_callback>, arg=arg@entry=0x7f851fd3cd00, rdataset=rdataset@entry=0x7f851fd39088, sigrdataset=sigrdataset@entry=0x0, fetchp=fetchp@entry=0x7f851fd39080) at resolver.c:9630
#7 0x00007f8a40d30269 in fetch_name (adbname=adbname@entry=0x7f851fd3cd00, start_at_zone=start_at_zone@entry=false, depth=depth@entry=2, qc=qc@entry=0xf7d4918, type=type@entry=1) at adb.c:4056
#8 0x00007f8a40d39dde in dns_adb_createfind2 (adb=0x7f851fd08aa0, task=0x7f89251db3f0, action=action@entry=0x7f8a40e2b4b0 <fctx_finddone>, arg=arg@entry=0x7f88eadec120, name=name@entry=0x7f8a362eadd0, qname=qname@entry=0x7f88eadec130, qtype=28, options=223, now=now@entry=1601867731, target=target@entry=0x0, port=53, depth=2, qc=0xf7d4918, findp=findp@entry=0x7f8a362ea8b8) at adb.c:3192
#9 0x00007f8a40e1f92d in findname (fctx=fctx@entry=0x7f88eadec120, name=name@entry=0x7f8a362eadd0, port=port@entry=0, options=<optimized out>, options@entry=31, flags=flags@entry=0, now=1601867731, overquota=overquota@entry=0x7f8a362ead70, need_alternate=need_alternate@entry=0x7f8a362ead57, no_addresses=no_addresses@entry=0x7f8a362ead5c) at resolver.c:3166
#10 0x00007f8a40e27ca2 in fctx_getaddresses (fctx=fctx@entry=0x7f88eadec120, badcache=badcache@entry=false) at resolver.c:3462
#11 0x00007f8a40e2a36a in fctx_try (fctx=0x7f88eadec120, retrying=<optimized out>, badcache=<optimized out>) at resolver.c:3819
#12 0x00007f8a40e2d994 in resquery_response (task=<optimized out>, event=<optimized out>) at resolver.c:8747
#13 0x00007f8a3fafd6bb in dispatch (manager=0x7f8a41514eb0) at task.c:1157
#14 run (uap=0x7f8a41514eb0) at task.c:1331
#15 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#16 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 4 (Thread 0x7f8a35aeb700 (LWP 20023)):
#0 0x00007f8a3ee2f995 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3fafd4af in dispatch (manager=0x7f8a41514eb0) at task.c:1103
#2 run (uap=0x7f8a41514eb0) at task.c:1331
#3 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 3 (Thread 0x7f8a352ea700 (LWP 20024)):
#0 0x00007f8a3ee2fd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007f8a3fb1a3c8 in isc_condition_waituntil (c=c@entry=0x7f8a4151c078, m=m@entry=0x7f8a4151c028, t=t@entry=0x7f8a4151c06c) at condition.c:59
#2 0x00007f8a3fb036b3 in run (uap=0x7f8a4151c010) at timer.c:811
#3 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 2 (Thread 0x7f8a34ae9700 (LWP 20025)):
#0 0x00007f8a3e7ec183 in epoll_wait () from /usr/lib64/libc.so.6
#1 0x00007f8a3fb112f6 in watcher (uap=0x7f8a4151e010) at socket.c:4302
#2 0x00007f8a3ee2be25 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007f8a3e7ebbad in clone () from /usr/lib64/libc.so.6
Thread 1 (Thread 0x7f8a41555840 (LWP 20007)):
#0 0x00007f8a3e7235f2 in sigsuspend () from /usr/lib64/libc.so.6
#1 0x00007f8a3fb04f00 in isc__app_ctxrun (ctx0=ctx0@entry=0x7f8a3fd38dc0 <isc_g_appctx>) at app.c:723
#2 0x00007f8a3fb0515c in isc__app_run () at app.c:756
#3 0x00007f8a3fb05a30 in isc_app_run () at ../app_api.c:207
#4 0x000000000042a905 in main ()Possible fixes
(If you can, link to the line of code that might be responsible for the problem.)