Statistics XML rendering does not seem to protect against concurrent changes

Summary

BIND version used

BIND 9.11.26-RedHat-9.11.26-6.el8 (Extended Support Version) <id:3ff8620>
running on Linux x86_64 4.18.0-348.el8.x86_64 #1 SMP Mon Oct 4 12:17:22 EDT 2021
built by make with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-python=/usr/libexec/platform-python' '--with-libtool' '--localstatedir=/var' '--enable-threads' '--enable-ipv6' '--enable-filter-aaaa' '--with-pic' '--disable-static' '--includedir=/usr/include/bind9' '--with-tuning=large' '--with-libidn2' '--enable-openssl-hash' '--with-geoip2' '--enable-native-pkcs11' '--with-pkcs11=/usr/lib64/pkcs11/libsofthsm2.so' '--with-dlopen=yes' '--with-dlz-ldap=yes' '--with-dlz-postgres=yes' '--with-dlz-mysql=yes' '--with-dlz-filesystem=yes' '--with-dlz-bdb=yes' '--with-gssapi=yes' '--disable-isc-spnego' '--with-lmdb=no' '--with-libjson' '--enable-dnstap' '--with-cmocka' '--enable-fixed-rrset' '--with-docbook-xsl=/usr/share/sgml/docbook/xsl-stylesheets' '--enable-full-report' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS= -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'LDFLAGS=-Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' 'CPPFLAGS= -DDIG_SIGCHASE' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'
compiled by GCC 8.5.0 20210514 (Red Hat 8.5.0-3)
compiled with OpenSSL version: OpenSSL 1.1.1k  FIPS 25 Mar 2021
linked to OpenSSL version: OpenSSL 1.1.1k  FIPS 25 Mar 2021
compiled with libxml2 version: 2.9.7
linked to libxml2 version: 20907
compiled with libjson-c version: 0.13.1
linked to libjson-c version: 0.13.1
compiled with zlib version: 1.2.11
linked to zlib version: 1.2.11
linked to maxminddb version: 1.2.0
compiled with protobuf-c version: 1.3.0
linked to protobuf-c version: 1.3.0
threads support is enabled

default paths:
  named configuration:  /etc/named.conf
  rndc configuration:   /etc/rndc.conf
  DNSSEC root key:      /etc/bind.keys
  nsupdate session key: /var/run/named/session.key
  named PID file:       /var/run/named/named.pid
  named lock file:      /var/run/named/named.lock
  geoip-directory:      /usr/share/GeoIP

Steps to reproduce

unreliable, not known exactly
have statistics channel enabled and used periodically
use bind-dyndb-ldap
perform runtime changes to

What is the current bug behavior?

Reported as RHBZ bug happening on 9.11
Haven't found any relevant change even on latest v9_16 branch
Crashes on invalid pointer in dns_zt_apply(), called from render_xml.

Thread 3 (Thread 0x7f98198c1700 (LWP 79413)):
#0  0x00007f981de2c3fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9820a29f8a in isc__rwlock_lock (type=isc_rwlocktype_write, rwl=0x55cc8a4b2670) at ../../../lib/isc-pkcs11/rwlock.c:398
#2  isc_rwlock_lock (rwl=rwl@entry=0x55cc8a4b2670, type=type@entry=isc_rwlocktype_write) at ../../../lib/isc-pkcs11/rwlock.c:426
#3  0x00007f9820e137ea in dns_zonemgr_releasezone (zone=0x7f97e5d167a0, zmgr=0x55cc8a4b2600) at ../../../lib/dns-pkcs11/zone.c:16946
#4  dns_zonemgr_releasezone (zmgr=0x55cc8a4b2600, zone=zone@entry=0x7f97e5d167a0) at ../../../lib/dns-pkcs11/zone.c:16939
#5  0x00007f9815e13d89 in delete_bind_zone (zt=zt@entry=0x7f9800341150, zonep=zonep@entry=0x7f98198bfea0) at zone_register.c:598
#6  0x00007f9815df64ed in empty_zone_unload (ezname=ezname@entry=0x7f98198c0070, zonetable=zonetable@entry=0x7f9800341150) at empty_zones.c:270
#7  0x00007f9815df662f in empty_zone_handle_conflicts (name=name@entry=0x7f98210c07c0 <root>, zonetable=0x7f9800341150, warn_only=false) at empty_zones.c:326
#8  0x00007f9815df7e43 in fwd_configure_zone (set=0x7f97fc3ea0e0, inst=inst@entry=0x7f980555a618, name=0x7f98210c07c0 <root>) at fwd.c:599
#9  0x00007f9815df81f4 in fwd_reconfig_global (inst=inst@entry=0x7f980555a618) at fwd.c:655
#10 0x00007f9815e013c0 in ldap_parse_serverconfigentry (inst=0x7f980555a618, entry=0x7f9812fbfd70) at ldap_helper.c:1535
#11 update_serverconfig (task=<optimized out>, event=<optimized out>) at ldap_helper.c:3844
#12 0x00007f9820a300ef in dispatch (manager=0x7f98214b3010) at ../../../lib/isc-pkcs11/task.c:1157
#13 run (uap=0x7f98214b3010) at ../../../lib/isc-pkcs11/task.c:1331
#14 0x00007f981de2617a in start_thread () from /lib64/libpthread.so.0
#15 0x00007f981d7eddc3 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f98188bf700 (LWP 79415)):
#0  0x00007f9820d2f513 in dns_rbtnodechain_next (chain=chain@entry=0x7f98188bcad0, name=name@entry=0x0, origin=origin@entry=0x0) at ../../../lib/dns-pkcs11/rbt.c:3516
#1  0x00007f9820e2bb98 in dns_zt_apply2 (zt=<optimized out>, stop=stop@entry=true, sub=sub@entry=0x0, action=action@entry=0x55cc88636e10 <zone_xmlrender>, uap=uap@entry=0x7f97f710e210) at ../../../lib/dns-pkcs11/zt.c:534
#2  0x00007f9820e2bf25 in dns_zt_apply (zt=<optimized out>, stop=stop@entry=true, action=action@entry=0x55cc88636e10 <zone_xmlrender>, uap=uap@entry=0x7f97f710e210) at ../../../lib/dns-pkcs11/zt.c:496
#3  0x000055cc886365ec in generatexml (server=<optimized out>, flags=2, buflen=buflen@entry=0x7f98188bebbc, buf=buf@entry=0x7f98188bebc0) at ../../../bin/named-pkcs11/statschannel.c:1842
#4  0x000055cc88636b55 in render_xml (flags=<optimized out>, arg=<optimized out>, retcode=0x7f97f558d518, retmsg=0x7f97f558d520, mimetype=0x7f97f558d510, b=0x7f97f558d528, freecb=0x7f97f558d568, freecb_args=0x7f97f558d570, 
    headers=<optimized out>, querystring=<optimized out>, urlinfo=<optimized out>, url=<optimized out>) at ../../../bin/named-pkcs11/statschannel.c:1995
#5  0x00007f9820a14040 in isc_httpd_recvdone (task=0x7f9821419970, ev=<optimized out>) at ../../../lib/isc-pkcs11/httpd.c:1015
#6  0x00007f9820a300ef in dispatch (manager=0x7f98214b3010) at ../../../lib/isc-pkcs11/task.c:1157
#7  run (uap=0x7f98214b3010) at ../../../lib/isc-pkcs11/task.c:1331
#8  0x00007f981de2617a in start_thread () from /lib64/libpthread.so.0
#9  0x00007f981d7eddc3 in clone () from /lib64/libc.so.6

What is the expected correct behavior?

Some obvious ZONE_LOCK happens to processed zones. If there is alternative protection against changes in zonemgr, please point me to it. I could not find how it is supposed to protect against invalidation of pointers. Does it mean named would never remove any zone from zone manager during normal runtime, where render_xml events are handled?

Relevant configuration files

dyndb "ipa" "/usr/lib64/bind/ldap.so" {
        uri "ldapi://%2fvar%2frun%2fslapd-CENSORED.socket";
        base "cn=dns,dc=example,dc=net";
        server_id "cnec2idm01.example.net";
        auth_method "sasl";
        sasl_mech "GSSAPI";
        sasl_user "DNS/cnec2idm01.example.net";
};

statistics-channels { inet 127.0.0.1 port 8053 allow { 127.0.0.1; }; };

Relevant logs and/or screenshots

(none)

Possible fixes

It is quite possible the problem lies in bind-dyndb-ldap plugin. It tries to imitate most of behaviours of real named configuration from text files. If provides zone definitions stored in LDAP. Unlike normal named it allows adding also forward zones runtime without a reconfiguration. I suspect that might be related, because such functionality is not possible (I think) even in the latest BIND9. If those were done by wrong design, please could you help making it more reliable?

dns_zt_load uses read lock as a protection.
bin/named/server.c does not use such protection. I am not sure why, but that seems error to me.

Edited Jul 21, 2022 by Petr Menšík