Bind9 9.16.33 (Debian 11.5) - High CPU usage with DNSSEC zones constantly resigned
Summary
Debian 11 (bullseye) pushed out an upgrade to 9.16.33 in response to a security flaw. Similar as others reported (eg https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1020529) our production infrastructure was affected by this change. We subsequently added the 'inline-signing yes' option to any zone that references a 'dnssec-policy' where that zone does not allow dynamic updates. Whilst Bind now starts we are experiencing high CPU usage and zone serial numbers keep on incrementing.
Initial symptom was that we were receiving messages such as the following for zones: Sep 24 23:58:55 unix-01 named[553]: zone redacted.co.za/IN (signed): update_soa_serial:new serial would be lower than old serial, using increment method instead
We subsequently left 'serial-update-method date' in the global options but set 'serial-update-method increment' for any DNSSEC enabled zones. This however really just masks the underlying problem with Bind constantly and unnecessarily resigning zones.
CPU utilisation since upgrading:
Oddly 'htop' doesn't show the CPU utilisation but standard 'top' does:
If we turn up logging to 'severity debug 3' we observe a constant stream of DNSSEC operations in the logs. We really only have about 40 DNSSEC signed zones at this stage:
tail -f /var/log/named/bind.log
zone_maintenance: zone 154.redacted.in-addr.arpa/IN (signed): enter
zone_nsec3chain: zone 154.redacted.in-addr.arpa/IN (signed): enter
zone_settimer: zone 150.redacted.in-addr.arpa/IN (signed): enter
zone_timer: zone 132.redacted.in-addr.arpa/IN (signed): enter
zone_maintenance: zone 132.redacted.in-addr.arpa/IN (signed): enter
zone_nsec3chain: zone 132.redacted.in-addr.arpa/IN (signed): enter
zone_settimer: zone 154.redacted.in-addr.arpa/IN (signed): enter
zone_timer: zone 157.redacted.in-addr.arpa/IN (signed): enter
zone_maintenance: zone 157.redacted.in-addr.arpa/IN (signed): enter
zone_nsec3chain: zone 157.redacted.in-addr.arpa/IN (signed): enter
zone_settimer: zone 132.redacted.in-addr.arpa/IN (signed): enter
zone_timer: zone 141.redacted.in-addr.arpa/IN (signed): enter
zone_maintenance: zone 141.redacted.in-addr.arpa/IN (signed): enter
zone_nsec3chain: zone 141.redacted.in-addr.arpa/IN (signed): enter
zone_settimer: zone 157.redacted.in-addr.arpa/IN (signed): enter
zone_timer: zone 142.redacted.in-addr.arpa/IN (signed): enter
zone_maintenance: zone 142.redacted.in-addr.arpa/IN (signed): enter
zone_nsec3chain: zone 142.redacted.in-addr.arpa/IN (signed): enter
zone_settimer: zone 141.redacted.in-addr.arpa/IN (signed): enter
zone_timer: zone 71.redacted.in-addr.arpa/IN (signed): enter
zone_maintenance: zone 71.redacted.in-addr.arpa/IN (signed): enter
zone_nsec3chain: zone 71.redacted.in-addr.arpa/IN (signed): enter
zone_settimer: zone 142.redacted.in-addr.arpa/IN (signed): enter
zone_timer: zone 145.redacted.in-addr.arpa/IN (signed): enter
zone_maintenance: zone 145.redacted.in-addr.arpa/IN (signed): enter
zone_settimer: zone 71.redacted.in-addr.arpa/IN (signed): enter
zone_nsec3chain: zone 145.redacted.in-addr.arpa/IN (signed): enter
zone_timer: zone 76.redacted.in-addr.arpa/IN (signed): enter
PS: There unfortunately isn't a date stamp but this is producing thousands of lines a second...
BIND version used
BIND 9.16.33-Debian (Extended Support Version) id:35e9c6e running on Linux x86_64 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) built by make with '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=/usr/lib/x86_64-linux-gnu' '--runstatedir=/run' '--disable-maintainer-mode' '--disable-dependency-tracking' '--libdir=/usr/lib/x86_64-linux-gnu' '--sysconfdir=/etc/bind' '--with-python=python3' '--localstatedir=/' '--enable-threads' '--enable-largefile' '--with-libtool' '--enable-shared' '--enable-static' '--with-gost=no' '--with-openssl=/usr' '--with-gssapi=/usr' '--with-libidn2' '--with-json-c' '--with-lmdb=/usr' '--with-gnu-ld' '--with-maxminddb' '--with-atf=no' '--enable-ipv6' '--enable-rrl' '--enable-filter-aaaa' '--disable-native-pkcs11' '--enable-dnstap' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -ffile-prefix-map=/build/bind9-kvFb77/bind9-9.16.33=. -fstack-protector-strong -Wformat -Werror=format-security -fno-strict-aliasing -fno-delete-null-pointer-checks -DNO_VERSION_DATE -DDIG_SIGCHASE' 'LDFLAGS=-Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' compiled by GCC 10.2.1 20210110 compiled with OpenSSL version: OpenSSL 1.1.1n 15 Mar 2022 linked to OpenSSL version: OpenSSL 1.1.1n 15 Mar 2022 compiled with libuv version: 1.40.0 linked to libuv version: 1.40.0 compiled with libxml2 version: 2.9.10 linked to libxml2 version: 20910 compiled with json-c version: 0.15 linked to json-c version: 0.15 compiled with zlib version: 1.2.11 linked to zlib version: 1.2.11 linked to maxminddb version: 1.5.2 compiled with protobuf-c version: 1.3.3 linked to protobuf-c version: 1.3.3 threads support is enabled
default paths: named configuration: /etc/bind/named.conf rndc configuration: /etc/bind/rndc.conf DNSSEC root key: /etc/bind/bind.keys nsupdate session key: //run/named/session.key named PID file: //run/named/named.pid named lock file: //run/named/named.lock geoip-directory: /usr/share/GeoIP
Steps to reproduce
/etc/bind/named.conf.options includes the following DNSSEC policy:
dnssec-policy alg13-ksk-unlimited-zsk-60day {
signatures-refresh P2D;
signatures-validity P7D;
signatures-validity-dnskey P7D;
keys {
ksk key-directory lifetime unlimited algorithm ECDSAP256SHA256;
zsk key-directory lifetime P60D algorithm ECDSAP256SHA256;
};
nsec3param iterations 10 optout no salt-length 16;
};
We have two types of zones. Herewith a static zone example:
zone "redacted.co.za" {
type master;
inline-signing yes;
serial-update-method increment;
dnssec-policy alg13-ksk-unlimited-zsk-60day;
file "master/redatec.co.za.domain";
};
And zones which allow dynamic updates:
zone "redacted2.co.za" {
type master;
serial-update-method increment;
dnssec-policy alg13-ksk-unlimited-zsk-60day;
file "master/redacted2.co.za.domain";
allow-update {
key firewall.redacted2.co.za.;
};
};
What is the current bug behavior?
High CPU utilisation, .signed files are constantly rewritten and zone SOAs constantly increment.
What is the expected correct behavior?
Zones should only be signed when the policy dictates that they should be updated