[question] Bind 9.16.28 upgrade: high memory utilization and OOM
We had recently upgraded our bind nameservers from 9.14.10 to 9.16.28. This led to the hosts gradually using up a lot of memory and eventually named was OOM killed as it consumed nearly 7GB out of total 8GB server memory. (This package was built from source for centos 7)
I’ve been looking into this and tested the performance of both 9.14 and 9.16 under the traffic of 600 queries per sec for 12 hours, which is the average qps our servers get. It was found that while 9.14 had a surge of around 2GB, 9.16 had a surge of 5.2GB during this time. I wanted to know whether this difference in memory consumption is expected while migrating from 9.14.10 to 9.16.28, or if this could be a memory leak that would keep building over time; it would really help if I can get some insights on what might be causing this, or if there’s any way to avoid this other than bumping up the RAM.
I’d be glad to provide more info if needed. Would really appreciate your inputs and suggestions on this.
Its running on 3.10.0-957.10.1.el7.x86_64, named is running as a systemd service, I'll attach configuration data and stats soon, compile time options below:
# named -V
BIND 9.16.28 (Extended Support Version) <id:7aea13f>
running on Linux x86_64 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019
built by make with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-python=/usr/bin/python2' '--with-libtool' '--localstatedir=/var' '--with-pic' '--disable-static' '--includedir=/usr/include/bind9' '--with-tuning=large' '--with-libidn2' '--disable-lock-free-queue' '--with-maxminddb' '--with-dlopen=yes' '--with-gssapi=yes' '--with-lmdb=yes' '--without-libjson' '--with-json-c' '--enable-dnstap' '--enable-fixed-rrset' '--enable-full-report' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS= -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro ' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'
compiled by GCC 4.8.5 20150623 (Red Hat 4.8.5-44)
compiled with OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017
linked to OpenSSL version: OpenSSL 1.0.2p.6.2.312-fips
compiled with libuv version: 1.44.1
linked to libuv version: 1.44.1
compiled with libxml2 version: 2.9.1
linked to libxml2 version: 20901
compiled with json-c version: 0.11
linked to json-c version: 0.11
compiled with zlib version: 1.2.7
linked to zlib version: 1.2.7
linked to maxminddb version: 1.2.0
compiled with protobuf-c version: 1.0.2
linked to protobuf-c version: 1.0.2
threads support is enabled
default paths:
named configuration: /etc/named.conf
rndc configuration: /etc/rndc.conf
DNSSEC root key: /etc/bind.keys
nsupdate session key: /var/run/named/session.key
named PID file: /var/run/named/named.pid
named lock file: /var/run/named/named.lock
geoip-directory: /usr/share/GeoIP