GitLab

Summary

On FBSD 12.2 running chroot'd bind 9.16.12 listening on all v4/v6 interfaces and bhyve running without any VMs, named leaks memory each time it attempts to listen on the virbr0 virtual bridge interface. It leaks until it has consumed all the available system memory - 38G+. In roughly 20 minutes it has inflated to ~1G, about 300M more than its typical state. It also hangs if sent a signal 15 and never dumps the memstats file.

BIND version used

running on FreeBSD amd64 12.2-RELEASE-p4 FreeBSD 12.2-RELEASE-p4 GENERIC
built by make with '--disable-linux-caps' '--localstatedir=/var' '--sysconfdir=/usr/local/etc/namedb' '--with-dlopen=yes' '--with-libxml2' '--with-openssl=/usr' '--with-readline=-L/usr/local/lib -ledit' '--with-dlz-filesystem=yes' '--enable-dnstap' '--disable-fixed-rrset' '--disable-geoip' '--without-maxminddb' '--without-gssapi' '--with-libidn2=/usr/local' '--with-json-c' '--disable-largefile' '--with-lmdb=/usr/local' '--disable-native-pkcs11' '--without-python' '--disable-querytrace' '--enable-tcp-fastopen' '--disable-symtable' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/share/info/' '--build=amd64-portbld-freebsd12.2' 'build_alias=amd64-portbld-freebsd12.2' 'CC=cc' 'CFLAGS=-O2 -pipe -DLIBICONV_PLUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing ' 'LDFLAGS= -L/usr/local/lib -ljson-c -fstack-protector-strong ' 'LIBS=-L/usr/local/lib' 'CPPFLAGS=-DLIBICONV_PLUG -isystem /usr/local/include' 'CPP=cpp' 'PKG_CONFIG=pkgconf'
compiled by CLANG FreeBSD Clang 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
compiled with OpenSSL version: OpenSSL 1.1.1h-freebsd  22 Sep 2020
linked to OpenSSL version: OpenSSL 1.1.1h-freebsd  22 Sep 2020
compiled with libuv version: 1.40.0
linked to libuv version: 1.41.0
compiled with libxml2 version: 2.9.10
linked to libxml2 version: 20910
compiled with json-c version: 0.15
linked to json-c version: 0.15
compiled with zlib version: 1.2.11
linked to zlib version: 1.2.11
compiled with protobuf-c version: 1.3.2
linked to protobuf-c version: 1.3.2
threads support is enabled

default paths:
  named configuration:  /usr/local/etc/namedb/named.conf
  rndc configuration:   /usr/local/etc/namedb/rndc.conf
  DNSSEC root key:      /usr/local/etc/namedb/bind.keys
  nsupdate session key: /var/run/named/session.key
  named PID file:       /var/run/named/pid
  named lock file:      /var/run/named/named.lock

Steps to reproduce

Run named chrooted on freebsd with bhyve's virbr0 interface up with an IPv4 address.  Watch it leak memory.

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e503bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:d9:e0:0a
        inet xxxx netmask 0xffffff80 broadcast xxxx
        inet6 fe80::ec4:7aff:fed9:e00a%igb0 prefixlen 64 scopeid 0x1
        inet6 2001:418:3fe::4 prefixlen 80
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:d9:e0:0b
        inet xxxx netmask 0xffffffe0 broadcast xxxx
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
tap0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:97:78
        groups: tap
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:13:28:b1:48:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
vm-bb: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether d6:5f:8b:8b:fe:e1
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge vm-switch viid-21ad0@
        nd6 options=1<PERFORMNUD>
virbr0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 52:54:00:ef:5e:80
        inet 192.168.122.1 netmask 0xffffff00 broadcast 192.168.122.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 4
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        groups: bridge
        nd6 options=1<PERFORMNUD>

If virbr0 is down, the problem continues.  Only by removing the address will the logs and leak cease.  Adding a v6 address produces the same problem.[tracefile](/uploads/a520ef32cc74c02ecb3a1b9aec35f46f/tracefile)

Also see the attached tracefile; it shows permission denied trying to bind a socket to the address of virbr0.

### What is the current *bug* behavior?

Memory leak.  eg:
PID  USER      PRI  NI VIRT  RES   S CPU%  MEM%   TIME+   Command
2932 bind       21   0 15.0G 13.3G S  0.0  10.2   0:56.49 /usr/local/sbin/named -....

What is the expected correct behavior?

(What you should see instead.)

Relevant configuration files

        listen-on       { 127.0.0.1; };
        listen-on       { any; };
        listen-on-v6    { any; };

Relevant logs and/or screenshots

named[2918]: listening on IPv6 interface igb0, fe80::....
named[2918]: listening on IPv6 interface igb0, 2001:....
named[2918]: listening on IPv4 interface igb1, 198....
named[2918]: listening on IPv6 interface lo0, ::1#53
named[2918]: listening on IPv6 interface lo0, fe80::1%3#53
named[2918]: listening on IPv4 interface lo0, 127.0.0.1#53

Ad nauseam:
named[2918]: network: info: listening on IPv4 interface virbr0, 192.168.122.1#53
named[2918]: network: error: creating IPv4 interface virbr0 failed; interface ignored

Possible fixes

(If you can, link to the line of code that might be responsible for the problem.)