"tcp:checking that BIND 9 doesn't crash on long TCP messages" gets named oom-killed

Summary

TCP test consumes excessive amounts of memory, which is not accounted for in memory statistics.

BIND version used

• Affects v9.19: 1b0e7e7a
• Affects v9.18: ff3d25a4

Versions tested but NOT affected:

• v9.16: adb71aff

Steps to reproduce

• Run test step "tcp:checking that BIND 9 doesn't crash on long TCP messages" in a loop. Essentially:

for I in $(seq 1 10); do perl ../packet.pl -a "127.0.0.1" -p "53" -t tcp -r 300000 1996-alloc_dnsbuf-crash-test.pkt; sleep 5; done

Source: https://gitlab.isc.org/isc-projects/bind9/-/blob/1b0e7e7a50733366dfae39de93f85f2293b725ff/bin/tests/system/tcp/tests.sh#L197

What is the current bug behavior?

Kernel shows excessive memory consumption and at the same time memory statistics in BIND do not show it. Apparently it is really consumed by something because sometimes OOM killer comes to the rescue.

Job #3348347 failed for 0f25d62a.

I:tcp:checking that BIND 9 doesn't crash on long TCP messages (10)
I:tcp:sending 300000 time(s): 00010000000100000000000003697363036f72670000fc0001
I:tcp:failed
I:tcp:exit status: 1
I:tcp:stopping servers
I:tcp:ns1 died before a SIGTERM was sent
I:tcp:ns1 didn't die when sent a SIGTERM
I:tcp:stopping servers failed
R:tcp:FAIL

After a local investigation, this is a likely case of running out of memory. This is what I get locally:

oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-3.scope,task=lt-named,pid=68053,uid=1000
Out of memory: Killed process 68053 (lt-named) total-vm:6015604kB, anon-rss:3451292kB, file-rss:5652kB, shmem-rss:0kB, UID:1000 pgtables:10744kB oom_score_adj:0

We should bump VM's memory to more than 4G, or rather don't run this particular test on QCOW2 images.