Resetting established TCP connections on shutdown crashes named on Solaris
When investigating !6111 (comment 279611) I noticed that dozens of system tests failed on Solaris 11.4.42 CBE and illumos, many with core dumps.
The checkds system test is a good reproducer (though many other fail, perhaps from different reasons). The first bad commit is !5955 (6ddac2d5) (!5955 (a761aa59) is the last good one).
The upcoming BIND 9.19.0 and 9.18.2 releases are affected.
Solaris 11
checkds NS9:
D:checkds:Core was generated by `/export/home/newman/bind9/bin/named/.libs/named -D checkds-ns9 -X named.lock -m'.
D:checkds:Program terminated with signal SIGKILL, Killed.
D:checkds:#0 0x00007fffbf2790fa in __nanosleep () from /usr/lib/amd64/libc.so.1
D:checkds:[Current thread is 12 (LWP 1 )]
D:checkds:#0 0x00007fffbf2790fa in __nanosleep () from /usr/lib/amd64/libc.so.1
D:checkds:#1 0x00007fffbf2600f3 in nanosleep () from /usr/lib/amd64/libc.so.1
D:checkds:#2 0x00007fffbf260834 in usleep () from /usr/lib/amd64/libc.so.1
D:checkds:#3 0x00007fffba87ad68 in isc__taskmgr_destroy (managerp=managerp@entry=0x583188 <named_g_taskmgr>) at task.c:1045
D:checkds:#4 0x00007fffba86b796 in isc_managers_destroy (netmgrp=netmgrp@entry=0x583170 <named_g_netmgr>, taskmgrp=taskmgrp@entry=0x583188 <named_g_taskmgr>, timermgrp=timermgrp@entry=0x583178 <named_g_timermgr>) at managers.c:101
D:checkds:#5 0x0000000000442c17 in destroy_managers () at main.c:935
D:checkds:#6 cleanup () at main.c:1257
D:checkds:#7 main (argc=16, argv=0x7fffbfffda38) at main.c:1493checkds in full
S:checkds:2022-04-14T14:18:59+0200
T:checkds:1:A
A:checkds:System test checkds
I:checkds:PORTS:5609,5610,5611,5612,5613,5614,5615,5616,5617,5618,5619,5620,5621
I:ns9:ns9/setup.sh
I:ns9:setting up zone: dspublished.checkds
I:ns9:setting up zone: reference.checkds
I:ns9:setting up zone: missing-dspublished.checkds
I:ns9:setting up zone: bad-dspublished.checkds
I:ns9:setting up zone: multiple-dspublished.checkds
I:ns9:setting up zone: incomplete-dspublished.checkds
I:ns9:setting up zone: bad2-dspublished.checkds
I:ns9:setting up zone: dswithdrawn.checkds
I:ns9:setting up zone: missing-dswithdrawn.checkds
I:ns9:setting up zone: bad-dswithdrawn.checkds
I:ns9:setting up zone: multiple-dswithdrawn.checkds
I:ns9:setting up zone: incomplete-dswithdrawn.checkds
I:ns9:setting up zone: bad2-dswithdrawn.checkds
I:ns5:ns5/setup.sh
I:ns2:ns2/setup.sh
I:checkds:starting servers
D:checkds:============================= test session starts ==============================
D:checkds:platform sunos5 -- Python 3.7.10, pytest-4.4.0, py-1.8.0, pluggy-0.9.0 -- /usr/bin/python3.7
D:checkds:cachedir: .pytest_cache
D:checkds:hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/export/home/newman/bind9/bin/tests/system/checkds/.hypothesis/examples')
D:checkds:rootdir: /export/home/newman/bind9/bin/tests/system/checkds
D:checkds:plugins: hypothesis-4.57.1
D:checkds:collecting ... collected 2 items
D:checkds:
D:checkds:tests-checkds.py::test_checkds_dspublished SKIPPED [ 50%]
D:checkds:tests-checkds.py::test_checkds_dswithdrawn SKIPPED [100%]
D:checkds:
D:checkds:========================== 2 skipped in 0.03 seconds ===========================
I:checkds:stopping servers
I:checkds:ns9 crashed on shutdown
...
I:checkds:ns9 crashed on shutdown
I:checkds:ns9 didn't die when sent a SIGTERM
I:checkds:ns9 died before a SIGABRT was sent
I:checkds:stopping servers failed
I:checkds:Core dump(s) found: checkds/ns9/core
D:checkds:backtrace from checkds/ns9/core:
D:checkds:--------------------------------------------------------------------------------
D:checkds:Core was generated by `/export/home/newman/bind9/bin/named/.libs/named -D checkds-ns9 -X named.lock -m'.
D:checkds:Program terminated with signal SIGKILL, Killed.
D:checkds:#0 0x00007fffbf2790fa in __nanosleep () from /usr/lib/amd64/libc.so.1
D:checkds:[Current thread is 12 (LWP 1 )]
D:checkds:#0 0x00007fffbf2790fa in __nanosleep () from /usr/lib/amd64/libc.so.1
D:checkds:#1 0x00007fffbf2600f3 in nanosleep () from /usr/lib/amd64/libc.so.1
D:checkds:#2 0x00007fffbf260834 in usleep () from /usr/lib/amd64/libc.so.1
D:checkds:#3 0x00007fffba87ad68 in isc__taskmgr_destroy (managerp=managerp@entry=0x583188 <named_g_taskmgr>) at task.c:1045
D:checkds:#4 0x00007fffba86b796 in isc_managers_destroy (netmgrp=netmgrp@entry=0x583170 <named_g_netmgr>, taskmgrp=taskmgrp@entry=0x583188 <named_g_taskmgr>, timermgrp=timermgrp@entry=0x583178 <named_g_timermgr>) at managers.c:101
D:checkds:#5 0x0000000000442c17 in destroy_managers () at main.c:935
D:checkds:#6 cleanup () at main.c:1257
D:checkds:#7 main (argc=16, argv=0x7fffbfffda38) at main.c:1493
D:checkds:--------------------------------------------------------------------------------
D:checkds:full backtrace from checkds/ns9/core saved in checkds/ns9/core-backtrace.txt
D:checkds:core dump checkds/ns9/core archived as checkds/ns9/core.gz
R:checkds:FAIL
E:checkds:2022-04-14T14:20:07+0200
FAIL: checkdssolaris-checkds-ns9-core-backtrace.txt
solaris-checkds-ns9-named.run:
14-Apr-2022 14:19:04.142 netmgr/udp.c:288: unexpected error:
14-Apr-2022 14:19:04.142 unable to convert libuv error code in route_connect_direct to isc_result: -99: protocol not available
14-Apr-2022 14:19:04.142 unable to open route socket: unexpected error
...
14-Apr-2022 14:19:04.167 route_connected: unexpected error
...
14-Apr-2022 14:19:04.690 netmgr/netmgr.c:2847: fatal error:
14-Apr-2022 14:19:04.690 uv_tcp_close_reset failed: invalid argument
14-Apr-2022 14:19:04.690 exiting (due to fatal error in library)OpenIndiana 2021.10 (illumos-9675e051a2)
checkds NS9:
D:checkds:Core was generated by `/export/home/newman/bind9/bin/named/.libs/named -D checkds-ns9 -X named.lock -m'.
D:checkds:Program terminated with signal SIGKILL, Killed.
D:checkds:#0 0x00007fffae566faa in _lwp_kill () from /lib/64/libc.so.1
D:checkds:[Current thread is 22 (Thread 2 (LWP 2))]
D:checkds:#0 0x00007fffae566faa in _lwp_kill () from /lib/64/libc.so.1
D:checkds:#1 0x00007fffae55d870 in thr_kill () from /lib/64/libc.so.1
D:checkds:#2 0x00007fffae4faefe in raise () from /lib/64/libc.so.1
D:checkds:#3 0x00007fffae4d4b58 in abort () from /lib/64/libc.so.1
D:checkds:#4 0x0000000000435016 in library_fatal_error (file=0x7fffa7dd25f5 "netmgr/netmgr.c", line=2847, format=<optimized out>, args=0x7fffabb09738) at main.c:277
D:checkds:#5 0x00007fffa7d2daa2 in isc_error_fatal (file=file@entry=0x7fffa7dd25f5 "netmgr/netmgr.c", line=line@entry=2847, format=format@entry=0x7fffa7dd2b5c "%s failed: %s\n") at error.c:68
D:checkds:#6 0x00007fffa7ce5443 in isc__nmsocket_reset (sock=sock@entry=0x7fffaade3c00) at netmgr.c:2847
D:checkds:#7 0x00007fffa7ce56de in shutdown_walk_cb (handle=0x7fffaade42f8, arg=<optimized out>) at netmgr.c:2900
D:checkds:#8 0x00007fffaf0f89b7 in uv_walk (loop=loop@entry=0x7fffac150010, walk_cb=walk_cb@entry=0x7fffa7ce55ee <shutdown_walk_cb>, arg=arg@entry=0x0) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/uv-common.c:500
D:checkds:#9 0x00007fffa7ce573b in isc__nm_async_shutdown (worker=worker@entry=0x7fffac150000, ev0=ev0@entry=0x5fffbe8f3480) at netmgr.c:2918
D:checkds:#10 0x00007fffa7cedb80 in process_netievent (worker=worker@entry=0x7fffac150000, ievent=ievent@entry=0x5fffbe8f3480) at netmgr.c:927
D:checkds:#11 0x00007fffa7cee157 in process_queue (worker=worker@entry=0x7fffac150000, type=type@entry=NETIEVENT_NORMAL) at netmgr.c:958
D:checkds:#12 0x00007fffa7cefadc in process_all_queues (worker=0x7fffac150000) at netmgr.c:711
D:checkds:#13 async_cb (handle=0x7fffac150350) at netmgr.c:741
D:checkds:#14 0x00007fffaf0f9666 in uv__async_io (loop=0x7fffac150010, w=<optimized out>, events=<optimized out>) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/unix/async.c:163
D:checkds:#15 0x00007fffaf10a1b4 in uv__io_poll (loop=loop@entry=0x7fffac150010, timeout=<optimized out>) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/unix/sunos.c:288
D:checkds:#16 0x00007fffaf0f9e33 in uv_run (loop=loop@entry=0x7fffac150010, mode=mode@entry=UV_RUN_DEFAULT) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/unix/core.c:381
D:checkds:#17 0x00007fffa7ceeb08 in nm_thread (worker0=0x7fffac150000) at netmgr.c:642
D:checkds:#18 0x00007fffa7d963c0 in isc__trampoline_run (arg=0x7fffac2a0080) at trampoline.c:187
D:checkds:#19 0x00007fffae55fb8c in _thrp_setup () from /lib/64/libc.so.1
D:checkds:#20 0x00007fffae55fea0 in ?? () from /lib/64/libc.so.1
D:checkds:#21 0x0000000000000000 in ?? ()checkds in full
S:checkds:2022-04-14T00:31:27+0200
T:checkds:1:A
A:checkds:System test checkds
I:checkds:PORTS:5595,5596,5597,5598,5599,5600,5601,5602,5603,5604,5605,5606,5607
I:ns9:ns9/setup.sh
I:ns9:setting up zone: dspublished.checkds
I:ns9:setting up zone: reference.checkds
I:ns9:setting up zone: missing-dspublished.checkds
I:ns9:setting up zone: bad-dspublished.checkds
I:ns9:setting up zone: multiple-dspublished.checkds
I:ns9:setting up zone: incomplete-dspublished.checkds
I:ns9:setting up zone: bad2-dspublished.checkds
I:ns9:setting up zone: dswithdrawn.checkds
I:ns9:setting up zone: missing-dswithdrawn.checkds
I:ns9:setting up zone: bad-dswithdrawn.checkds
I:ns9:setting up zone: multiple-dswithdrawn.checkds
I:ns9:setting up zone: incomplete-dswithdrawn.checkds
I:ns9:setting up zone: bad2-dswithdrawn.checkds
I:ns5:ns5/setup.sh
I:ns2:ns2/setup.sh
I:checkds:starting servers
D:checkds:============================= test session starts ==============================
D:checkds:platform sunos5 -- Python 3.9.12, pytest-6.1.2, py-1.11.0, pluggy-0.13.1 -- /usr/bin/python3.9
D:checkds:cachedir: .pytest_cache
D:checkds:rootdir: /export/home/newman/bind9/bin/tests/system/checkds
D:checkds:collecting ... collected 2 items
D:checkds:
D:checkds:tests-checkds.py::test_checkds_dspublished PASSED [ 50%]
D:checkds:tests-checkds.py::test_checkds_dswithdrawn PASSED [100%]
D:checkds:
D:checkds:============================== 2 passed in 0.96s ===============================
I:checkds:stopping servers
I:checkds:ns2 crashed on shutdown
...
I:checkds:ns9 crashed on shutdown
I:checkds:ns2 didn't die when sent a SIGTERM
I:checkds:ns2 died before a SIGABRT was sent
I:checkds:ns6 didn't die when sent a SIGTERM
I:checkds:ns6 died before a SIGABRT was sent
I:checkds:ns9 didn't die when sent a SIGTERM
I:checkds:ns9 died before a SIGABRT was sent
I:checkds:stopping servers failed
I:checkds:Core dump(s) found: checkds/ns2/core.18972 checkds/ns6/core.19065 checkds/ns9/core.19134
D:checkds:backtrace from checkds/ns2/core.18972:
D:checkds:--------------------------------------------------------------------------------
D:checkds:Core was generated by `/export/home/newman/bind9/bin/named/.libs/named -D checkds-ns2 -X named.lock -m'.
D:checkds:Program terminated with signal SIGKILL, Killed.
D:checkds:#0 0x00007fffabbd6faa in _lwp_kill () from /lib/64/libc.so.1
D:checkds:[Current thread is 14 (Thread 2 (LWP 2))]
D:checkds:#0 0x00007fffabbd6faa in _lwp_kill () from /lib/64/libc.so.1
D:checkds:#1 0x00007fffabbcd870 in thr_kill () from /lib/64/libc.so.1
D:checkds:#2 0x00007fffabb6aefe in raise () from /lib/64/libc.so.1
D:checkds:#3 0x00007fffabb44b58 in abort () from /lib/64/libc.so.1
D:checkds:#4 0x0000000000435016 in library_fatal_error (file=0x7fffae5d25f5 "netmgr/netmgr.c", line=2847, format=<optimized out>, args=0x7fffad183738) at main.c:277
D:checkds:#5 0x00007fffae52daa2 in isc_error_fatal (file=file@entry=0x7fffae5d25f5 "netmgr/netmgr.c", line=line@entry=2847, format=format@entry=0x7fffae5d2b5c "%s failed: %s\n") at error.c:68
D:checkds:#6 0x00007fffae4e5443 in isc__nmsocket_reset (sock=sock@entry=0x7fffaca03c00) at netmgr.c:2847
D:checkds:#7 0x00007fffae4e56de in shutdown_walk_cb (handle=0x7fffaca042f8, arg=<optimized out>) at netmgr.c:2900
D:checkds:#8 0x00007fffaf4189b7 in uv_walk (loop=loop@entry=0x7fffad710010, walk_cb=walk_cb@entry=0x7fffae4e55ee <shutdown_walk_cb>, arg=arg@entry=0x0) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/uv-common.c:500
D:checkds:#9 0x00007fffae4e573b in isc__nm_async_shutdown (worker=worker@entry=0x7fffad710000, ev0=ev0@entry=0x5fffbed04d80) at netmgr.c:2918
D:checkds:#10 0x00007fffae4edb80 in process_netievent (worker=worker@entry=0x7fffad710000, ievent=ievent@entry=0x5fffbed04d80) at netmgr.c:927
D:checkds:#11 0x00007fffae4ee157 in process_queue (worker=worker@entry=0x7fffad710000, type=type@entry=NETIEVENT_NORMAL) at netmgr.c:958
D:checkds:#12 0x00007fffae4efadc in process_all_queues (worker=0x7fffad710000) at netmgr.c:711
D:checkds:#13 async_cb (handle=0x7fffad710350) at netmgr.c:741
D:checkds:#14 0x00007fffaf419666 in uv__async_io (loop=0x7fffad710010, w=<optimized out>, events=<optimized out>) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/unix/async.c:163
D:checkds:#15 0x00007fffaf42a1b4 in uv__io_poll (loop=loop@entry=0x7fffad710010, timeout=<optimized out>) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/unix/sunos.c:288
D:checkds:#16 0x00007fffaf419e33 in uv_run (loop=loop@entry=0x7fffad710010, mode=mode@entry=UV_RUN_DEFAULT) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/unix/core.c:381
D:checkds:#17 0x00007fffae4eeb08 in nm_thread (worker0=0x7fffad710000) at netmgr.c:642
D:checkds:#18 0x00007fffae5963c0 in isc__trampoline_run (arg=0x7fffad8c0080) at trampoline.c:187
D:checkds:#19 0x00007fffabbcfb8c in _thrp_setup () from /lib/64/libc.so.1
D:checkds:#20 0x00007fffabbcfea0 in ?? () from /lib/64/libc.so.1
D:checkds:#21 0x0000000000000000 in ?? ()
D:checkds:--------------------------------------------------------------------------------
D:checkds:full backtrace from checkds/ns2/core.18972 saved in checkds/ns2/core.18972-backtrace.txt
D:checkds:core dump checkds/ns2/core.18972 archived as checkds/ns2/core.18972.gz
D:checkds:backtrace from checkds/ns6/core.19065:
D:checkds:--------------------------------------------------------------------------------
D:checkds:Core was generated by `/export/home/newman/bind9/bin/named/.libs/named -D checkds-ns6 -X named.lock -m'.
D:checkds:Program terminated with signal SIGKILL, Killed.
D:checkds:#0 0x00007fffae3a6faa in _lwp_kill () from /lib/64/libc.so.1
D:checkds:[Current thread is 14 (Thread 2 (LWP 2))]
D:checkds:#0 0x00007fffae3a6faa in _lwp_kill () from /lib/64/libc.so.1
D:checkds:#1 0x00007fffae39d870 in thr_kill () from /lib/64/libc.so.1
D:checkds:#2 0x00007fffae33aefe in raise () from /lib/64/libc.so.1
D:checkds:#3 0x00007fffae314b58 in abort () from /lib/64/libc.so.1
D:checkds:#4 0x0000000000435016 in library_fatal_error (file=0x7fffae9d25f5 "netmgr/netmgr.c", line=2847, format=<optimized out>, args=0x7fffac874738) at main.c:277
D:checkds:#5 0x00007fffae92daa2 in isc_error_fatal (file=file@entry=0x7fffae9d25f5 "netmgr/netmgr.c", line=line@entry=2847, format=format@entry=0x7fffae9d2b5c "%s failed: %s\n") at error.c:68
D:checkds:#6 0x00007fffae8e5443 in isc__nmsocket_reset (sock=sock@entry=0x7fffac5a3c00) at netmgr.c:2847
D:checkds:#7 0x00007fffae8e56de in shutdown_walk_cb (handle=0x7fffac5a42f8, arg=<optimized out>) at netmgr.c:2900
D:checkds:#8 0x00007fffacbd89b7 in uv_walk (loop=loop@entry=0x7fffacad0010, walk_cb=walk_cb@entry=0x7fffae8e55ee <shutdown_walk_cb>, arg=arg@entry=0x0) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/uv-common.c:500
D:checkds:#9 0x00007fffae8e573b in isc__nm_async_shutdown (worker=worker@entry=0x7fffacad0000, ev0=ev0@entry=0x5fffbe981900) at netmgr.c:2918
D:checkds:#10 0x00007fffae8edb80 in process_netievent (worker=worker@entry=0x7fffacad0000, ievent=ievent@entry=0x5fffbe981900) at netmgr.c:927
D:checkds:#11 0x00007fffae8ee157 in process_queue (worker=worker@entry=0x7fffacad0000, type=type@entry=NETIEVENT_NORMAL) at netmgr.c:958
D:checkds:#12 0x00007fffae8efadc in process_all_queues (worker=0x7fffacad0000) at netmgr.c:711
D:checkds:#13 async_cb (handle=0x7fffacad0350) at netmgr.c:741
D:checkds:#14 0x00007fffacbd9666 in uv__async_io (loop=0x7fffacad0010, w=<optimized out>, events=<optimized out>) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/unix/async.c:163
D:checkds:#15 0x00007fffacbea1b4 in uv__io_poll (loop=loop@entry=0x7fffacad0010, timeout=<optimized out>) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/unix/sunos.c:288
D:checkds:#16 0x00007fffacbd9e33 in uv_run (loop=loop@entry=0x7fffacad0010, mode=mode@entry=UV_RUN_DEFAULT) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/unix/core.c:381
D:checkds:#17 0x00007fffae8eeb08 in nm_thread (worker0=0x7fffacad0000) at netmgr.c:642
D:checkds:#18 0x00007fffae9963c0 in isc__trampoline_run (arg=0x7fffacd10080) at trampoline.c:187
D:checkds:#19 0x00007fffae39fb8c in _thrp_setup () from /lib/64/libc.so.1
D:checkds:#20 0x00007fffae39fea0 in ?? () from /lib/64/libc.so.1
D:checkds:#21 0x0000000000000000 in ?? ()
D:checkds:--------------------------------------------------------------------------------
D:checkds:full backtrace from checkds/ns6/core.19065 saved in checkds/ns6/core.19065-backtrace.txt
D:checkds:core dump checkds/ns6/core.19065 archived as checkds/ns6/core.19065.gz
D:checkds:backtrace from checkds/ns9/core.19134:
D:checkds:--------------------------------------------------------------------------------
D:checkds:Core was generated by `/export/home/newman/bind9/bin/named/.libs/named -D checkds-ns9 -X named.lock -m'.
D:checkds:Program terminated with signal SIGKILL, Killed.
D:checkds:#0 0x00007fffae566faa in _lwp_kill () from /lib/64/libc.so.1
D:checkds:[Current thread is 22 (Thread 2 (LWP 2))]
D:checkds:#0 0x00007fffae566faa in _lwp_kill () from /lib/64/libc.so.1
D:checkds:#1 0x00007fffae55d870 in thr_kill () from /lib/64/libc.so.1
D:checkds:#2 0x00007fffae4faefe in raise () from /lib/64/libc.so.1
D:checkds:#3 0x00007fffae4d4b58 in abort () from /lib/64/libc.so.1
D:checkds:#4 0x0000000000435016 in library_fatal_error (file=0x7fffa7dd25f5 "netmgr/netmgr.c", line=2847, format=<optimized out>, args=0x7fffabb09738) at main.c:277
D:checkds:#5 0x00007fffa7d2daa2 in isc_error_fatal (file=file@entry=0x7fffa7dd25f5 "netmgr/netmgr.c", line=line@entry=2847, format=format@entry=0x7fffa7dd2b5c "%s failed: %s\n") at error.c:68
D:checkds:#6 0x00007fffa7ce5443 in isc__nmsocket_reset (sock=sock@entry=0x7fffaade3c00) at netmgr.c:2847
D:checkds:#7 0x00007fffa7ce56de in shutdown_walk_cb (handle=0x7fffaade42f8, arg=<optimized out>) at netmgr.c:2900
D:checkds:#8 0x00007fffaf0f89b7 in uv_walk (loop=loop@entry=0x7fffac150010, walk_cb=walk_cb@entry=0x7fffa7ce55ee <shutdown_walk_cb>, arg=arg@entry=0x0) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/uv-common.c:500
D:checkds:#9 0x00007fffa7ce573b in isc__nm_async_shutdown (worker=worker@entry=0x7fffac150000, ev0=ev0@entry=0x5fffbe8f3480) at netmgr.c:2918
D:checkds:#10 0x00007fffa7cedb80 in process_netievent (worker=worker@entry=0x7fffac150000, ievent=ievent@entry=0x5fffbe8f3480) at netmgr.c:927
D:checkds:#11 0x00007fffa7cee157 in process_queue (worker=worker@entry=0x7fffac150000, type=type@entry=NETIEVENT_NORMAL) at netmgr.c:958
D:checkds:#12 0x00007fffa7cefadc in process_all_queues (worker=0x7fffac150000) at netmgr.c:711
D:checkds:#13 async_cb (handle=0x7fffac150350) at netmgr.c:741
D:checkds:#14 0x00007fffaf0f9666 in uv__async_io (loop=0x7fffac150010, w=<optimized out>, events=<optimized out>) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/unix/async.c:163
D:checkds:#15 0x00007fffaf10a1b4 in uv__io_poll (loop=loop@entry=0x7fffac150010, timeout=<optimized out>) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/unix/sunos.c:288
D:checkds:#16 0x00007fffaf0f9e33 in uv_run (loop=loop@entry=0x7fffac150010, mode=mode@entry=UV_RUN_DEFAULT) at /export/home/newman/oi-userland/components/library/libuv/libuv-v1.38.0/src/unix/core.c:381
D:checkds:#17 0x00007fffa7ceeb08 in nm_thread (worker0=0x7fffac150000) at netmgr.c:642
D:checkds:#18 0x00007fffa7d963c0 in isc__trampoline_run (arg=0x7fffac2a0080) at trampoline.c:187
D:checkds:#19 0x00007fffae55fb8c in _thrp_setup () from /lib/64/libc.so.1
D:checkds:#20 0x00007fffae55fea0 in ?? () from /lib/64/libc.so.1
D:checkds:#21 0x0000000000000000 in ?? ()
D:checkds:--------------------------------------------------------------------------------
D:checkds:full backtrace from checkds/ns9/core.19134 saved in checkds/ns9/core.19134-backtrace.txt
D:checkds:core dump checkds/ns9/core.19134 archived as checkds/ns9/core.19134.gz
R:checkds:FAIL
E:checkds:2022-04-14T00:32:43+0200
FAIL: checkdscheckds-ns9-core.19134-backtrace.txt
illumos-checkds-ns9-named.run:
14-Apr-2022 00:31:35.355 netmgr/netmgr.c:2847: fatal error:
14-Apr-2022 00:31:35.355 uv_tcp_close_reset failed: invalid argument
14-Apr-2022 00:31:35.355 exiting (due to fatal error in library)