TSAN: TLS and DoH shutdown sequence
TSAN reports a warning where the underlying socket is not yet destroyed after isc_nm_stoplistening(), but the top socket already is.
The problem is that isc_nm_stoplistening() pretends to be synchronous, but it really isn't and we need to add per-thread child socket for TLS(stream) and DoH code, so we don't destroy the top socket before the underlying socket gets destroyed. Alternatively, the isc_nm_stoplistening() needs to take a callback and be converted to asynchronous code.
WARNING: ThreadSanitizer: data race (pid=7728)
Write of size 8 at 0x7b080002cbc0 by thread T11:
#0 free <null> (libtsan.so.0+0x37a28)
#1 OPENSSL_sk_free <null> (libcrypto.so.1.1+0x18156a)
#2 isc__nm_tls_cleanup_data netmgr/tlsstream.c:1122 (libisc-9.19.5-dev.so+0x8bd0d)
#3 nmsocket_cleanup netmgr/netmgr.c:729 (libisc-9.19.5-dev.so+0x252fd)
#4 nmsocket_maybe_destroy netmgr/netmgr.c:799 (libisc-9.19.5-dev.so+0x2556b)
#5 isc___nmsocket_prep_destroy netmgr/netmgr.c:861 (libisc-9.19.5-dev.so+0x256de)
#6 isc___nmsocket_detach netmgr/netmgr.c:888 (libisc-9.19.5-dev.so+0x25818)
#7 isc__nm_tls_cleanup_data netmgr/tlsstream.c:1141 (libisc-9.19.5-dev.so+0x8bb84)
#8 nmsocket_cleanup netmgr/netmgr.c:729 (libisc-9.19.5-dev.so+0x252fd)
#9 nmsocket_maybe_destroy netmgr/netmgr.c:799 (libisc-9.19.5-dev.so+0x2556b)
#10 isc___nmsocket_prep_destroy netmgr/netmgr.c:861 (libisc-9.19.5-dev.so+0x256de)
#11 tcp_close_sock netmgr/tcp.c:1209 (libisc-9.19.5-dev.so+0x29a8e)
#12 tcp_close_cb netmgr/tcp.c:1217 (libisc-9.19.5-dev.so+0x29b43)
#13 uv__finish_close /usr/src/libuv-v1.44.1/src/unix/core.c:308 (libuv.so.1+0x119b9)
#14 isc__trampoline_run /builds/isc-projects/bind9/lib/isc/trampoline.c:189 (libisc-9.19.5-dev.so+0x79d29)
Previous read of size 8 at 0x7b080002cbc0 by thread T16:
#0 memcpy <null> (libtsan.so.0+0x5da1e)
#1 OPENSSL_sk_dup <null> (libcrypto.so.1.1+0x183231)
#2 tls_readcb netmgr/tlsstream.c:606 (libisc-9.19.5-dev.so+0x89bbe)
#3 isc__nm_async_readcb netmgr/netmgr.c:2217 (libisc-9.19.5-dev.so+0x26e6b)
#4 isc__nm_readcb netmgr/netmgr.c:2190 (libisc-9.19.5-dev.so+0x2704f)
#5 isc__nm_tcp_read_cb netmgr/tcp.c:897 (libisc-9.19.5-dev.so+0x2bb8c)
#6 uv__read /usr/src/libuv-v1.44.1/src/unix/stream.c:1247 (libuv.so.1+0x22d21)
#7 isc__trampoline_run /builds/isc-projects/bind9/lib/isc/trampoline.c:189 (libisc-9.19.5-dev.so+0x79d29)
Thread T11 'isc-loop-0007' (tid=8083, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.0+0x5bef5)
#1 isc_thread_create /builds/isc-projects/bind9/lib/isc/thread.c:70 (libisc-9.19.5-dev.so+0x72017)
#2 isc_loopmgr_run /builds/isc-projects/bind9/lib/isc/loop.c:467 (libisc-9.19.5-dev.so+0x596b1)
#3 run_test_doh_recv_send_POST_TLS /builds/isc-projects/bind9/tests/isc/doh_test.c:1035 (doh_test+0x40d088)
#4 cmocka_run_one_test_or_fixture <null> (libcmocka.so.0+0x5cb0)
#5 __libc_start_call_main <null> (libc.so.6+0x4043f)
Thread T16 'isc-loop-0012' (tid=8088, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.0+0x5bef5)
#1 isc_thread_create /builds/isc-projects/bind9/lib/isc/thread.c:70 (libisc-9.19.5-dev.so+0x72017)
#2 isc_loopmgr_run /builds/isc-projects/bind9/lib/isc/loop.c:467 (libisc-9.19.5-dev.so+0x596b1)
#3 run_test_doh_recv_send_POST_TLS /builds/isc-projects/bind9/tests/isc/doh_test.c:1035 (doh_test+0x40d088)
#4 cmocka_run_one_test_or_fixture <null> (libcmocka.so.0+0x5cb0)
#5 __libc_start_call_main <null> (libc.so.6+0x4043f)
SUMMARY: ThreadSanitizer: data race (/lib64/libtsan.so.0+0x37a28) in freeand
WARNING: ThreadSanitizer: data race (pid=7728)
Write of size 8 at 0x7b04000055e0 by thread T11:
#0 free <null> (libtsan.so.0+0x37a28)
#1 SSL_SESSION_free <null> (libssl.so.1.1+0x3fa88)
#2 isc__nm_tls_cleanup_data netmgr/tlsstream.c:1122 (libisc-9.19.5-dev.so+0x8bd0d)
#3 nmsocket_cleanup netmgr/netmgr.c:729 (libisc-9.19.5-dev.so+0x252fd)
#4 nmsocket_maybe_destroy netmgr/netmgr.c:799 (libisc-9.19.5-dev.so+0x2556b)
#5 isc___nmsocket_prep_destroy netmgr/netmgr.c:861 (libisc-9.19.5-dev.so+0x256de)
#6 isc___nmsocket_detach netmgr/netmgr.c:888 (libisc-9.19.5-dev.so+0x25818)
#7 isc__nm_tls_cleanup_data netmgr/tlsstream.c:1141 (libisc-9.19.5-dev.so+0x8bb84)
#8 nmsocket_cleanup netmgr/netmgr.c:729 (libisc-9.19.5-dev.so+0x252fd)
#9 nmsocket_maybe_destroy netmgr/netmgr.c:799 (libisc-9.19.5-dev.so+0x2556b)
#10 isc___nmsocket_prep_destroy netmgr/netmgr.c:861 (libisc-9.19.5-dev.so+0x256de)
#11 tcp_close_sock netmgr/tcp.c:1209 (libisc-9.19.5-dev.so+0x29a8e)
#12 tcp_close_cb netmgr/tcp.c:1217 (libisc-9.19.5-dev.so+0x29b43)
#13 uv__finish_close /usr/src/libuv-v1.44.1/src/unix/core.c:308 (libuv.so.1+0x119b9)
#14 isc__trampoline_run /builds/isc-projects/bind9/lib/isc/trampoline.c:189 (libisc-9.19.5-dev.so+0x79d29)
Previous read of size 1 at 0x7b04000055e1 by thread T16:
#0 memcpy <null> (libtsan.so.0+0x5da1e)
#1 ssl_session_dup <null> (libssl.so.1.1+0x3fd8f)
#2 tls_readcb netmgr/tlsstream.c:606 (libisc-9.19.5-dev.so+0x89bbe)
#3 isc__nm_async_readcb netmgr/netmgr.c:2217 (libisc-9.19.5-dev.so+0x26e6b)
#4 isc__nm_readcb netmgr/netmgr.c:2190 (libisc-9.19.5-dev.so+0x2704f)
#5 isc__nm_tcp_read_cb netmgr/tcp.c:897 (libisc-9.19.5-dev.so+0x2bb8c)
#6 uv__read /usr/src/libuv-v1.44.1/src/unix/stream.c:1247 (libuv.so.1+0x22d21)
#7 isc__trampoline_run /builds/isc-projects/bind9/lib/isc/trampoline.c:189 (libisc-9.19.5-dev.so+0x79d29)
Thread T11 'isc-loop-0007' (tid=8083, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.0+0x5bef5)
#1 isc_thread_create /builds/isc-projects/bind9/lib/isc/thread.c:70 (libisc-9.19.5-dev.so+0x72017)
#2 isc_loopmgr_run /builds/isc-projects/bind9/lib/isc/loop.c:467 (libisc-9.19.5-dev.so+0x596b1)
#3 run_test_doh_recv_send_POST_TLS /builds/isc-projects/bind9/tests/isc/doh_test.c:1035 (doh_test+0x40d088)
#4 cmocka_run_one_test_or_fixture <null> (libcmocka.so.0+0x5cb0)
#5 __libc_start_call_main <null> (libc.so.6+0x4043f)
Thread T16 'isc-loop-0012' (tid=8088, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.0+0x5bef5)
#1 isc_thread_create /builds/isc-projects/bind9/lib/isc/thread.c:70 (libisc-9.19.5-dev.so+0x72017)
#2 isc_loopmgr_run /builds/isc-projects/bind9/lib/isc/loop.c:467 (libisc-9.19.5-dev.so+0x596b1)
#3 run_test_doh_recv_send_POST_TLS /builds/isc-projects/bind9/tests/isc/doh_test.c:1035 (doh_test+0x40d088)
#4 cmocka_run_one_test_or_fixture <null> (libcmocka.so.0+0x5cb0)
#5 __libc_start_call_main <null> (libc.so.6+0x4043f)
SUMMARY: ThreadSanitizer: data race (/lib64/libtsan.so.0+0x37a28) in freeThis issue also affects 9.18, but is more prominent in 9.19 loopmgr code.