Intermittent crashes in the "tkey" system test caused by broken dns_name_t structures
The following crash occurred on OpenBSD in the daily pipeline run for
main today:
https://gitlab.isc.org/isc-projects/bind9/-/jobs/1737486
D:tkey:--------------------------------------------------------------------------------
D:tkey:Core was generated by `named'.
D:tkey:Program terminated with signal SIGABRT, Aborted.
D:tkey:#0 thrkill () at /tmp/-:3
D:tkey:[Current thread is 1 (process 151515)]
D:tkey:#0 thrkill () at /tmp/-:3
D:tkey:#1 0x00000b2a71112e0e in _libc_abort () at /usr/src/lib/libc/stdlib/abort.c:51
D:tkey:#2 0x00000b27e595d563 in assertion_failed (file=<optimized out>, line=<optimized out>, type=<optimized out>, cond=<optimized out>) at main.c:249
D:tkey:#3 0x00000b2a8312f3a0 in isc_assertion_failed (file=0x0, line=6, type=isc_assertiontype_require, cond=0xb2a7109410a <thrkill+10> "r\001\303d\211\004% ") at assertions.c:52
D:tkey:#4 0x00000b2a9f5a2a14 in dns_name_getlabelsequence (source=<optimized out>, first=<optimized out>, n=<optimized out>, target=<optimized out>) at name.c:922
D:tkey:#5 0x00000b2a9f546950 in dns_compress_add (cctx=0xb2a3d5c6f98, name=<optimized out>, prefix=<optimized out>, offset=279) at compress.c:433
D:tkey:#6 0x00000b2a9f5a47c9 in dns_name_towire2 (name=0xb2a3d5c6a90, cctx=0xb2a3d5c6f98, target=0xb2a3d5c7f30, comp_offsetp=<optimized out>) at name.c:2048
D:tkey:#7 0x00000b2a9f60cb82 in towiresorted (rdataset=0xb2aaa240a00, owner_name=<optimized out>, cctx=<optimized out>, target=0xb2a3d5c7f30, order=0xa, order_arg=0xb2a00000117, partial=<optimized out>, options=1, countp=0xb2a3d5c6f4c, state=<optimized out>) at rdataset.c:456
D:tkey:#8 0x00000b2a9f60c4b4 in dns_rdataset_towiresorted (rdataset=0x0, owner_name=0x6, cctx=0x0, target=0xb2a7109410a <thrkill+10>, order=0xb29e76ab640, order_arg=0x0, options=1, countp=0xb2a3d5c6f4c) at rdataset.c:554
D:tkey:#9 0x00000b2a9f58dfae in dns_message_rendersection (msg=<optimized out>, sectionid=<optimized out>, options=<optimized out>) at message.c:2079
D:tkey:#10 0x00000b2a8d4b881d in ns_client_send (client=0xb2a5d77e2c8) at client.c:532
D:tkey:#11 0x00000b2a8d4c7639 in query_send (client=0xb2a5d77e2c8) at query.c:576
D:tkey:#12 0x00000b2a8d4c8054 in ns_query_start (client=0xb2a5d77e2c8, handle=<optimized out>) at query.c:12014
D:tkey:#13 0x00000b2a8d4bb163 in ns__client_request (handle=0xb2a5d77e060, eresult=<optimized out>, region=<optimized out>, arg=<optimized out>) at client.c:2176
D:tkey:#14 0x00000b2a83109b47 in isc__nm_async_readcb (worker=<optimized out>, ev0=<optimized out>) at netmgr/netmgr.c:2701
D:tkey:#15 0x00000b2a83109a20 in isc__nm_readcb (sock=0xb2a5d7985f0, uvreq=0xb2ad0c0f000, eresult=0) at netmgr/netmgr.c:2676
D:tkey:#16 0x00000b2a8310fbc8 in isc__nm_tcpdns_processbuffer (sock=0xb2a5d7985f0) at netmgr/tcpdns.c:817
D:tkey:#17 0x00000b2a83108d12 in processbuffer (sock=0xb2a5d7985f0) at netmgr/netmgr.c:2252
D:tkey:#18 isc__nm_process_sock_buffer (sock=0xb2a5d7985f0) at netmgr/netmgr.c:2277
D:tkey:#19 0x00000b2a8310fd76 in isc__nm_tcpdns_read_cb (stream=<optimized out>, nread=304, buf=0xb2a3d5c9218) at netmgr/tcpdns.c:880
D:tkey:#20 0x00000b29f22923af in uv.stream_io () from /usr/local/lib/libuv.so.3.0
D:tkey:#21 0x00000b29f2298a19 in uv.io_poll () from /usr/local/lib/libuv.so.3.0
D:tkey:#22 0x00000b29f22870b8 in uv_run () from /usr/local/lib/libuv.so.3.0
D:tkey:#23 0x00000b2a83100e8b in nm_thread (worker0=0xb2aa115b0c0) at netmgr/netmgr.c:713
D:tkey:#24 0x00000b2a83153e23 in isc__trampoline_run (arg=0xb2aa1160d60) at trampoline.c:184
D:tkey:#25 0x00000b2a24a34f51 in _rthread_start (v=<optimized out>) at /usr/src/lib/librthread/rthread.c:96
D:tkey:#26 0x00000b2a710b18ea in __tfork_thread () at /usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:84
D:tkey:--------------------------------------------------------------------------------One other similar crash happened on Ubuntu Xenial in a pipeline run for
!5072 (merged), which is a not-yet-merged v9_16 backport of !5071 (merged) (which is
already merged to main):
https://gitlab.isc.org/isc-projects/bind9/-/jobs/1736455
D:tkey:--------------------------------------------------------------------------------
D:tkey:Core was generated by `/builds/isc-projects/bind9/bin/named/.libs/lt-named -D tkey-ns1 -X named.lock -'.
D:tkey:Program terminated with signal SIGABRT, Aborted.
D:tkey:#0 0x00007f7eeeb99438 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
D:tkey:[Current thread is 1 (Thread 0x7f7ee5ad9700 (LWP 12588))]
D:tkey:#0 0x00007f7eeeb99438 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
D:tkey:#1 0x00007f7eeeb9b03a in __GI_abort () at abort.c:89
D:tkey:#2 0x000000000042d655 in assertion_failed (file=<optimized out>, line=<optimized out>, type=<optimized out>, cond=<optimized out>) at ./main.c:266
D:tkey:#3 0x00007f7ef05ce3aa in isc_assertion_failed (file=file@entry=0x7f7ef105df2f "name.c", line=line@entry=1721, type=type@entry=isc_assertiontype_insist, cond=cond@entry=0x7f7ef10513dd "count <= 63") at assertions.c:46
D:tkey:#4 0x00007f7ef0f3e82d in set_offsets (name=name@entry=0x7f7ee5ad2eb0, offsets=offsets@entry=0x7f7ee5ad2d60 "", set_name=set_name@entry=0x7f7ee5ad2eb0) at name.c:1721
D:tkey:#5 0x00007f7ef0f4089e in dns_name_fromregion (name=name@entry=0x7f7ee5ad2eb0, r=r@entry=0x7f7ee5ad2e50) at name.c:1033
D:tkey:#6 0x00007f7ef0eefa77 in dns_compress_add (cctx=cctx@entry=0x7f7ee5ad3d90, name=name@entry=0x7f7ee5ad3450, prefix=<optimized out>, offset=279) at compress.c:424
D:tkey:#7 0x00007f7ef0f40461 in dns_name_towire2 (name=name@entry=0x7f7ee5ad3450, cctx=cctx@entry=0x7f7ee5ad3d90, target=target@entry=0x7f7ee5ad3cd0, comp_offsetp=comp_offsetp@entry=0x7f7ee5ad31d6) at name.c:2048
D:tkey:#8 0x00007f7ef0f42198 in dns_name_towire2 (name=name@entry=0x7f7ee5ad3450, cctx=cctx@entry=0x7f7ee5ad3d90, target=target@entry=0x7f7ee5ad3cd0, comp_offsetp=comp_offsetp@entry=0x7f7ee5ad31d6) at name.c:1952
D:tkey:#9 0x00007f7ef0fa6875 in towiresorted (rdataset=rdataset@entry=0x7f7edd05cad8, owner_name=owner_name@entry=0x7f7edd057430, cctx=<optimized out>, target=<optimized out>, order=<optimized out>, order_arg=order_arg@entry=0x7f7edd04a1f0, partial=false, options=1, countp=0x7f7ee5ad3c64, state=0x0) at rdataset.c:456
D:tkey:#10 0x00007f7ef0fa7464 in dns_rdataset_towiresorted (rdataset=rdataset@entry=0x7f7edd05cad8, owner_name=owner_name@entry=0x7f7edd057430, cctx=<optimized out>, target=<optimized out>, order=<optimized out>, order_arg=order_arg@entry=0x7f7edd04a1f0, options=1, countp=0x7f7ee5ad3c64) at rdataset.c:554
D:tkey:#11 0x00007f7ef0f2da3c in dns_message_rendersection (msg=0x7f7edd04a020, sectionid=sectionid@entry=1, options=options@entry=6) at message.c:2078
D:tkey:#12 0x00007f7ef12ca7ed in ns_client_send (client=client@entry=0x7f7ec8005588) at client.c:531
D:tkey:#13 0x00007f7ef12d6a7c in query_send (client=client@entry=0x7f7ec8005588) at query.c:568
D:tkey:#14 0x00007f7ef12e4de7 in ns_query_start (client=client@entry=0x7f7ec8005588, handle=handle@entry=0x7f7ec8005420) at query.c:11722
D:tkey:#15 0x00007f7ef12cdb02 in ns__client_request (handle=<optimized out>, eresult=<optimized out>, region=<optimized out>, arg=<optimized out>) at client.c:2163
D:tkey:#16 0x00007f7ef05ead94 in isc__nm_async_readcb (worker=worker@entry=0x0, ev0=ev0@entry=0x7f7ee5ad5980) at netmgr.c:2546
D:tkey:#17 0x00007f7ef05eaf26 in isc__nm_readcb (sock=sock@entry=0x7f7ec8000990, uvreq=uvreq@entry=0x7f7ed4015630, eresult=eresult@entry=0) at netmgr.c:2521
D:tkey:#18 0x00007f7ef05f4c96 in isc__nm_tcpdns_processbuffer (sock=sock@entry=0x7f7ec8000990) at tcpdns.c:813
D:tkey:#19 0x00007f7ef05e82fd in processbuffer (sock=0x7f7ec8000990) at netmgr.c:2150
D:tkey:#20 isc__nm_process_sock_buffer (sock=sock@entry=0x7f7ec8000990) at netmgr.c:2173
D:tkey:#21 0x00007f7ef05f4e22 in isc__nm_tcpdns_read_cb (stream=<optimized out>, nread=304, buf=0x7f7ee5ad5ab0) at tcpdns.c:876
D:tkey:#22 0x00007f7eef7a6aff in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1
D:tkey:#23 0x00007f7eef7a724c in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1
D:tkey:#24 0x00007f7eef7ac055 in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1
D:tkey:#25 0x00007f7eef79defc in uv_run () from /usr/lib/x86_64-linux-gnu/libuv.so.1
D:tkey:#26 0x00007f7ef05eba8d in nm_thread (worker0=0x24f1860) at netmgr.c:710
D:tkey:#27 0x00007f7ef0604dfb in isc__trampoline_run (arg=0x291ca80) at trampoline.c:191
D:tkey:#28 0x00007f7eef57e6ba in start_thread (arg=0x7f7ee5ad9700) at pthread_create.c:333
D:tkey:#29 0x00007f7eeec6b4dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
D:tkey:--------------------------------------------------------------------------------This strongly suggests that some issue with !5071 (merged) has been overlooked due to the intermittent nature of the problem.