GitLab

filter-aaaa: Purge memory pool upon plugin destruction

Closes #1041

See merge request !1957

5238.	[bug]		filter-aaaa: named crashed upon shutdown if it was in
			the process of recursing for A RRsets. [GL #1040]

The typical sequence of events for AAAA queries which trigger recursion
for an A RRset at the same name is as follows:

 1. Original query context is created.
 2. An AAAA RRset is found in cache.
 3. Client-specific data is allocated from the filter-aaaa memory pool.
 4. Recursion is triggered for an A RRset.
 5. Original query context is torn down.

 6. Recursion for an A RRset completes.
 7. A second query context is created.
 8. Client-specific data is retrieved from the filter-aaaa memory pool.
 9. The response to be sent is processed according to configuration.
10. The response is sent.
11. Client-specific data is returned to the filter-aaaa memory pool.
12. The second query context is torn down.

However, steps 6-12 are not executed if recursion for an A RRset is
canceled.  Thus, if named is in the process of recursing for A RRsets
when a shutdown is requested, the filter-aaaa memory pool will have
outstanding allocations which will never get released.  This in turn
leads to a crash since every memory pool must not have any outstanding
allocations by the time isc_mempool_destroy() is called.

Fix by creating a stub query context whenever fetch_callback() is called,
including cancellation events. When the qctx is destroyed, it will ensure
the client is detached and the plugin memory is freed.

Resolve "Add RFC8906 to list of RFCs in doc/arm/general.rst"

Closes #2184

See merge request !4180

Resolve "[Support#12071] [RT#46548] Output stale/expired data with 'rndc dumpdb'"

Closes #1870

See merge request !4088

When calling 'rndc dumpdb -expired', also print when the RRset expired.

This is a new features so it requires a CHANGE and release notes entry.

An ancient RRset is one still in the cache but expired, and awaiting
cleanup.

By changing the check in 'rdatasetiter_first' and 'rdatasetiter_next'
from "now > header->rdh_ttl" to "now - RBDTB_VIRTUAL > header->rdh_ttl"
we include expired rdataset entries so that they can be used for
"rndc dumpdb -expired".

This test makes sure that expired records are dumped with rndc's
'dumpdb' command if the '-expired' flag is used.

Minor changes are:
- Replace the "$RNDCCMD dumpdb" logic with "rndc_dumpdb" from
  conf.sh.common (it does the same thing).
- Update a comment to match the grep calls below it (comment said the
  rest should be expired, while the grep calls indicate that they
  are still in the cache, the comment now explains why).

This flag is the same as -cache, but will use a different style format
that will also print expired entries (awaiting cleanup) from the cache.

Resolve "ThreadSanitizer: data race in epoll_ctl"

Closes #2162

See merge request !4171

    WARNING: ThreadSanitizer: data race
    Read of size 8 at 0x000000000001 by thread T1 (mutexes: write M1):
    #0 epoll_ctl <null>
    #1 watch_fd lib/isc/unix/socket.c:704:8
    #2 wakeup_socket lib/isc/unix/socket.c:897:11
    #3 process_ctlfd lib/isc/unix/socket.c:3362:3
    #4 process_fds lib/isc/unix/socket.c:3275:10
    #5 netthread lib/isc/unix/socket.c:3516:10

    Previous write of size 8 at 0x000000000001 by thread T2 (mutexes: write M2):
    #0 connect <null>
    #1 isc_socket_connect lib/isc/unix/socket.c:4737:7
    #2 resquery_send lib/dns/resolver.c:2892:13
    #3 fctx_query lib/dns/resolver.c:2202:12
    #4 fctx_try lib/dns/resolver.c:4300:11
    #5 resquery_connected lib/dns/resolver.c:3130:4
    #6 dispatch lib/isc/task.c:1152:7
    #7 run lib/isc/task.c:1344:2

    Location is file descriptor 513 created by thread T2 at:
    #0 connect <null>
    #1 isc_socket_connect lib/isc/unix/socket.c:4737:7
    #2 resquery_send lib/dns/resolver.c:2892:13
    #3 fctx_query lib/dns/resolver.c:2202:12
    #4 fctx_try lib/dns/resolver.c:4300:11
    #5 resquery_connected lib/dns/resolver.c:3130:4
    #6 dispatch lib/isc/task.c:1152:7
    #7 run lib/isc/task.c:1344:2

    Mutex M1 (0x000000000016) created at:
    #0 pthread_mutex_init <null>
    #1 isc__mutex_init lib/isc/pthreads/mutex.c:288:8
    #2 setup_thread lib/isc/unix/socket.c:3584:3
    #3 isc_socketmgr_create2 lib/isc/unix/socket.c:3825:3
    #4 create_managers bin/named/main.c:932:11
    #5 setup bin/named/main.c:1223:11
    #6 main bin/named/main.c:1523:2

    Mutex M2 is already destroyed.

    Thread T1 'isc-socket-1' (running) created by main thread at:
    #0 pthread_create <null>
    #1 isc_thread_create lib/isc/pthreads/thread.c:73:8
    #2 isc_socketmgr_create2 lib/isc/unix/socket.c:3826:3
    #3 create_managers bin/named/main.c:932:11
    #4 setup bin/named/main.c:1223:11
    #5 main bin/named/main.c:1523:2

    Thread T2 (running) created by main thread at:
    #0 pthread_create <null>
    #1 isc_thread_create lib/isc/pthreads/thread.c:73:8
    #2 isc_taskmgr_create lib/isc/task.c:1434:3
    #3 create_managers bin/named/main.c:915:11
    #4 setup bin/named/main.c:1223:11
    #5 main bin/named/main.c:1523:2

    SUMMARY: ThreadSanitizer: data race in epoll_ctl

Merge branch '2156-threadsanitizer-lock-order-inversion-potential-deadlock-in-pthread_mutex_lock' into 'main'

Resolve "ThreadSanitizer: lock-order-inversion (potential deadlock) in pthread_mutex_lock"

Closes #2156

See merge request !4150

    WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock)
    Cycle in lock order graph: M1 (0x000000000000) => M2 (0x000000000000) => M1

    Mutex M2 acquired here while holding mutex M1 in thread T1:
    #0 pthread_mutex_lock <null>
    #1 dns_view_findzonecut lib/dns/view.c:1310:2
    #2 fctx_create lib/dns/resolver.c:5070:13
    #3 dns_resolver_createfetch lib/dns/resolver.c:10813:12
    #4 dns_resolver_prime lib/dns/resolver.c:10442:12
    #5 dns_view_find lib/dns/view.c:1176:4
    #6 dbfind_name lib/dns/adb.c:3833:11
    #7 dns_adb_createfind lib/dns/adb.c:3155:12
    #8 findname lib/dns/resolver.c:3497:11
    #9 fctx_getaddresses lib/dns/resolver.c:3808:3
    #10 fctx_try lib/dns/resolver.c:4197:12
    #11 fctx_start lib/dns/resolver.c:4824:4
    #12 dispatch lib/isc/task.c:1152:7
    #13 run lib/isc/task.c:1344:2

    Mutex M1 previously acquired by the same thread here:
    #0 pthread_mutex_lock <null>
    #1 dns_resolver_createfetch lib/dns/resolver.c:10767:2
    #2 dns_resolver_prime lib/dns/resolver.c:10442:12
    #3 dns_view_find lib/dns/view.c:1176:4
    #4 dbfind_name lib/dns/adb.c:3833:11
    #5 dns_adb_createfind lib/dns/adb.c:3155:12
    #6 findname lib/dns/resolver.c:3497:11
    #7 fctx_getaddresses lib/dns/resolver.c:3808:3
    #8 fctx_try lib/dns/resolver.c:4197:12
    #9 fctx_start lib/dns/resolver.c:4824:4
    #10 dispatch lib/isc/task.c:1152:7
    #11 run lib/isc/task.c:1344:2

    Mutex M1 acquired here while holding mutex M2 in thread T1:
    #0 pthread_mutex_lock <null>
    #1 dns_resolver_shutdown lib/dns/resolver.c:10530:4
    #2 view_flushanddetach lib/dns/view.c:632:4
    #3 dns_view_detach lib/dns/view.c:689:2
    #4 qctx_destroy lib/ns/query.c:5152:2
    #5 fetch_callback lib/ns/query.c:5749:3
    #6 dispatch lib/isc/task.c:1152:7
    #7 run lib/isc/task.c:1344:2

    Mutex M2 previously acquired by the same thread here:
    #0 pthread_mutex_lock <null>
    #1 view_flushanddetach lib/dns/view.c:630:3
    #2 dns_view_detach lib/dns/view.c:689:2
    #3 qctx_destroy lib/ns/query.c:5152:2
    #4 fetch_callback lib/ns/query.c:5749:3
    #5 dispatch lib/isc/task.c:1152:7
    #6 run lib/isc/task.c:1344:2

    Thread T1 (running) created by main thread at:
    #0 pthread_create <null>
    #1 isc_thread_create lib/isc/pthreads/thread.c:73:8
    #2 isc_taskmgr_create lib/isc/task.c:1434:3
    #3 create_managers bin/named/main.c:915:11
    #4 setup bin/named/main.c:1223:11
    #5 main bin/named/main.c:1523:2

    SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) in pthread_mutex_lock

Resolve "kind of use-after-free condition in SIG(0) signing"

Closes #2109

See merge request !4168

The message buffer passed to ns__client_request is only valid for
the life of the the ns__client_request call.  Save a copy of it
when we recurse or process a update as ns__client_request will
return before those operations complete.

Merge branch '2156-threadsanitizer-lock-order-inversion-potential-deadlock-in-pthread_mutex_lock-2' into 'main'

Resolve "ThreadSanitizer: lock-order-inversion (potential deadlock) in pthread_mutex_lock"

Closes #2156

See merge request !4157

The dotat() function has been changed to send the TAT
query asynchronously, so there's no lock order loop
because we initialize the data first and then we schedule
the TAT send to happen asynchronously.

This breaks following lock-order loops:

zone->lock (dns_zone_setviewcommit) while holding view->lock
(dns_view_setviewcommit)

keytable->lock (dns_keytable_find) while holding zone->lock
(zone_asyncload)

view->lock (dns_view_findzonecut) while holding keytable->lock
(dns_keytable_forall)

Merge branch '2157-threadsanitizer-lock-order-inversion-potential-deadlock-in-pthread_rwlock_wrlock' into 'main'

Resolve "ThreadSanitizer: lock-order-inversion (potential deadlock) in pthread_rwlock_wrlock"

Closes #2157

See merge request !4158

    WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock)
    Cycle in lock order graph: M1 (0x000000000001) => M2 (0x000000000002) => M1

    Mutex M2 acquired here while holding mutex M1 in thread T1:
    #0 pthread_rwlock_wrlock <null>
    #1 isc_rwlock_lock lib/isc/rwlock.c:52:4
    #2 zone_postload lib/dns/zone.c:5101:2
    #3 receive_secure_db lib/dns/zone.c:16206:11
    #4 dispatch lib/isc/task.c:1152:7
    #5 run lib/isc/task.c:1344:2

    Mutex M1 previously acquired by the same thread here:
    #0 pthread_mutex_lock <null>
    #1 receive_secure_db lib/dns/zone.c:16204:2
    #2 dispatch lib/isc/task.c:1152:7
    #3 run lib/isc/task.c:1344:2

    Mutex M1 acquired here while holding mutex M2 in thread T1:
    #0 pthread_mutex_lock <null>
    #1 get_raw_serial lib/dns/zone.c:2518:2
    #2 zone_gotwritehandle lib/dns/zone.c:2559:4
    #3 dispatch lib/isc/task.c:1152:7
    #4 run lib/isc/task.c:1344:2

    Mutex M2 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock <null>
    #1 isc_rwlock_lock lib/isc/rwlock.c:48:3
    #2 zone_gotwritehandle lib/dns/zone.c:2552:2
    #3 dispatch lib/isc/task.c:1152:7
    #4 run lib/isc/task.c:1344:2

    Thread T1 (running) created by main thread at:
    #0 pthread_create <null>
    #1 isc_thread_create lib/isc/pthreads/thread.c:73:8
    #2 isc_taskmgr_create lib/isc/task.c:1434:3
    #3 create_managers bin/named/main.c:915:11
    #4 setup bin/named/main.c:1223:11
    #5 main bin/named/main.c:1523:2

    SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) in pthread_rwlock_wrlock

Resolve "double-attach when prefetching"

Closes #2144

See merge request !4124

As the query_prefetch() or query_rpzfetch() could be called during
"regular" fetch, we need to introduce separate storage for attaching
the nmhandle during prefetching the records.  The query_prefetch()
and query_rpzfetch() are guarded for re-entrance by .query.prefetch
member of ns_client_t, so we can reuse the same .prefetchhandle for
both.

Resolve "Update ARM with lost changes since the conversion to RST files"

Closes #2127

See merge request !4112

Various language specific improvements, from Suzanne Goldlust's
review.

This was originally done in commit
da0ae529 but was lost when the
documentation was converted to RST files.

This was added previously in commit
3a3f40e3 but was lost when the
documentation was converted to RST files.

The option 'dnssec-keys' was introduced in 9.15 and also renamed to
'trust-anchors'. Rename the leftover references to 'trust-anchors'.

Minimize stdout logging in pairwise testing jobs

See merge request !4159

The size of the log generated by each GitLab CI job is limited to 4 MB
by default.  While this limit is configurable, it makes little sense to
print build logs to standard output if they are being captured to files
anyway.  Limit use of "tee" in util/pairwise-testing.sh to printing the
combination of configure switches used for a given build.  This way the
job should never exceed the default 4 MB log size limit, yet it will
still indicate its progress in a concise way.

Extend the artifact list for the "pairwise" CI job

See merge request !4156

Include the log file generated by the last failed pairwise build among
the artifacts produced by the "pairwise" GitLab CI job in order to
simplify troubleshooting pairwise build failures.

Fix deprected typo

See merge request !4155

Add pairwise testing

Closes isc-private/bind-qa#21

See merge request !3784

Pairwise testing is a test case generation technique based on the
observation that most faults are caused by interactions of at most two
factors.  For BIND, its configure options can be thought of as such
factors.

Process BIND configure options into a model that is subsequently
processed by the PICT tool in order to find an effective test vector.
That test vector is then used for configuring and building BIND using
various combinations of configure options.