[9.16.49] Reduce lock contention during RBTDB tree pruning
The log message for commit c3377cbf explained:
Instead of issuing a separate isc_task_send() call for every RBTDB node
that triggers tree pruning, maintain a list of nodes from which tree
pruning can be started from and only issue an isc_task_send() call if
pruning has not yet been triggered by another RBTDB node.
The extra queuing overhead eliminated by this change could be remotely
exploited to cause excessive memory use.
However, it turned out that having a single queue for the nodes to be pruned increased lock contention to a level where cleaning up nodes from the RBTDB took too long, causing the amount of memory used by the cache to grow indefinitely over time.
This commit makes the prunenodes list bucketed, adds a quantum of 10 items per prune_tree() run, and simplifies parent node cleaning in the prune_tree() logic.
Instead of juggling node locks in a cycle, only clean up the node currently being pruned and queue its parent (if it is also eligible) for pruning in the same way (by sending an event).
This simplifies the code and also spreads the pruning load across more task loop ticks, which is better for lock contention as less things run in a tight loop.
(cherry picked from commit 2df147cb)
Closes #4596 (closed)
Backport of !8765 (merged)