Potential improvements to post-zone-load response policy processing for 9.16 and beyond
The zone in this example is exceptionally big and takes 20-24s to complete the entire post-load processing of the response policy afet the zone itself is loaded or updated via IXFR
Demonstrably, the policy updating is iterating, but it is doing so on threads that might otherwise be handling queries, so it is the luck of the inbound query->thread allocation whether or not something gets temporarily delayed while the current iteration completes and returns to the back of the "to do" queue.
Noted from the stack traces, that there is evidence of:
a) It's trying to tweak the quantum for how long it runs (should it even be trying to do this - is this going to be effective or not on 9.16 anyway, and will it further slow things down by going through this additional processing?)
b) There are hash tables involved - we see this sort of thing on the tops of stacks - are the hash tables big enough for managing a really mammoth response policy (do we scale them accordingly)?
#0 0x00007f2394f847a5 in malloc () from /usr/lib64/libc.so.6 #1 0x00007f2396a5a2c7 in default_memalloc () from /usr/lib64/libisc-9.16.26-S1.so #2 0x00007f2396a59f21 in mem_get () from /usr/lib64/libisc-9.16.26-S1.so #3 0x00007f2396a5b917 in isc___mem_get () from /usr/lib64/libisc-9.16.26-S1.so #4 0x00007f2396a5f25d in isc__mem_get () from /usr/lib64/libisc-9.16.26-S1.so #5 0x00007f2396a4b390 in isc_ht_add () from /usr/lib64/libisc-9.16.26-S1
#0 0x00007f2396a7ec9e in isc_siphash24 () from /usr/lib64/libisc-9.16.26-S1.so #1 0x00007f2396a4ac8f in isc_hash64 () from /usr/lib64/libisc-9.16.26-S1.so #2 0x00007f2396a4b2f9 in isc_ht_add () from /usr/lib64/libisc-9.16.26-S1.so
c) Why is this a problem on 9.16 but wasn't on 9.11? Likely bad luck - some queries get allocated to a thread that happens to be handling an RPZ policy update, and have to wait for it to finish its current iteration before they can be processed?