Resizing (growing) of cache hash tables causes delays in processing of client queries

From Support ticket #16212

During investigations of intermittent 'brownouts' - periods in which named seemingly stops actioning client queries for a short period, and then resumes processing a second or two later (yes, delays of seconds not ms from this) we 'caught' one culprit red-handed in a pstack run that was automatically triggered by an 'alarm' in monitoring inbound and outbound server traffic rates.

The thread in question was holding the cache tree lock, while growing the hash table:

Thread 21 (Thread 0x7f54d8b2f700 (LWP 19115)):
#0  0x000000000052bc7b in rehash (rbt=0x7f54b8c04058, newcount=<optimized out>) at rbt.c:2376
#1  0x000000000052da99 in hash_node (name=0x7f53d9562bb0, node=0x7f541cf79538, rbt=0x7f54b8c04058) at rbt.c:2389
#2  dns_rbt_addnode (rbt=0x7f54b8c04058, name=0x7f53d9562bb0, nodep=0x7f54d8b2dd28) at rbt.c:1451
#3  0x00000000005367ef in rbt_addnode_withdata (rbtdb=0x7f54b8c03010, rbt=0x7f54b8c04058, name=<optimized out>, nodep=0x7f54d8b2dd28) at rbtdb.c:2016
#4  0x000000000053ba42 in findnodeintree (rbtdb=0x7f54b8c03010, tree=0x7f54b8c04058, name=0x7f53d9562bb0, create=true, nodep=0x7f54d8b2ed30) at rbtdb.c:3339
#5  0x00000000005babb5 in cache_name (now=1587326409, zerottl=false, name=0x7f53d9562bb0, section=1, query=0x7f54600100d0, fctx=0x7f5449e172d0) at resolver.c:5876
#6  cache_message (now=1587326409, zerottl=false, query=0x7f54600100d0, fctx=0x7f5449e172d0) at resolver.c:6336
#7  resquery_response (task=0x7f5387cbb628, event=<optimized out>) at resolver.c:9166
#8  0x000000000068a8b1 in dispatch (manager=0x7f54dedc7010) at task.c:1157
#9  run (uap=0x7f54dedc7010) at task.c:1331
#10 0x00007f54dd90cdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f54dd635ead in clone () from /lib64/libc.so.6

The other cause of similar problems is when growing the ADB tables - that one however is logged, whereas it doesn't look like 'rehash' or anything that calls it owns up (via logging) to what it is doing.

Our immediate quick-fix wish is for a solution to the delays caused by growing hash tables that is along the lines of being able to specify the starting size as named is launched. This needs to be either run-time or configurable in named.conf. (It is not helpful to make it build-time only because in many environments there will be a single build that is distributed to many servers whose needs/sizing can vary.)

It would also be really helpful if any hash table growing could be logged - to include what the size is expanding to (this will help admins to tune their servers accordingly).

====

Longer term, I understand that the wish is to replace the current and now fairly ancient hashing solution with something more modern, faster, and in particular, that doesn't need to block access when resizing - I'll leave engineering to open a new and independent ticket for that. For the here and now, we need a quicker fix, not a new development feature that can't be back-ported or easily applied.