correctly limit hash resize to RBTDB_GLUE_TABLE_MAX_BITS

Evan Hunt requested to merge 3498-glue-rehash-overflow into bind-9.18

Use < instead of <= when testing the new new hash bits size, otherwise it can exceed the limit.

Closes #3498 (closed)

Merge request reports