[9.16] correctly limit hash resize to RBTDB_GLUE_TABLE_MAX_BITS
Use < instead of <= when testing the new new hash bits size, otherwise it can exceed the limit.
(cherry picked from commit 8f738144)
Backport of MR !8465 (merged)
Edited by Evan Hunt