[9.18] fix several bugs in the RBTDB dbiterator implementation
- the DNS_DB_NSEC3ONLY and DNS_DB_NONSEC3 flags are mutually exclusive; it never made sense to set both at the same time. to enforce this, it is now a fatal error to do so. the dbiterator implementation has been cleaned up to remove code that treated the two as independent: if nonsec3 is true, we can be certain nsec3only is false, and vice versa.
- previously, iterating a database backwards omitted NSEC3 records even if DNS_DB_NONSEC3 had not been set. this has been corrected.
- when an iterator reaches the origin node of the NSEC3 tree, we need to skip over it and go to the next node in the sequence. the NSEC3 origin node is there for housekeeping purposes and never contains data.
- the dbiterator_test unit test has been expanded, several incorrect expectations have been fixed. (for example, the expected number of iterations has been reduced by one; we were previously counting the NSEC3 origin node and we should not have been doing so.)
Backport of MR !8741 (merged)
Edited by Evan Hunt