Updating contents of a mixed-case RPZ may cause some rules to be ignored

https://support.isc.org/Ticket/Display.html?id=17112

When an RPZ contains names spelled using varying letter case, updating it may lead to certain rules being erroneously ignored due to the way the summary RPZ database is updated when an RPZ is updated. The problem is caused by the fact that the hash function used by isc_ht_t is case-sensitive while the RBT code is case-insensitive.

Each dns_rpz_zone_t structure keeps a hash table of the names this RPZ database contains. Here is what happens when an RPZ is updated:

a new hash table is prepared for the new version of the RPZ by iterating over it; each name found is added to the summary RPZ database,
every name added to the new hash table is searched for in the old hash table; if found, it is removed from the old hash table,
the old hash table is iterated over; all names found in it are removed from the summary RPZ database (because at that point the old hash table should only contain names which are not present in the new version of the RPZ),
the new hash table replaces the old hash table.

When the new version of the RPZ is iterated over, if a given name is spelled using a different letter case than in the old version of the RPZ, the new variant will hash to a different value than the old variant, which means it will not be removed from the old hash table. When the old hash table is subsequently iterated over to remove seemingly deleted names, the old variant of the name will still be there, causing the name to be deleted from the summary RPZ database (which effectively causes a given rule to be ignored).

The issue can be triggered not just by altering the case of existing names in an RPZ, but also by adding sibling names spelled with a different letter case. This is because RBT code preserves case when node splitting occurs. The end result is that when the RPZ is iterated over, a given name may be using a different case than in the zone file (or XFR contents).

Here is an example. Consider what happens for the following RPZ:

mixed-case-rpz.	SOA	...
ISC.ORG		CNAME	.

Here is how the RBT for this zone will be built:

Now let's add a sibling record to the RPZ:

mixed-case-rpz.	SOA	...
ietf.org	CNAME	.
ISC.ORG		CNAME	.

Here is how the RBT for this zone will be built:

When the above RBT is iterated over, ISC.ORG will be "converted" to ISC.org due to node splitting. Since ISC.ORG hashes to a different value than ISC.org, the isc.org rule will be erroneously removed from the summary RPZ database even though it is still present in the latest version of the RPZ.

The problem was introduced in fa9b4de7 (big RPZ refactoring), which means 9.12.0+ is affected.