Measurable performance drop between BIND 9.10.6 and 9.10.7 [ISC-support #13354]

This was originally reported to ISC by a customer as a drop in performance between BIND 9.10 and 9.11, but later was confirmed to be seen also between 9.10.6-P1 and 9.10.7. BIND 9.11.3 and 9.11.4-P1 have similarly decreased performance. (This reporter has not tested BIND 9.11.2).

Test results can be found in https://support.isc.org/Ticket/Display.html?id=13354

All the most recent tests run with Debian 9 on the same platform: Dell PowerEdge R330 with 4 cores 3.0GHz, 16GB RAM. All 3 tests reported below are authoritative only, but the submitter says that they ran other scenarios with different HW and also recursive only and consistently observed a major drop in performance when running bind 9.11.3 versus BIND 9.10.6.

The test environment uses IXIA to simulate 100 clients generating DNS queries with a max QPS of 400K

The degradation in the most recent test set (9%) clearly starts in 9.10.7.

9.10.6-P1: 330 KQPS 9.10.7: 300 KQPS 9.11.4-P1: 300 KQPS

CPU is close to 100% in all tests. Available are top and ps outputs plus bind stats every 15 seconds during the tests.

The issue has also been seen in perflab: https://perflab.isc.org/#/config/run/56d4626b797da24b2741b1e9/

Opening this issue ticket at the request of engineering for them to home in on the underlying cause (or causes).