GitLab

Assertion failure: rdataset.c:527: INSIST(rrbuffer.used < 65536) failed, back trace

This one was reported in https://support.isc.org/Ticket/Display.html?id=16195 .

The problem is in lib/ns/include/ns/client.h:

#define NS_CLIENT_TCP_BUFFER_SIZE  (65535 + 2)

It's a leftover from 'old' socket code where the TCP buffer for rendering included the 2 bytes length in front. With netmgr it's the buffer we do rendering to, and we might overflow it.

There are two 'modes' of this bug - the first one is when we don't render partial rdatasets in ANSWER section- and that's what happening when we have EDNS0 enabled (msg->reserved != 0). In this mode we can render a 65536 or 65537-byte-long message which is then sent over TCP. The 'length' is still set using htons, so the 2-byte length is then either [0 0] or [0 1] - we send a malformed packet.

The second mode is worse - with EDNS0 disabled msg->reserved == 0, and a carefully crafted zone we can go over 65535-bytes limit, then go over 65537-limit with triggers ISC_R_NOSPACE on the buffer, and then we check that the buffer 'used' was below 65536 bytes - which is untrue, triggering an assertion:

rollback:
        if (partial && result == ISC_R_NOSPACE) {
                INSIST(rrbuffer.used < 65536);

I wasn't able to trigger the issue with recursion (as we'd have to have a rdataset that's > 65535 bytes long in cache, and there's no way to receive that).

The fix is trivial (remove '+2' from the aforementioned line).