Issue #49 (closed) concerns the handling of packet spikes. Up to Kea 1.4, Kea handles all packets in the order that they arrive. If there is a spike in traffic, a lot of packets can arrive at once. By the time Kea gets round to handling the late arrivals, the clients have timed out and they have sent additional requests.
The solution proposed (and being implemented) is to read packets as soon as they arrive and store them in a ring buffer. If the buffer gets full, the least recently arrived packets are overwritten with new ones. That way, Kea focuses on packets for which the client has not timed out; packets for which the client has timed out are discarded.
This document describes the testing required to ensure that the feature has been correctly implemented.
System Test Outline
The main aim of the test is to ensure that the packets being discarded are the right ones. In turn, this means that we need to ensure that we overfill the buffer before Kea can process the received packets. To do this:
A simple hook is written that calls
sleepthe first time it is called. The hook immediately returns immediately for subsequent calls. This will cause Kea to delay for long enough for the buffer to fill up. Since the Kea packet processing code is single-threaded, it does not matter which hook the code is connected to, so the
pkt6_receivehooks are as good as any.
The test framework sends "n + 1" DISCOVER (SOLICIT) packets to Kea and reads the responses;
- If n <= R (where R is the size of the ring buffer), OFFER (ADVERTISE) responses should be received for all packets.
- If n > R, a response should be received for the first packet, and responses received for the last R packets, but no responses should be received for intermediate packets.
Running this with different values of n (1, n < R, n = R, n = R + 1, n > R + 1) should check that the feature behaves correctly.
System Test Refinements
The above test only checks that the DISCOVER/SOLICIT packets are discarded correctly. The feature should takes no account of the types of packet in the queue, but we should test that it discards other types of packets as well. This could be done by extending the hook so that it initiates the delay on the first receipt of a particular packet type (perhaps specified via an environment variable, set before the Kea process is started). For example, to check that REQUEST packets are discarded:
- Send "n + 1" DISCOVER packets to Kea. The hook won't activate, so we should get n + 1 OFFER packets back.
- After all the DISCOVER packets have been received, send "n + 1" REQUEST packets to Kea. The hook will sleep when the first REQUEST is received and remaining packets will/will not be dropped depending on the value of "n".
The effect of the fix will be tested in the perflab. To ensure that a backlog of packets build up, Kea will be run with a relatively slow backend (MySQl or PostgreSQL) but hammered with a very high rate of packets (the rate used for memfile without a backend).