Anomalous IP Allocation Behavior for FLQ Allocator After Address Pool Exhaustion

name: Bug report
about: Create a report to help us improve

If you believe your bug report is a security issue (e.g. a packet that can kill the server), DO NOT REPORT IT HERE. Please use https://www.isc.org/community/report-bug/ instead or send mail to security-office(at)isc(dot)org.

Describe the bug

At first, everything went smoothly when the flq was used to allocate IP addresses for the 10.0.0.0/24 subnet (the address pool ranges from 10.0.0.31 to 10.0.0.254). But once the pool was depleted, and we waited for the valid-lifetime=60s to assign IPs to other clients requesting this subnet, we would get a number of new leases, such as 62. However, even after the hold-reclaimed-time=120s had passed, with only 62 leases left in the database, the flq wouldn't assign any more leases until either indefinitely or before the next configuration reload.

To Reproduce Steps to reproduce the behavior:

Run Kea with the following config

{
    "Dhcp4": {
        "control-socket": {
            "socket-name": "/tmp/kea4-ctrl-socket",
            "socket-type": "unix"
        },
        "expired-leases-processing": {
            "flush-reclaimed-timer-wait-time": 25,
            "hold-reclaimed-time": 120,
            "reclaim-timer-wait-time": 5,
            "max-reclaim-leases": 0,
            "max-reclaim-time": 1000
        },
        "authoritative": true,
        "interfaces-config": {
            "interfaces": [
                "lo",
                "*"
            ],
            "dhcp-socket-type": "udp"
        },
        "multi-threading": {
            "enable-multi-threading": true,
            "thread-pool-size": 0,
            "packet-queue-size": 1584
        },
        "lease-database": {
            "type": "mysql",
            "name": "kea_dhcp",
            "host": "xxx",
            "port": 3306,
            "user": "xxx",
            "password": "xxx",
            "reconnect-wait-time": 3000,
            "max-reconnect-tries": 20,
            "on-fail": "stop-retry-exit",
            "retry-on-startup": true,
            "connect-timeout": 3
        },
        "hosts-databases": [
            {
                "type": "mysql",
                "name": "kea_dhcp",
                "host": "xxx",
                "port": 3306,
                "user": "xxx",
                "password": "xxx",
                "reconnect-wait-time": 3000,
                "max-reconnect-tries": 15,
                "on-fail": "serve-retry-continue",
                "retry-on-startup": true,
                "connect-timeout": 3
            }
        ],
        "hooks-libraries": [
            {
                "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_stat_cmds.so"
            },
            {
                "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_mysql_cb.so"
            },
            {
                "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so",
                "parameters": {}
            }
        ],
        "subnet4": [
            {
                "id": 1,
                "subnet": "10.0.0.0/24",
                "pools": [
                    {
                        "pool": "10.0.0.31 - 10.0.0.254"
                    }
                ],
                "option-data": [
                    {
                        "name": "routers",
                        "data": "10.0.0.1"
                    },
                    {
                        "name": "subnet-mask",
                        "data": "255.255.255.0"
                    }
                ],
                "valid-lifetime": 60,
                "max-valid-lifetime": 600,
                "user-context":{
                    "comment": "test-test",
                    "site": "test",
                    "description": "test"
                }
            }
        ],
        "allocator": "flq",
        "loggers": [
            {
                "name": "kea-dhcp4",
                "output-options": [
                    {
                        "output": "/var/log/kea/dhcp4.log",
                        "maxver": 8,
                        "maxsize": 204800000,
                        "flush": false,
                        "pattern": "%D{%Y-%m-%d %H:%M:%S.%q} %-5p [%c] %m%n"
                    }
                ],
                "severity": "INFO"
            }
        ]
    }
}

Exhausted the 10.0.0.0/24 subnet address pool. [server-ip: 192.168.1.1]

# perfdhcp -W 3 -4 -R 1000 -t 3 -d 2 -r 30 -b mac=00:0c:01:02:03:05 192.168.1.1
Running: perfdhcp -W 3 -4 -R 1000 -t 3 -d 2 -r 30 -b mac=00:0c:01:02:03:05 192.168.1.1
Scenario: basic.
Multi-thread mode enabled.
sent: 89/85; received: 85/83; drops: 4/2; rejected: 0/0
sent: 179/115; received: 115/113; drops: 64/2; rejected: 0/0
sent: 269/173; received: 173/171; drops: 96/2; rejected: 0/0
sent: 359/224; received: 224/224; drops: 135/0; rejected: 0/0
sent: 449/224; received: 224/224; drops: 225/0; rejected: 0/0
^C***Rate statistics***
Rate: 14.5247 4-way exchanges/second, expected rate: 30

***Malformed Packets***
Malformed packets: 0
***Statistics for: DISCOVER-OFFER***
sent packets: 462
received packets: 224
drops: 238
drops ratio: 51.5152 %
orphans: 0
rejected leases: 0
non unique addresses: 0

min delay: 65.220 ms
avg delay: 72.877 ms
max delay: 128.144 ms
std deviation: 7.593 ms
collected packets: 0

***Statistics for: REQUEST-ACK***
sent packets: 224
received packets: 224
drops: 0
drops ratio: 0.000 %
orphans: 0
rejected leases: 0
non unique addresses: 0

min delay: 76.468 ms
avg delay: 85.300 ms
max delay: 145.063 ms
std deviation: 9.478 ms
collected packets: 0

Waited for the valid-lifetime=60s to assign IPs to other clients requesting this subnet, we would get a number of new leases, say 62.

# perfdhcp -W 3 -4 -R 1000 -t 3 -d 2 -r 30 -b mac=00:0c:01:02:04:05 192.168.1.1
Running: perfdhcp -W 3 -4 -R 1000 -t 3 -d 2 -r 30 -b mac=00:0c:01:02:04:05 192.168.1.1
Scenario: basic.
Multi-thread mode enabled.
sent: 89/62; received: 62/62; drops: 27/0; rejected: 0/0
sent: 179/62; received: 62/62; drops: 117/0; rejected: 0/0
sent: 269/62; received: 62/62; drops: 207/0; rejected: 0/0
sent: 359/62; received: 62/62; drops: 297/0; rejected: 0/0
^C***Rate statistics***
Rate: 5.05274 4-way exchanges/second, expected rate: 30

***Malformed Packets***
Malformed packets: 0
***Statistics for: DISCOVER-OFFER***
sent packets: 368
received packets: 62
drops: 306
drops ratio: 83.1522 %
orphans: 0
rejected leases: 0
non unique addresses: 0

min delay: 66.088 ms
avg delay: 70.761 ms
max delay: 90.039 ms
std deviation: 4.651 ms
collected packets: 0

***Statistics for: REQUEST-ACK***
sent packets: 62
received packets: 62
drops: 0
drops ratio: 0.000 %
orphans: 0
rejected leases: 0
non unique addresses: 0

min delay: 77.220 ms
avg delay: 81.845 ms
max delay: 99.832 ms
std deviation: 4.153 ms
collected packets: 0

even after the hold-reclaimed-time=120s had passed, with only 62 leases left in the database, the flq wouldn't assign any more leases until either indefinitely or before the next configuration reload.

Detail

# perfdhcp -W 3 -4 -R 1000 -t 3 -d 2 -r 30 -b mac=00:0c:01:02:03:05 192.168.1.1
Running: perfdhcp -W 3 -4 -R 1000 -t 3 -d 2 -r 30 -b mac=00:0c:01:02:03:05 192.168.1.1
Scenario: basic.
Multi-thread mode enabled.
sent: 89/0; received: 0/0; drops: 89/0; rejected: 0/0
^C***Rate statistics***
Rate: 0 4-way exchanges/second, expected rate: 30

***Malformed Packets***
Malformed packets: 0
***Statistics for: DISCOVER-OFFER***
sent packets: 154
received packets: 0
drops: 154
drops ratio: 100 %
orphans: 0
rejected leases: 0
non unique addresses: 0

min delay: inf ms
avg delay: min delay: n/a
avg delay: n/a
max delay: n/a
std deviation: n/a
collected packets: 0

***Statistics for: REQUEST-ACK***
sent packets: 0
received packets: 0
drops: 0
drops ratio: -nan %
orphans: 0
rejected leases: 0
non unique addresses: 0

min delay: inf ms
avg delay: min delay: n/a
avg delay: n/a
max delay: n/a
std deviation: n/a
collected packets: 0

# perfdhcp -W 3 -4 -R 1000 -t 3 -d 2 -r 30 -b mac=00:0c:01:02:04:05 192.168.1.1
Running: perfdhcp -W 3 -4 -R 1000 -t 3 -d 2 -r 30 -b mac=00:0c:01:02:04:05 192.168.1.1
Scenario: basic.
Multi-thread mode enabled.
sent: 89/0; received: 0/0; drops: 89/0; rejected: 0/0
sent: 179/0; received: 0/0; drops: 179/0; rejected: 0/0
^C***Rate statistics***
Rate: 0 4-way exchanges/second, expected rate: 30

***Malformed Packets***
Malformed packets: 0
***Statistics for: DISCOVER-OFFER***
sent packets: 195
received packets: 0
drops: 195
drops ratio: 100 %
orphans: 0
rejected leases: 0
non unique addresses: 0

min delay: inf ms
avg delay: min delay: n/a
avg delay: n/a
max delay: n/a
std deviation: n/a
collected packets: 0

***Statistics for: REQUEST-ACK***
sent packets: 0
received packets: 0
drops: 0
drops ratio: -nan %
orphans: 0
rejected leases: 0
non unique addresses: 0

min delay: inf ms
avg delay: min delay: n/a
avg delay: n/a
max delay: n/a
std deviation: n/a
collected packets: 0

# perfdhcp -W 3 -4 -R 1000 -t 3 -d 2 -r 30 -b mac=00:0c:01:02:02:05 192.168.1.1
Running: perfdhcp -W 3 -4 -R 1000 -t 3 -d 2 -r 30 -b mac=00:0c:01:02:02:05 192.168.1.1
Scenario: basic.
Multi-thread mode enabled.
sent: 89/0; received: 0/0; drops: 89/0; rejected: 0/0
^C***Rate statistics***
Rate: 0 4-way exchanges/second, expected rate: 30

***Malformed Packets***
Malformed packets: 0
***Statistics for: DISCOVER-OFFER***
sent packets: 114
received packets: 0
drops: 114
drops ratio: 100 %
orphans: 0
rejected leases: 0
non unique addresses: 0

min delay: inf ms
avg delay: min delay: n/a
avg delay: n/a
max delay: n/a
std deviation: n/a
collected packets: 0

***Statistics for: REQUEST-ACK***
sent packets: 0
received packets: 0
drops: 0
drops ratio: -nan %
orphans: 0
rejected leases: 0
non unique addresses: 0

min delay: inf ms
avg delay: min delay: n/a
avg delay: n/a
max delay: n/a
std deviation: n/a
collected packets: 0

Expected behavior

Normal IP Allocation Behavior for FLQ Allocator After Address Pool Exhaustion

Environment:

Two or more KEA servers are sharing the same lease database.

Kea version:

# kea-dhcp4 -V
2.6.0 (isc20240525133310 deb)
premium: yes (isc20240525133310 deb)
linked with:
- log4cplus 2.0.5
- OpenSSL 3.0.2 15 Mar 2022
backends:
- MySQL backend 22.0, library 8.0.39
- PostgreSQL backend 22.0, library 140012
- Memfile backend 3.0

# cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Contacting you

q981331502@163.com

Edited Sep 14, 2024 by a76yyyy

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information