Memory leak in HA scenario with backup server down
name: Memory leak in HA scenario with backup server down
about: Memory loss is created on running instances
Describe the bug
HA mode is configured with three servers (primary, secondary, backup) and is serving clients. When the backup server becomes unavailable, the primary and secondary experience a continuous memory leak which is manifested as a continuous increase in RSS memory use for the isc-kea-dhcp4-server process. The size of the memory leak is in direct correlation with the number of active clients - the larger number, the greater the memory leak. Once the backup server is deleted from the configuration or it becomes active again, there is no more memory increase, but the old memory is not freed.
To Reproduce
Steps to reproduce the behavior:
- Run KEA (DHCP4 only) in HA scenario with two load-balancing servers (primary and secondary) and a single backup server
- Start serving clients (40k in our scenario) and monitoring RSS usage for the KEA server process
- Disable backup server
- Verify that RSS usage is increasing continuously
- Enable backup server
- Verify that RSS usage is stable
Expected behavior
The servers should not have any memory leaks.
Environment:
- Kea version: 1.8.2, 2.0.2
- OS: Ubuntu 18.04
- Memfile
- libdhcp_lease_cmds, libdhcp_stat_cmds, libdhcp_ha
Additional Information
{
"Dhcp4": {
"dhcp-queue-control": {
"enable-queue": true,
"queue-type": "kea-ring4",
"capacity": 256
},
"interfaces-config": {
"interfaces": [
"eth1"
],
"dhcp-socket-type": "udp"
},
"control-socket": {
"socket-type": "unix",
"socket-name": "/tmp/kea-dhcp4-ctrl.sock"
},
"lease-database": {
"type": "memfile",
"persist": true,
"name": "/var/lib/kea/dhcp4.leases",
"lfc-interval": 3600,
"port": 0
},
"expired-leases-processing": {
"reclaim-timer-wait-time": 10,
"flush-reclaimed-timer-wait-time": 25,
"hold-reclaimed-time": 3600,
"max-reclaim-leases": 100,
"max-reclaim-time": 250,
"unwarned-reclaim-cycles": 5
},
"renew-timer": 60,
"rebind-timer": 100,
"valid-lifetime": 120,
"option-data": [],
"hooks-libraries": [
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so",
"parameters": {}
},
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_stat_cmds.so"
},
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [
{
"this-server-name": "server3",
"mode": "load-balancing",
"heartbeat-delay": 3000,
"max-response-delay": 7000,
"max-ack-delay": 7000,
"max-unacked-clients": 20,
"peers": [
{
"name": "server2",
"url": "http://<XXX>:8080/",
"role": "secondary",
"auto-failover": true
},
{
"name": "server1",
"url": "http://<YYY>:8080/",
"role": "primary",
"auto-failover": true
},
{
"name": "server3",
"url": "http://<ZZZ>:8080/",
"role": "backup",
"auto-failover": true
}
]
}
]
}
}
],
"option-def": [
{
"name": "classless-static-route",
"code": 121,
"space": "dhcp4",
"type": "record",
"array": true,
"record-types": "uint8, uint8"
}
],
"client-classes": [
// anonymized
],
"subnet4": [
// anonymized
],
"reservations": [],
"loggers": [
{
"name": "kea-dhcp4",
"output_options": [
{
"output": "syslog"
}
],
"severity": "error",
"debuglevel": 0
}
]
}
}
Contacting you
Email/Github, telephone is available after contact