Kea issueshttps://gitlab.isc.org/isc-projects/kea/-/issues2023-11-10T09:50:24Zhttps://gitlab.isc.org/isc-projects/kea/-/issues/2932Kea HA issue with terminating connection2023-11-10T09:50:24ZNick HahnKea HA issue with terminating connectionWe recently migrated our DHCP setup from dhcpd to Kea. It runs on
two servers with hot standby and a memfile backend for the leases. Kea
assigns IP addresses for around 7000 pools.
Over the past few months the HA connection terminated...We recently migrated our DHCP setup from dhcpd to Kea. It runs on
two servers with hot standby and a memfile backend for the leases. Kea
assigns IP addresses for around 7000 pools.
Over the past few months the HA connection terminated in random intervals.
From looking at the logs on the passive node I can see a lot of
'ResourceBusy: IP address ... could not be updated' warnings prior to
the connection terminating. Since multithreading is enabled I suspected
this may be due to the threads encountering a resource lock on the memfile.
I suppose after the lease update fails a few times, the connection is terminated.
Is the 'ResourceBusy' warning the cause for the terminating HA connection and
is there any way to fix the underlying issue? Any ideas on the issue are greatly
appraciated.
Here are the logs from the primary server:
```
Jun 12 15:04:31 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625735366400] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1, cid=[], tid=0x0: failed to allocate an IPv4 lease in the subnet 123.123.123.123/30, subnet-id 30926, shared network (none)
Jun 12 15:04:31 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625735366400] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 address after 1 attempt(s)
Jun 12 15:04:31 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625735366400] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1], cid=[], tid=0x0: Failed to allocate an IPv4 address for client with classes: ALL, HA_primary-dhcp, VENDOR_CLASS_MSFT 5.0, UNKNOWN
Jun 12 15:04:39 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625726973696] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 lease in the subnet 123.123.123.123/30, subnet-id 30926, shared network (none)
Jun 12 15:04:39 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625726973696] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 address after 1 attempt(s)
Jun 12 15:04:39 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625726973696] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1], cid=[], tid=0x0: Failed to allocate an IPv4 address for client with classes: ALL, HA_primary-dhcp, VENDOR_CLASS_MSFT 5.0, UNKNOWN
Jun 12 15:04:45 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.ha-hooks.139625718580992] HA_LEASE_UPDATE_CONFLICT [hwtype=1], cid=[], tid=0x0: lease update to standby-dhcp (http://dhcp-2:8001/) returned conflict status code: ResourceBusy: IP address:123.123.123.123 could not be updated. (error code 4)
Jun 12 15:04:56 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625735366400] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 lease in the subnet 123.123.123.123/30, subnet-id 30926, shared network (none)
Jun 12 15:04:56 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625735366400] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 address after 1 attempt(s)
Jun 12 15:04:56 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625735366400] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1], cid=[], tid=0x0: Failed to allocate an IPv4 address for client with classes: ALL, HA_primary-dhcp, VENDOR_CLASS_MSFT 5.0, UNKNOWN
Jun 12 15:05:28 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625726973696] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 lease in the subnet 123.123.123.123/30, subnet-id 30926, shared network (none)
Jun 12 15:05:28 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625726973696] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 address after 1 attempt(s)
Jun 12 15:05:28 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625726973696] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1], cid=[], tid=0x0: Failed to allocate an IPv4 address for client with classes: ALL, HA_primary-dhcp, VENDOR_CLASS_MSFT 5.0, UNKNOWN
Jun 12 15:05:31 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625752151808] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 lease in the subnet 123.123.123.123/30, subnet-id 30926, shared network (none)
Jun 12 15:05:31 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625752151808] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 address after 1 attempt(s)
Jun 12 15:05:31 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625752151808] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1], cid=[], tid=0x0: Failed to allocate an IPv4 address for client with classes: ALL, HA_primary-dhcp, VENDOR_CLASS_MSFT 5.0, UNKNOWN
Jun 12 15:05:39 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.ha-hooks.139625701795584] HA_LEASE_UPDATE_CONFLICT [hwtype=1], cid=[], tid=0x0: lease update to standby-dhcp (http://dhcp-2:8001/) returned conflict status code: ResourceBusy: IP address:123.123.123.123 could not be updated. (error code 4)
Jun 12 15:05:39 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.ha-hooks.139625718580992] HA_LEASE_UPDATE_CONFLICT [hwtype=1], cid=[], tid=0x0: lease update to standby-dhcp (http://dhcp-2:8001/) returned conflict status code: ResourceBusy: IP address:123.123.123.123 could not be updated. (error code 4)
Jun 12 15:05:39 dhcp-1 kea-dhcp4[564812]: ERROR [kea-dhcp4.ha-hooks.139625718580992] HA_LEASE_UPDATE_REJECTS_CAUSED_TERMINATION too many rejected lease updates cause the HA service to terminate
Jun 12 15:05:39 dhcp-1 kea-dhcp4[564812]: ERROR [kea-dhcp4.ha-hooks.139625718580992] HA_TERMINATED HA service terminated due to an unrecoverable condition. Check previous error message(s), address the problem and restart!
```
Here are the logs from the standby server:
```
Mar 12 19:25:06 dhcp-2 kea-dhcp4[203037]: WARN [kea-dhcp4.lease-cmds-hooks.139670034884352] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678688706, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 2907, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
Mar 12 19:25:06 dhcp-2 kea-dhcp4[203037]: WARN [kea-dhcp4.lease-cmds-hooks.139670009706240] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678688706, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 2907, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
Mar 12 19:27:28 dhcp-2 kea-dhcp4[203037]: WARN [kea-dhcp4.lease-cmds-hooks.139670009706240] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678688848, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 3812, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
Mar 12 19:32:05 dhcp-2 kea-dhcp4[203037]: WARN [kea-dhcp4.lease-cmds-hooks.139670018098944] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678689125, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 274, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
Mar 12 19:32:34 dhcp-2 kea-dhcp4[203037]: WARN [kea-dhcp4.lease-cmds-hooks.139670009706240] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678689154, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 113, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
Mar 12 19:32:36 dhcp-2 kea-dhcp4[203037]: ERROR [kea-dhcp4.ha-hooks.139670104323840] HA_TERMINATED HA service terminated due to an unrecoverable condition. Check previous error message(s), address the problem and restart!
Mar 12 22:11:09 dhcp-2 kea-dhcp4[203037]: ERROR [kea-dhcp4.packets.139670138794688] DHCP4_BUFFER_RECEIVE_FAIL error on attempt to receive packet: Truncated DHCPv4 packet (len=0) received, at least 236 is expected.
```
The relevant config is the following on both hosts, differing only in the "this-server-name" property.
```
"hooks-libraries": [{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so",
"parameters": {}
},
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_stat_cmds.so",
"parameters": {}
},
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [{
"this-server-name": "standby-dhcp",
"mode": "hot-standby",
"heartbeat-delay": 10000,
"max-response-delay": 60000,
"max-ack-delay": 5000,
"max-unacked-clients": 5,
"peers": [{
"name": "primary-dhcp",
"url": "http://dhcp-1:8001/",
"role": "primary",
"auto-failover": true
}, {
"name": "standby-dhcp",
"url": "http://dhcp-2:8001/",
"role": "standby",
"auto-failover": true
}]
}]
}
}]
```next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/2374kea-admin lease-dump file permisions2022-11-02T15:10:41ZMarcin Godzinakea-admin lease-dump file permisionsCommand `kea-admin lease-dump` exports database as memfile file with permissions of user running the command.
If Kea is running from packages on Ubuntu it expects `/var/lib/kea/kea-leases4.csv`(or v6) file to be owned by `_kea` user/gro...Command `kea-admin lease-dump` exports database as memfile file with permissions of user running the command.
If Kea is running from packages on Ubuntu it expects `/var/lib/kea/kea-leases4.csv`(or v6) file to be owned by `_kea` user/group.
**In this case, we can not directly copy exported file without changing its permissions, or Kea will not be able to read the file and start.**
The problem was discovered during forge tests. The test Starts Kea with database and exports database to the memfile location. Then it restarts Kea using the exported file.
Test failed on Ubuntu using deb packages.
**Possible solutions:**
- We can modify the forge test to account for different permissions and modify help text of `lease-dump` command to warn user about possible permission clash.
- We can modify `lease-dump` script to detect what permissions the file should have, and apply them during export. This will solve import and export on the same system, but the problem will still exist if migrating to other systems with a different way of installing Kea.backloghttps://gitlab.isc.org/isc-projects/kea/-/issues/2305Kea should always return lease file location if memfile is used2022-11-02T15:10:41ZTomek MrugalskiKea should always return lease file location if memfile is usedThis issue came up during Stork sanity checks: https://gitlab.isc.org/isc-projects/stork/-/issues/667. The background of the problem is that Kea does not return lease file location if a default value is used. However, for an average user...This issue came up during Stork sanity checks: https://gitlab.isc.org/isc-projects/stork/-/issues/667. The background of the problem is that Kea does not return lease file location if a default value is used. However, for an average user to determine where the lease file is located, he'd have to figure out compilation options and what prefix was configured during compilation. This is not something average user can do.
The proposal is to always return the lease file location if memfile is used, even if the location is "the default" (because effectively, the default can change from one deployment to another).backloghttps://gitlab.isc.org/isc-projects/kea/-/issues/833LFC logs are not under control of config2022-11-02T15:10:20ZMichal NowikowskiLFC logs are not under control of confige.g. log pattern does not follow the one set for DHCP4:
````
2019-08-13 18:19:16.205 INFO [kea-dhcp4.dhcpsrv/25380] DHCPSRV_MEMFILE_LFC_START starting Lease File Cleanup
2019-08-13 18:19:16.206 INFO [kea-dhcp4.dhcpsrv/25380] DHCPSRV_M...e.g. log pattern does not follow the one set for DHCP4:
````
2019-08-13 18:19:16.205 INFO [kea-dhcp4.dhcpsrv/25380] DHCPSRV_MEMFILE_LFC_START starting Lease File Cleanup
2019-08-13 18:19:16.206 INFO [kea-dhcp4.dhcpsrv/25380] DHCPSRV_MEMFILE_LFC_EXECUTE executing Lease File Cleanup using: /usr/sbin/kea-lfc -4
INFO [DhcpLFC] LFC_START Starting lease file cleanup
INFO [DhcpLFC] LFC_PROCESSING Previous file: /var/lib/kea/kea-leases4.csv.2, copy file: /var/lib/kea/kea-leases4.csv.1
INFO [DhcpLFC.dhcpsrv] DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file /var/lib/kea/kea-leases4.csv.2
```backloghttps://gitlab.isc.org/isc-projects/kea/-/issues/266ISC DHCP release on roam flag2022-11-02T15:08:42ZFrancis DupontISC DHCP release on roam flag>>>
release-on-roam flag;
When enabled and the dhcpd server detects that a DHCPv6 client
(IAID+DUID) has roamed to a new network, it will release the
pre-existing leases on t...>>>
release-on-roam flag;
When enabled and the dhcpd server detects that a DHCPv6 client
(IAID+DUID) has roamed to a new network, it will release the
pre-existing leases on the old network and emit a log statement
similiar to the following:
"Client: <id> roamed to new network, releasing lease:
<address>"
The server will carry out all of the same steps that would nor-
mally occur when a client explicitly releases a lease. When
release-on-roam is disabled (the default) the server makes such
leases unavailable until they expire or the server is restarted.
Clients that need leases in multiple networks must supply a
unique IAID in each IA. This parameter may only be specified at
the global level.
>>>
Make sense for a moving client which for some reasons does not release its address at the previous location. Can be implemented in Kea if there is an easy and fast way to get leases by IAID+DUID only.backloghttps://gitlab.isc.org/isc-projects/kea/-/issues/77memfile: add a command to force writing in-memory DB to file2022-11-02T15:08:43ZGhost Usermemfile: add a command to force writing in-memory DB to filememfile keeps leases in memory and writes changes to disk. If the leasefile is lost for whatever reason, it may be useful to tell Kea to write is entire lease file to disk.memfile keeps leases in memory and writes changes to disk. If the leasefile is lost for whatever reason, it may be useful to tell Kea to write is entire lease file to disk.backlog