Skip to content

ha-maintenance-cancel does not cancel maintenance if DHCP traffic is present

ha-maintenance-cancel seems to only cancel in-maintenance (or partner-in-maintenance) state if no traffic is arriving at the servers for some period of time.

I started Kea with simple configurations: kea-dhcp4-ha-primary.json kea-dhcp4-ha-standby.json (NOTE: The configuration was slightly different for 2.4.1 due to file locations.)

I ran the servers and waited for them to enter the hot-standby state. Then I sent some DHCP traffic with perfdhcp: perfdhcp -4 -r 1 -R 18 -p 86400 -l enp0s8

I sent ha-maintenance-start to primary: echo '{"command":"ha-maintenance-start"}' | sudo socat UNIX:/tmp/socket4 -,ignoreeof | jq

Logs from primary:

2024-11-04 08:39:47.737 INFO  [kea-dhcp4.ha-hooks/120617.140638165357632] HA_STATE_TRANSITION primary: server transitions from READY to PARTNER-IN-MAINTENANCE state, partner state is WAITING
2024-11-04 08:39:47.737 INFO  [kea-dhcp4.ha-hooks/120617.140638165357632] HA_LEASE_UPDATES_ENABLED primary: lease updates will be sent to the partner while in PARTNER-IN-MAINTENANCE state
2024-11-04 08:39:47.737 INFO  [kea-dhcp4.ha-hooks/120617.140638165357632] HA_LOCAL_DHCP_ENABLE local DHCP service is enabled while the primary is in the PARTNER-IN-MAINTENANCE state
2024-11-04 08:39:47.737 INFO  [kea-dhcp4.ha-hooks/120617.140638165357632] HA_MAINTENANCE_STARTED primary: the server is now in the partner-in-maintenance state and the partner is in-maintenance state

Logs from standby:

2024-11-04 08:39:42.505 INFO  [kea-dhcp4.ha-hooks/108440.139816923600576] HA_STATE_TRANSITION standby: server transitions from WAITING to SYNCING state, partner state is READY
2024-11-04 08:39:42.505 INFO  [kea-dhcp4.ha-hooks/108440.139816923600576] HA_LEASE_UPDATES_DISABLED standby: lease updates will not be sent to the partner while in SYNCING state
2024-11-04 08:39:42.505 INFO  [kea-dhcp4.ha-hooks/108440.139816923600576] HA_SYNC_START standby: starting lease database synchronization with primary (http://172.28.0.253:8000/)
2024-11-04 08:39:42.507 INFO  [kea-dhcp4.ha-hooks/108440.139816923600576] HA_LEASES_SYNC_LEASE_PAGE_RECEIVED standby: received 18 leases from primary (http://172.28.0.253:8000/)
2024-11-04 08:39:42.508 INFO  [kea-dhcp4.ha-hooks/108440.139816923600576] HA_LEASES_SYNC_APPLIED_LEASES standby: applied 0 leases received from the partner in the local lease database
2024-11-04 08:39:42.508 INFO  [kea-dhcp4.ha-hooks/108440.139816923600576] HA_SYNC_SUCCESSFUL standby: lease database synchronization with primary (http://172.28.0.253:8000/) completed successfully in 2.661 ms
2024-11-04 08:39:42.508 INFO  [kea-dhcp4.ha-hooks/108440.139816923600576] HA_STATE_TRANSITION standby: server transitions from SYNCING to READY state, partner state is READY
2024-11-04 08:39:42.508 INFO  [kea-dhcp4.ha-hooks/108440.139816923600576] HA_LEASE_UPDATES_DISABLED standby: lease updates will not be sent to the partner while in READY state
2024-11-04 08:39:47.687 INFO  [kea-dhcp4.commands/108440.139816890029760] COMMAND_RECEIVED Received command 'ha-maintenance-notify'
2024-11-04 08:39:47.687 INFO  [kea-dhcp4.ha-hooks/108440.139816890029760] HA_STATE_TRANSITION standby: server transitions from READY to IN-MAINTENANCE state, partner state is READY
2024-11-04 08:39:47.687 INFO  [kea-dhcp4.ha-hooks/108440.139816890029760] HA_LEASE_UPDATES_DISABLED standby: lease updates will not be sent to the partner while in IN-MAINTENANCE state
2024-11-04 08:39:47.687 INFO  [kea-dhcp4.ha-hooks/108440.139816890029760] HA_MAINTENANCE_SHUTDOWN_SAFE standby: the server can now be shutdown for maintenance as the partner has taken over the DHCP traffic

I checked status of both servers echo '{"command":"status-get"}' | sudo socat UNIX:/tmp/socket4 -,ignoreeof | jq and primary had "state": "partner-in-maintenance" and standby had "state": "in-maintenance"

I then sent ha-maintenance-cancel to the primary: echo '{"command":"ha-maintenance-cancel"}' | sudo socat UNIX:/tmp/socket4 -,ignoreeof | jq

I then sent status again but they had not exited their previous status: Logs from primary:

2024-11-04 08:43:31.782 INFO  [kea-dhcp4.commands/120617.140638165357632] COMMAND_RECEIVED Received command 'ha-maintenance-cancel'
2024-11-04 08:43:31.785 INFO  [kea-dhcp4.ha-hooks/120617.140638165357632] HA_STATE_TRANSITION primary: server transitions from PARTNER-IN-MAINTENANCE to HOT-STANDBY state, partner state is IN-MAINTENANCE
2024-11-04 08:43:31.785 INFO  [kea-dhcp4.ha-hooks/120617.140638165357632] HA_LEASE_UPDATES_ENABLED primary: lease updates will be sent to the partner while in HOT-STANDBY state
2024-11-04 08:43:32.540 INFO  [kea-dhcp4.dhcp4/120617.140638041446080] DHCP4_QUERY_LABEL received query: [hwtype=1 00:0c:01:02:03:06], cid=[01:00:0c:01:02:03:06], tid=0xec
2024-11-04 08:43:32.540 INFO  [kea-dhcp4.packets/120617.140638041446080] DHCP4_PACKET_RECEIVED [hwtype=1 00:0c:01:02:03:06], cid=[01:00:0c:01:02:03:06], tid=0xec: DHCPDISCOVER (type 1) received from 172.28.0.250 to 255.255.255.255 on interface enp0s8
2024-11-04 08:43:32.540 INFO  [kea-dhcp4.leases/120617.140638041446080] DHCP4_LEASE_OFFER [hwtype=1 00:0c:01:02:03:06], cid=[01:00:0c:01:02:03:06], tid=0xec: lease 172.28.0.13 will be offered
2024-11-04 08:43:32.540 INFO  [kea-dhcp4.packets/120617.140638041446080] DHCP4_PACKET_SEND [hwtype=1 00:0c:01:02:03:06], cid=[01:00:0c:01:02:03:06], tid=0xec: trying to send packet DHCPOFFER (type 2) from 172.28.0.253:67 to 172.28.0.250:67 on interface enp0s8
2024-11-04 08:43:32.541 INFO  [kea-dhcp4.dhcp4/120617.140638049838784] DHCP4_QUERY_LABEL received query: [hwtype=1 00:0c:01:02:03:06], cid=[01:00:0c:01:02:03:06], tid=0xec
2024-11-04 08:43:32.541 INFO  [kea-dhcp4.packets/120617.140638049838784] DHCP4_PACKET_RECEIVED [hwtype=1 00:0c:01:02:03:06], cid=[01:00:0c:01:02:03:06], tid=0xec: DHCPREQUEST (type 3) received from 172.28.0.250 to 255.255.255.255 on interface enp0s8
2024-11-04 08:43:32.541 INFO  [kea-dhcp4.leases/120617.140638049838784] DHCP4_LEASE_ALLOC [hwtype=1 00:0c:01:02:03:06], cid=[01:00:0c:01:02:03:06], tid=0xec: lease 172.28.0.13 has been allocated for 300 seconds
2024-11-04 08:43:32.542 INFO  [kea-dhcp4.ha-hooks/120617.140638116980416] HA_STATE_TRANSITION primary: server transitions from HOT-STANDBY to PARTNER-IN-MAINTENANCE state, partner state is IN-MAINTENANCE
2024-11-04 08:43:32.543 INFO  [kea-dhcp4.ha-hooks/120617.140638116980416] HA_LEASE_UPDATES_ENABLED primary: lease updates will be sent to the partner while in PARTNER-IN-MAINTENANCE state
2024-11-04 08:43:32.543 INFO  [kea-dhcp4.ha-hooks/120617.140638116980416] HA_MAINTENANCE_STARTED primary: the server is now in the partner-in-maintenance state and the partner is in-maintenance state
2024-11-04 08:43:32.543 INFO  [kea-dhcp4.packets/120617.140638033053376] DHCP4_PACKET_SEND [hwtype=1 00:0c:01:02:03:06], cid=[01:00:0c:01:02:03:06], tid=0xec: trying to send packet DHCPACK (type 5) from 172.28.0.253:67 to 172.28.0.250:67 on interface enp0s8
2024-11-04 08:43:33.540 INFO  [kea-dhcp4.dhcp4/120617.140638058231488] DHCP4_QUERY_LABEL received query: [hwtype=1 00:0c:01:02:03:07], cid=[01:00:0c:01:02:03:07], tid=0xed
2024-11-04 08:43:33.540 INFO  [kea-dhcp4.packets/120617.140638058231488] DHCP4_PACKET_RECEIVED [hwtype=1 00:0c:01:02:03:07], cid=[01:00:0c:01:02:03:07], tid=0xed: DHCPDISCOVER (type 1) received from 172.28.0.250 to 255.255.255.255 on interface enp0s8

Logs from standby:

2024-11-04 08:43:31.746 INFO  [kea-dhcp4.commands/108440.139816873244352] COMMAND_RECEIVED Received command 'ha-maintenance-notify'
2024-11-04 08:43:31.746 INFO  [kea-dhcp4.ha-hooks/108440.139816873244352] HA_STATE_TRANSITION standby: server transitions from IN-MAINTENANCE to HOT-STANDBY state, partner state is PARTNER-IN-MAINTENANCE
2024-11-04 08:43:31.746 INFO  [kea-dhcp4.ha-hooks/108440.139816873244352] HA_LEASE_UPDATES_ENABLED standby: lease updates will be sent to the partner while in HOT-STANDBY state
2024-11-04 08:43:31.746 INFO  [kea-dhcp4.ha-hooks/108440.139816873244352] HA_LOCAL_DHCP_ENABLE local DHCP service is enabled while the standby is in the HOT-STANDBY state
2024-11-04 08:43:32.504 INFO  [kea-dhcp4.commands/108440.139816898422464] COMMAND_RECEIVED Received command 'lease4-update'
2024-11-04 08:43:33.504 INFO  [kea-dhcp4.commands/108440.139816890029760] COMMAND_RECEIVED Received command 'lease4-update'
2024-11-04 08:43:34.504 INFO  [kea-dhcp4.commands/108440.139816873244352] COMMAND_RECEIVED Received command 'lease4-update'
2024-11-04 08:43:35.504 INFO  [kea-dhcp4.commands/108440.139816890029760] COMMAND_RECEIVED Received command 'lease4-update'
2024-11-04 08:43:36.504 INFO  [kea-dhcp4.commands/108440.139816873244352] COMMAND_RECEIVED Received command 'lease4-update'
2024-11-04 08:43:37.503 INFO  [kea-dhcp4.commands/108440.139816881637056] COMMAND_RECEIVED Received command 'lease4-update'
2024-11-04 08:43:38.506 INFO  [kea-dhcp4.commands/108440.139816898422464] COMMAND_RECEIVED Received command 'lease4-update'
2024-11-04 08:43:39.505 INFO  [kea-dhcp4.commands/108440.139816881637056] COMMAND_RECEIVED Received command 'lease4-update'
2024-11-04 08:43:40.504 INFO  [kea-dhcp4.ha-hooks/108440.139816931993280] HA_STATE_TRANSITION standby: server transitions from HOT-STANDBY to IN-MAINTENANCE state, partner state is PARTNER-IN-MAINTENANCE
2024-11-04 08:43:40.504 INFO  [kea-dhcp4.ha-hooks/108440.139816931993280] HA_LEASE_UPDATES_DISABLED standby: lease updates will not be sent to the partner while in IN-MAINTENANCE state
2024-11-04 08:43:40.504 INFO  [kea-dhcp4.ha-hooks/108440.139816931993280] HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the standby is in the IN-MAINTENANCE state
2024-11-04 08:43:40.504 INFO  [kea-dhcp4.ha-hooks/108440.139816931993280] HA_MAINTENANCE_SHUTDOWN_SAFE standby: the server can now be shutdown for maintenance as the partner has taken over the DHCP traffic
2024-11-04 08:43:40.504 INFO  [kea-dhcp4.commands/108440.139816881637056] COMMAND_RECEIVED Received command 'ha-heartbeat'

Some notes about the above:

  • The cancel would SOMETIMES work when when there was traffic present. It seems to be some kind of timing issue.
  • It always worked if no traffic was present.
  • It didn't seem to matter to which server I sent the ha-maintenance-start command, the cancel seemed to have an equal chance of failure.
  • Restarting the in-maintenance server (and thus having no use for the cancel command) worked fine. When the previously in-maintenance server returned to service, the servers both returned to the normal hot-standby state.
  • This was repeatable on both Kea 2.4.1 and 2.6.1.

RT24138

Edited by Darren Ankney
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information