Kea HA load balancing setup API failed to communicate with DHCP4 (GH #108)
<migrated from Github issue #108. opened there by learn3r@github.com on October 15, 2018>
Describe the bug HA load balancing setup not working with API failed to communicate with dhcp4 server
Server 1 primary 018-10-15 06:25:57.954 WARN [kea-dhcp4.ha-hooks/12395] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.12:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:26:08.967 WARN [kea-dhcp4.ha-hooks/12395] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.12:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:26:19.981 WARN [kea-dhcp4.ha-hooks/12395] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.12:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:26:30.994 WARN [kea-dhcp4.ha-hooks/12395] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.12:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:26:42.008 WARN [kea-dhcp4.ha-hooks/12395] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.12:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1
Server 2 secondary
[root@test2 tmp]# keactrl status DHCPv4 server: active DHCPv6 server: active DHCP DDNS: inactive Control Agent: active Kea DHCPv4 configuration file: /etc/kea/kea-dhcp4.conf Kea DHCPv6 configuration file: /etc/kea/kea-dhcp6.conf Kea DHCP DDNS configuration file: /etc/kea/kea-dhcp-ddns.conf Kea Control Agent configuration file: /etc/kea/kea-ctrl-agent.conf keactrl configuration file: /etc/kea/keactrl.conf [root@test2 tmp]# ps -aef | grep kea root 29062 1 0 06:27 pts/2 00:00:00 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf root 29069 1 0 06:27 pts/2 00:00:00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf root 29078 1 0 06:27 pts/2 00:00:00 /usr/sbin/kea-ctrl-agent -c /etc/kea/kea-ctrl-agent.conf root 29131 4193 0 06:31 pts/2 00:00:00 grep --color=auto kea
-agent.commands/29078] COMMAND_RECEIVED Received command 'ha-heartbeat' 2018-10-15 06:29:04.182 INFO [kea-ctrl-agent.commands/29078] COMMAND_RECEIVED Received command 'ha-heartbeat' 2018-10-15 06:29:15.196 INFO [kea-ctrl-agent.commands/29078] COMMAND_RECEIVED Received command 'ha-heartbeat'
2018-10-15 06:28:59.221 WARN [kea-dhcp4.ha-hooks/29062] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.13:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:29:10.236 WARN [kea-dhcp4.ha-hooks/29062] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.13:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:29:21.249 WARN [kea-dhcp4.ha-hooks/29062] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.13:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:29:32.264 WARN [kea-dhcp4.ha-hooks/29062] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.13:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:29:43.277 WARN [kea-dhcp4.ha-hooks/29062] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.13:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 To Reproduce Steps to reproduce the behavior:
Run Kea (which daemon? dhcpv4, dhcpv6, ddns, ca?) with the following config '...' configure kea HA with load balancing setup A client does A and sends packet B with options C,D,E via relay F that does '...' client A sends heartbeat but client B failed to handle both clients are same The server then '...' See error failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 Environment:
Kea version: which release? if it's compiled from git, which revision. Use kea-dhcp4 -V to find out. kea-dhcp4 -V ** 1.4.0 tarball linked with: log4cplus 1.1.3 OpenSSL 1.0.2k-fips 26 Jan 2017 database: MySQL backend 6.0, library 5.5.60-MariaDB Memfile backend 2.0 **
OS: [e.g. Ubuntu 16.04 x64] Centos 7 x64 Which features were compiled in (in particular which backends) If/which hooks where loaded in control agent config : { "Control-agent": { "http-host": "10.25.133.13", "http-port": 8080,
"control-sockets": {
"dhcp4": {
"socket-type": "unix",
"socket-name": "/tmp/socket.sock"
},
"dhcp6": {
"socket-type": "unix",
"socket-name": "/tmp/socket.sock"
}
},
"hooks-libraries": [{
"library": "/usr/local/lib/hooks/libdhcp_lease_cmds.so"
},
{
"library": "/usr/local/lib/hooks/libdhcp_stat_cmds.so"
}
]
},
"Logging": {
"loggers": [{
"name": "kea-ctrl-agent",
"severity": "DEBUG",
"output_options": [{
"output": "/usr/local/var/log/kea-ctrl-agent.log"
}]
}]
}
} Note: checked firewall, both API able to talk but not able communicate with running dhcp4 daemons. Both API receives heartbeat :
-agent.commands/29078] COMMAND_RECEIVED Received command 'ha-heartbeat' 2018-10-15 06:29:04.182 INFO [kea-ctrl-agent.commands/29078] COMMAND_RECEIVED Received command 'ha-heartbeat' 2018-10-15 06:29:15.196 INFO [kea-ctrl-agent.commands/29078] COMMAND_RECEIVED Received command 'ha-heartbeat' anyone could have suggestions ?