Kea HA load balancing setup API failed to communicate with DHCP4
Describe the bug HA load balancing setup not working with API failed to communicate with dhcp4 server
Server 1 primary 018-10-15 06:25:57.954 WARN [kea-dhcp4.ha-hooks/12395] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.12:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:26:08.967 WARN [kea-dhcp4.ha-hooks/12395] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.12:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:26:19.981 WARN [kea-dhcp4.ha-hooks/12395] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.12:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:26:30.994 WARN [kea-dhcp4.ha-hooks/12395] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.12:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:26:42.008 WARN [kea-dhcp4.ha-hooks/12395] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.12:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1
Server 2 secondary
[root@test2 tmp]# keactrl status DHCPv4 server: active DHCPv6 server: active DHCP DDNS: inactive Control Agent: active Kea DHCPv4 configuration file: /etc/kea/kea-dhcp4.conf Kea DHCPv6 configuration file: /etc/kea/kea-dhcp6.conf Kea DHCP DDNS configuration file: /etc/kea/kea-dhcp-ddns.conf Kea Control Agent configuration file: /etc/kea/kea-ctrl-agent.conf keactrl configuration file: /etc/kea/keactrl.conf [root@test2 tmp]# ps -aef | grep kea root 29062 1 0 06:27 pts/2 00:00:00 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf root 29069 1 0 06:27 pts/2 00:00:00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf root 29078 1 0 06:27 pts/2 00:00:00 /usr/sbin/kea-ctrl-agent -c /etc/kea/kea-ctrl-agent.conf root 29131 4193 0 06:31 pts/2 00:00:00 grep --color=auto kea
-agent.commands/29078] COMMAND_RECEIVED Received command 'ha-heartbeat' 2018-10-15 06:29:04.182 INFO [kea-ctrl-agent.commands/29078] COMMAND_RECEIVED Received command 'ha-heartbeat' 2018-10-15 06:29:15.196 INFO [kea-ctrl-agent.commands/29078] COMMAND_RECEIVED Received command 'ha-heartbeat'
2018-10-15 06:28:59.221 WARN [kea-dhcp4.ha-hooks/29062] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.13:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:29:10.236 WARN [kea-dhcp4.ha-hooks/29062] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.13:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:29:21.249 WARN [kea-dhcp4.ha-hooks/29062] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.13:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:29:32.264 WARN [kea-dhcp4.ha-hooks/29062] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.13:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1 2018-10-15 06:29:43.277 WARN [kea-dhcp4.ha-hooks/29062] HA_HEARTBEAT_FAILED heartbeat to (http://10.25.133.13:8080/) failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1
To Reproduce Steps to reproduce the behavior:
Run Kea (which daemon? dhcpv4, dhcpv6, ddns, ca?) with the following config '...' configure kea HA with load balancing setup A client does A and sends packet B with options C,D,E via relay F that does '...' client A sends heartbeat but client B failed to handle both clients are same The server then '...' See error failed: unable to forward command to the dhcp4 service: No such file or directory. The server is likely to be offline, error code 1
Environment:
Kea version: which release? if it's compiled from git, which revision. Use kea-dhcp4 -V to find out. kea-dhcp4 -V ** 1.4.0 tarball linked with: log4cplus 1.1.3 OpenSSL 1.0.2k-fips 26 Jan 2017 database: MySQL backend 6.0, library 5.5.60-MariaDB Memfile backend 2.0
**
OS: [e.g. Ubuntu 16.04 x64] Centos 7 x64
Which features were compiled in (in particular which backends) If/which hooks where loaded in control agent config : { "Control-agent": { "http-host": "10.25.133.13", "http-port": 8080,
"control-sockets": {
"dhcp4": {
"socket-type": "unix",
"socket-name": "/tmp/socket.sock"
},
"dhcp6": {
"socket-type": "unix",
"socket-name": "/tmp/socket.sock"
}
},
"hooks-libraries": [{
"library": "/usr/local/lib/hooks/libdhcp_lease_cmds.so"
},
{
"library": "/usr/local/lib/hooks/libdhcp_stat_cmds.so"
}
]
},
"Logging": {
"loggers": [{
"name": "kea-ctrl-agent",
"severity": "DEBUG",
"output_options": [{
"output": "/usr/local/var/log/kea-ctrl-agent.log"
}]
}]
}
}
kea-dhcp4.conf :
{ "Dhcp4": { "interfaces-config": { "interfaces": ["ens160"], "dhcp-socket-type": "raw"
}, "valid-lifetime": 60000, "renew-timer": 10000, "rebind-timer": 20000, "subnet4": [{ "id": 2, "subnet": "10.25.133.0/24", "pools": [{ "pool": "10.25.133.2-10.25.133.100" }] }], "lease-database": { "type": "mysql", "name": "kea", "user": "kea", "password": "***", "host": "10.25.133.13", "port": 3306 }, "hosts-database": { "type": "mysql", "name": "kea", "user": "kea", "password": "***", "host": "10.25.133.13", "port": 3306 }, "hooks-libraries": [{ "library": "/usr/lib64/hooks/libdhcp_lease_cmds.so" }, { "library": "/usr/lib64/hooks/libdhcp_stat_cmds.so" }, { "library": "/usr/lib64/hooks/libdhcp_ha.so", "parameters": { "high-availability": [{ "this-server-name": "test2", "mode": "load-balancing", "heartbeat-interval": 60, "max-response-delay": 100, "max-ack-delay": 100, "max-unacked-messages": 10, "peers": [{ "name": "test3", "url": "http://10.25.133.13:8080/", "role": "primary", "auto-failover": true }, { "name": "test2", "url": "http://10.25.133.12:8080/", "role": "secondary", "auto-failover": true } ] }] } } ] }, "Logging": { "loggers": [{ "name": "kea-dhcp4", "output_options": [{ "output": "/var/log/kea-dhcp4.log" }], "severity": "DEBUG" }] }
}
Note: checked firewall, both API able to talk but not able communicate with running dhcp4 daemons. Both API receives heartbeat :
-agent.commands/29078] COMMAND_RECEIVED Received command 'ha-heartbeat' 2018-10-15 06:29:04.182 INFO [kea-ctrl-agent.commands/29078] COMMAND_RECEIVED Received command 'ha-heartbeat' 2018-10-15 06:29:15.196 INFO [kea-ctrl-agent.commands/29078] COMMAND_RECEIVED Received command 'ha-heartbeat'
anyone could have suggestions ?