ISC Open Source Projects issueshttps://gitlab.isc.org/groups/isc-projects/-/issues2024-02-08T14:35:31Zhttps://gitlab.isc.org/isc-projects/kea/-/issues/3040failure to start up with ppp interfaces2024-02-08T14:35:31ZJaco Kroonfailure to start up with ppp interfaces2023-08-31 23:34:48.506 WARN [kea-dhcp6.dhcpsrv/27456.140590317625408] DHCPSRV_OPEN_SOCKET_FAIL failed to open socket: Failed to open multicast socket on interface ppp0, reason: Failed to open link-local socket on interface ppp0: Failed...2023-08-31 23:34:48.506 WARN [kea-dhcp6.dhcpsrv/27456.140590317625408] DHCPSRV_OPEN_SOCKET_FAIL failed to open socket: Failed to open multicast socket on interface ppp0, reason: Failed to open link-local socket on interface ppp0: Failed to bind socket 20 to fe80::e/port=547: Cannot assign requested address
```
26: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1350 qdisc tbf state UNKNOWN group default qlen 3
inet6 fe80::14e4:139f:8862:3dfd peer fe80::e/128 scope link
valid_lft forever preferred_lft forever
```
Obviously we should bind to the local link-local, not the remote one.
kea 2.4.0. 2.5.1 segfaults on my configuration:
```
{
"Dhcp6": {
"interfaces-config": {
"interfaces": [ "ppp0" ]
},
"control-socket": {
"socket-type": "unix",
"socket-name": "/run/kea/kea6-ctrl-socket"
},
"lease-database": {
"type": "memfile",
"lfc-interval": 3600
},
"expired-leases-processing": {
"reclaim-timer-wait-time": 10,
"flush-reclaimed-timer-wait-time": 25,
"hold-reclaimed-time": 3600,
"max-reclaim-leases": 100,
"max-reclaim-time": 250,
"unwarned-reclaim-cycles": 5
},
"renew-timer": 1000,
"rebind-timer": 2000,
"preferred-lifetime": 3000,
"valid-lifetime": 4000,
"option-data": [
{
"name": "dns-servers",
"data": "2c0f:f720::..., 2c0f:f720::..."
}
],
"subnet6": [
{
"subnet": "2c0f:f720:fcf0::/44",
"pd-pools": [
{
"prefix": "2c0f:f720:fcf0::",
"prefix-len": 44,
"delegated-len": 56
}
]
}
],
"loggers": [
{
"name": "kea-dhcp6",
"output_options": [
{
"output": "/var/log/kea/kea-dhcp6.log"
}
],
"severity": "DEBUG",
"debuglevel": 99
}
]
}
}
```
Strace from a re-run (so socket number is different):
```
bind(19, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0
getsockname(19, {sa_family=AF_NETLINK, nl_pid=6688, nl_groups=00000000}, [12]) = 0
sendto(19, [{nlmsg_len=20, nlmsg_type=RTM_GETLINK, nlmsg_flags=NLM_F_REQUEST|NLM_F_DUMP, nlmsg_seq=1, nlmsg_pid=0}, {ifi_family=AF_PACKET, ...}], 20, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 20
recvmsg(19, {msg_name={sa_family=AF_NETLINK ...
ifa_index=if_nametoindex("ppp0")}, [[{nla_len=20, nla_type=IFA_LOCAL}, **inet_pton(AF_INET6, "fe80::14e4:139f:8862:3dfd")], [{nla_len=20, nla_type=IFA_ADDRESS}, inet_pton(AF_INET6, "fe80::e")],** [{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=168714876, tstamp=168714876}], [{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT]]], [{nlmsg_len=72, nlmsg_type=RTM_NEWADDR, nlmsg_flags=NLM_F_MULTI, nlmsg_seq=2, nlmsg_pid=6688}, {ifa_family=AF_INET6, ifa_prefixlen=64, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_LINK,
...
close(19) = 0
socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 19
ioctl(19, SIOCGIFINDEX, {ifr_name="ppp0", ifr_ifindex=26}) = 0
close(19) = 0
socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 19
fcntl(19, F_SETFD, FD_CLOEXEC) = 0
setsockopt(19, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
setsockopt(19, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0
setsockopt(19, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
bind(19, {sa_family=AF_INET6, sin6_port=htons(547), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "**fe80::e**", &sin6_addr), sin6_scope_id=if_nametoindex("ppp0")}, 28) = -1 EADDRNOTAVAIL (Cannot assign requested address)
close(19) = 0
```next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/3035Expose interface-id in dhcpv6 hook points2023-09-21T13:22:34ZSorin EsanuExpose interface-id in dhcpv6 hook pointsHello!
I see useful to expose option18 (interface id) and, eventually, option37 (remote id) to dhcpv6 hook points, in a similar way that option82 is sent in dhcpv4 hook points (QUERY4_OPTION_82 or QUERY4_OPTION_82_SUB_OPTION_1).
Thank you!Hello!
I see useful to expose option18 (interface id) and, eventually, option37 (remote id) to dhcpv6 hook points, in a similar way that option82 is sent in dhcpv4 hook points (QUERY4_OPTION_82 or QUERY4_OPTION_82_SUB_OPTION_1).
Thank you!next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/3024Post audit: Write KB article/ARM section about securing DB connection2023-09-21T10:10:03ZTomek MrugalskiPost audit: Write KB article/ARM section about securing DB connectionAs pointed out in [@manu's audit](https://gitlab.isc.org/isc-private/kea/-/wikis/Kea-Security-Review-02-2023#12-secure-db-connection), we should have a kind of tutorial or step by step guide how to secure Kea and MariaDB/Postgres connect...As pointed out in [@manu's audit](https://gitlab.isc.org/isc-private/kea/-/wikis/Kea-Security-Review-02-2023#12-secure-db-connection), we should have a kind of tutorial or step by step guide how to secure Kea and MariaDB/Postgres connection.
I think this would be best achieved by having a config example in the ARM + a KB article explaining how to do the changes in Maria/Postgres installations.next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/3022Post audit: retire MD5 and SHA12023-09-21T10:10:13ZTomek MrugalskiPost audit: retire MD5 and SHA1By using MD5 in particular and SHA1 to some degree, we risk getting more complaints, even if those are used in contexts that are not security related. Some procurement processes might simply ask if obsolete and broken algorithms are in u...By using MD5 in particular and SHA1 to some degree, we risk getting more complaints, even if those are used in contexts that are not security related. Some procurement processes might simply ask if obsolete and broken algorithms are in use. If they are, the software might be determined not suitable.
In one case, we had an interaction with a user who had those explicitly disabled in their deployment for security reasons.
If we decide to keep those, there should be a very good explanation in the ARM why keeping those is acceptable. "because getting rid of them is a lot of work" is not a good explanation.next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/3021Post audit: update ARM to show how to confirm source code integrity2023-09-21T10:09:54ZTomek MrugalskiPost audit: update ARM to show how to confirm source code integrityAnother proposal by [@manu's audit](https://gitlab.isc.org/isc-private/kea/-/wikis/Kea-Security-Review-02-2023#10-remote-administrative-access-authentication-restful-api). We should document how to check the integrity of the source code ...Another proposal by [@manu's audit](https://gitlab.isc.org/isc-private/kea/-/wikis/Kea-Security-Review-02-2023#10-remote-administrative-access-authentication-restful-api). We should document how to check the integrity of the source code (and packages, too).
With the SBOM being increasingly focused, this is an important aspect. Fortunately, it's very easy doc update.next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/3020Post audit: investigate if basic auth with http could be discouraged2023-09-21T10:09:47ZTomek MrugalskiPost audit: investigate if basic auth with http could be discouragedAnother [report by @manu](https://gitlab.isc.org/isc-private/kea/-/wikis/Kea-Security-Review-02-2023#10-remote-administrative-access-authentication-restful-api). The overall idea is to strongly discourage people from running basic auth o...Another [report by @manu](https://gitlab.isc.org/isc-private/kea/-/wikis/Kea-Security-Review-02-2023#10-remote-administrative-access-authentication-restful-api). The overall idea is to strongly discourage people from running basic auth over plain HTTP. It's insecure, but maybe in some experiments it might be useful, so we probably shouldn't forcibly abort.
But perhaps print a warning with a link to ARM section explaining why it's wrong would do?next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/3015Post audit: update docs2023-09-21T10:10:46ZTomek MrugalskiPost audit: update docs@manu completed an audit and pointed the following problems [in his report](https://gitlab.isc.org/isc-private/kea/-/wikis/Kea-Security-Review-02-2023):
- [ ] 2. Elaborate more about \[meta\] package, in particular how to run only selec...@manu completed an audit and pointed the following problems [in his report](https://gitlab.isc.org/isc-private/kea/-/wikis/Kea-Security-Review-02-2023):
- [ ] 2. Elaborate more about \[meta\] package, in particular how to run only selected service. Looks like a good idea also to update the security section and point out that one should only run the services that are strictly needed.
- [ ] 3. Elaborate more about `interface` requirement for making the service running in [ARM QuickStart section](https://kea.readthedocs.io/en/kea-2.2.0/arm/quickstart.html#quick-start-guide-for-dhcpv4-and-dhcpv6-services) - this should be easy, a sentence or two pointing out that binding address is by default set to defensive 127.0.0.1.
- [ ] 4. Correct references in configs
- [ ] 5. Clarify sentence in DDNS (it's `dhcp-ddns` in `Dhcp4` or `Dhcp6`)
- [ ] 6. Clarify about sockets open by default (add something like this "While Kea doesn't open any sockets by default on its own, the default configs shipped with packages do define some socket and Kea opens them).next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/3005ddns CHG_ADD before CHG_REMOVE2023-11-16T18:46:52Zphilip-smartbitddns CHG_ADD before CHG_REMOVE---
name: ddns add nsupdate before remove nsupdate
---
**Describe the bug**
we updated kea to 2.4.0 and set ddns-update-on-renew to true. Since the update we noticed that some hosts lost their dns records (but had a correct lease). fr...---
name: ddns add nsupdate before remove nsupdate
---
**Describe the bug**
we updated kea to 2.4.0 and set ddns-update-on-renew to true. Since the update we noticed that some hosts lost their dns records (but had a correct lease). from around 13:00 07/aug/2023 until now we had 6 hosts losing their dns record. we updated to kea 2.4.0 on 13:00 07/aug/2023
In the kea ddns log we noticed that the problem hosts all showed a CHG_ADD before a CHG_REMOVE:
2023-08-07 18:07:36.634 INFO [kea-dhcp-ddns.d2-to-dns/1587] DHCP_DDNS_ADD_SUCCEEDED DHCP_DDNS Request ID 000201B0E0B8DAF410D5E089236F7462BDCB78A628FBA03A6D38ADF43A848AF348D3F3: successfully added the DNS mapping addition for this request: Type: 0 (CHG_ADD)
Forward Change: yes
Reverse Change: yes
FQDN: [host01.internal.]
IP Address: [10.20.30.40]
DHCID: [000201B0E0B8DAF410D5E089236F7462BDCB78A628FBA03A6D38ADF43A848AF348D3F3]
Lease Expires On: 20230807161112
Lease Length: 216
Conflict Resolution: no
2023-08-07 18:07:36.674 INFO [kea-dhcp-ddns.d2-to-dns/1587] DHCP_DDNS_REMOVE_SUCCEEDED DHCP_DDNS Request ID 000201B0E0B8DAF410D5E089236F7462BDCB78A628FBA03A6D38ADF43A848AF348D3F3: successfully removed the DNS mapping addition for this request: Type: 1 (CHG_REMOVE)
Forward Change: yes
Reverse Change: yes
FQDN: [host01.internal.]
IP Address: [10.20.30.40]
DHCID: [000201B0E0B8DAF410D5E089236F7462BDCB78A628FBA03A6D38ADF43A848AF348D3F3]
Lease Expires On: 20230807154204
Lease Length: 216
Conflict Resolution: no
**To Reproduce**
Steps to reproduce the behavior:
1. Run Kea dhcp4 with the following settings enabled:
"ddns-update-on-renew": true,
"ddns-use-conflict-resolution": false
2. A few hunderd vm's doing a renew every 1800 seconds
3. wait and randomly some hosts lose their dns record's
4. See error above
**Expected behavior**
we expect that kea ddns *always* does a CHG_REMOVE before a CHG_ADD
**Environment:**
- Kea version: 2.4.0 with default multithreading on, package installed via cloudsmith debian repo
- OS: Debian 11
- ha is enabled with default multithreading on, hot-standby
- auth dns server is powerdns 4.8.1
**Additional Information**
we only use dhcp4, no dhcp6
we configured powerdns with distributor-threads=1 and reuseport=no
The kea and (power)dns vm's didn't have high cpu usage or iowait, they weren't overload in any way.
**Contacting you**
contact via gitlab or emailnext-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/2999configure makes erroneous assumption to discover location of OpenSSL librarie...2023-08-10T13:50:53ZMark Priorconfigure makes erroneous assumption to discover location of OpenSSL libraries on macOSThe configure script supplied in the tarball aborts when trying to determine support for OpenSSL routines on macOS due to an erroneous assumption.
The configure script builds code snippets to determine availability of various EVP routin...The configure script supplied in the tarball aborts when trying to determine support for OpenSSL routines on macOS due to an erroneous assumption.
The configure script builds code snippets to determine availability of various EVP routines. To determine the location of the OpenSSL libraries it executes the following code
```
# Now search for the system OpenSSL library.
if test "${use_openssl}" = "yes" ; then
for d in /usr /usr/local /usr/local/ssl /usr/local/opt/openssl /usr/pkg /usr/sfw; do
if test -f $d/include/openssl/opensslv.h; then
use_openssl="${d}"
openssl_headers="${d}/include"
for l in lib lib64; do
if test -f "${d}/${l}/libssl.so"; then
openssl_libraries="${d}/${l}"
break
fi
done
if test -n "${openssl_headers}" && test -n "${openssl_libraries}"; then
break
fi
fi
done
fi
```
macOS does not use .so type libraries and so even though this code successfully sets use_openssl and openssl_headers if OpenSSL is installed it fails to set openssl_libraries.
The following code is then executed
```
if test "${openssl_headers}" = "/usr/include" ; then
CRYPTO_CFLAGS=""
CRYPTO_INCLUDES=""
CRYPTO_LIBS="-lssl -lcrypto"
else
CRYPTO_CFLAGS=""
CRYPTO_INCLUDES="-I${openssl_headers}"
case $host in
*-solaris*)
CRYPTO_LIBS="-L${openssl_libraries} -R${openssl_libraries} -lssl -lcrypto"
;;
*-hp-hpux*)
CRYPTO_LIBS="-L${openssl_libraries} -Wl,+b: -lssl -lcrypto"
;;
*-apple-darwin*)
if test -f "${openssl_libraries}/libcrypto.dylib" ; then
CRYPTO_LIBS="-L${openssl_libraries} -lssl -lcrypto"
else
CRYPTO_LIBS="${openssl_libraries}/libssl.a ${openssl_libraries}/libcrypto.a"
fi
;;
*)
CRYPTO_LIBS="-L${openssl_libraries} -lssl -lcrypto"
;;
esac
fi
```
If openssl_headers is NOT `/usr/include`, which is likely on macOS since it's not an Apple supplied library and a standard build of OpenSSL will install in `/usr/local` instead of `/usr`, then CRYPTO_LIBS will be set to `/libssl.a /lib/crypto.a`, neither of which exist.
This will cause the following code snippets that test the existence of various EVP routing to all fail and hence configure aborts.
**Environment:**
- Kea version: kea-2.4.0
- OS: macOS 10.13.6next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/2981"make coverage" doesn't work2023-08-10T13:44:47ZAndrei Pavelandrei@isc.org"make coverage" doesn't workThe coverage job in our internal Jenkins test environment started failing, when we upgraded Fedora from 36 to 38, likely because of a new gcovr version. That job uses an out-of-band set of actions to build the coverage report. While look...The coverage job in our internal Jenkins test environment started failing, when we upgraded Fedora from 36 to 38, likely because of a new gcovr version. That job uses an out-of-band set of actions to build the coverage report. While looking for alternatives, it was noticed that `make coverage` doesn't work either:
```
$ make coverage
[... goes on to run unit tests, but when the report needs to be made aka "make report-coverage" ...]
/bin/sh: -c: line 9: syntax error: unexpected end of file
make: *** [Makefile:1126: report-cpp-coverage] Error 2
```next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/2974New localize DHCP server command2023-09-01T14:23:21ZFrancis DupontNew localize DHCP server commandThe idea is to add a new command which returns the selected subnet (id and text) with eventually the shared network from supplied parametes as an IP address, client classes (for guards) or interface name.The idea is to add a new command which returns the selected subnet (id and text) with eventually the shared network from supplied parametes as an IP address, client classes (for guards) or interface name.next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/2961Remove support for the deprecated auto-generated subnet IDs feature2023-08-10T13:36:06ZAndrei Pavelandrei@isc.orgRemove support for the deprecated auto-generated subnet IDs feature[As mentioned in the ARM](https://kea.readthedocs.io/en/latest/arm/dhcp4-srv.html#ipv4-subnet-identifier):
> It is one of the reasons why auto-generated subnet identifiers are deprecated starting from Kea version 2.4.0.
The deprecatio...[As mentioned in the ARM](https://kea.readthedocs.io/en/latest/arm/dhcp4-srv.html#ipv4-subnet-identifier):
> It is one of the reasons why auto-generated subnet identifiers are deprecated starting from Kea version 2.4.0.
The deprecation happened in Kea 2.4. Kea 2.4 shipped with the deprecation warning in the ARM. Kea 2.5 can have the feature removed.next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/2960Remove support for the deprecated libreload command2023-08-10T13:36:06ZAndrei Pavelandrei@isc.orgRemove support for the deprecated libreload command[As mentioned in the ARM](https://kea.readthedocs.io/en/latest/arm/install.html#libreload-command):
> The libreload command was deprecated in Kea 2.3.4. The code to handle this command is still there, but there are reports of it being b...[As mentioned in the ARM](https://kea.readthedocs.io/en/latest/arm/install.html#libreload-command):
> The libreload command was deprecated in Kea 2.3.4. The code to handle this command is still there, but there are reports of it being buggy and not really usable. Kea 2.3 and 2.4 versions will produce a warning when this command is used, and it will be removed entirely sometime in the 2.5 branch.next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/2959Remove support for the deprecated reservation-mode config flag2023-08-10T13:36:06ZAndrei Pavelandrei@isc.orgRemove support for the deprecated reservation-mode config flag[As mentioned in the ARM](https://kea.readthedocs.io/en/latest/arm/dhcp4-srv.html#address-reservation-types):
> Since Kea 1.9.1, reservation mode has been replaced by three boolean flags, reservations-global, reservations-in-subnet, and...[As mentioned in the ARM](https://kea.readthedocs.io/en/latest/arm/dhcp4-srv.html#address-reservation-types):
> Since Kea 1.9.1, reservation mode has been replaced by three boolean flags, reservations-global, reservations-in-subnet, and reservations-out-of-pool, which allow the configuration of host reservations both globally and in a subnet.
The deprecation happened in Kea 1.9. Kea 2.0 shipped with the deprecation warning in the ARM. Kea 2.1 could have had support for these parameters removed.next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/2958Remove support for the deprecated dhcp-ddns config flags2023-08-10T13:36:06ZAndrei Pavelandrei@isc.orgRemove support for the deprecated dhcp-ddns config flags[As mentioned in the ARM](https://kea.readthedocs.io/en/latest/arm/dhcp4-srv.html#ddns-for-dhcpv4):
> [the D2 parameters] have been moved out of the dhcp-ddns section so that they may be specified at the global, shared-network, and/or s...[As mentioned in the ARM](https://kea.readthedocs.io/en/latest/arm/dhcp4-srv.html#ddns-for-dhcpv4):
> [the D2 parameters] have been moved out of the dhcp-ddns section so that they may be specified at the global, shared-network, and/or subnet levels.
The deprecation happened in Kea 1.7. Kea 1.8 shipped with the deprecation warning in the ARM. Kea 1.9 could have had support for these parameters removed.
As a side note, the quotes statement contradicts another one in the ARM that says:
> These parameters can only be specified within the top-level dhcp-ddns section in the kea-dhcp4 configuration.
Also `ncr-format"` has an extraneous quote around that part of the ARM.next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/2953migrate hammer installations from mysql to mariadb2023-11-07T10:25:44ZWlodzimierz Wencelmigrate hammer installations from mysql to mariadbhammer install mysql in ubuntu and debian systems, change it for mariadbhammer install mysql in ubuntu and debian systems, change it for mariadbnext-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/2943strict check for prefix and prefix length for ipv6 PDs in lease and host cons...2023-07-01T15:01:21ZRazvan Becheriustrict check for prefix and prefix length for ipv6 PDs in lease and host constructorsThe safest way to make sure Kea internally does not construct a PD lease with invalid prefix and prefix length is to add the check in lease and host constructors.
Currently (after the merge of #2725) the possibility to send invalid PD l...The safest way to make sure Kea internally does not construct a PD lease with invalid prefix and prefix length is to add the check in lease and host constructors.
Currently (after the merge of #2725) the possibility to send invalid PD leases to clients comes from the database (either lease or host) which are maintained consistent when using KEA API, but the user can always add entries using database CLI and the data will reach the clients unchecked.
This proposal is robust and simple.
If some flexibility is needed in UTs, a 'test' flag can be set in a static/global variable which can disable these checks.next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/2940return hash in config-set response from CA2023-06-29T13:33:06ZTomek Mrugalskireturn hash in config-set response from CAThe #2707 introduced hash being returned by `config-set` response for DHCPv4, DHCPv6 and D2. This ticket is about adding the same functionality for Ctrl Agent. This is a completion of the task started in #2707.The #2707 introduced hash being returned by `config-set` response for DHCPv4, DHCPv6 and D2. This ticket is about adding the same functionality for Ctrl Agent. This is a completion of the task started in #2707.next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/2933Kea 2.2.0 bad UDP checksum in virtual with virtio NIC2023-06-22T13:45:35ZJan CernohorskyKea 2.2.0 bad UDP checksum in virtual with virtio NIC**Describe the bug**
Hi,
I have a problem with Kea v2.2.0 from Debian Bookworm.
Requests from clients are delivered to the server, but reply is not delivered to the client because bad UDP checksum.
I have installed Kea in KVM virtual....**Describe the bug**
Hi,
I have a problem with Kea v2.2.0 from Debian Bookworm.
Requests from clients are delivered to the server, but reply is not delivered to the client because bad UDP checksum.
I have installed Kea in KVM virtual. NIC is virtio.
Same problem was with ISC-DHCP-SERVER ... but after while it was corrected. Like https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717217 or https://lists.isc.org/pipermail/dhcp-workers/2021-October/000296.html .
**To Reproduce**
Steps to reproduce the behavior:
1. Run Kea dhcpv4 in virtual host Debian Bookworm, hosted on KVM, NIC driver virtio, listening on vlan interface
2. Client send request over DHCP Relay, relay is set to forward requests to the server
3. Server receive request
2023-06-19T21:53:55.033537+02:00 kea-server kea-dhcp4[1006] 2023-06-19 21:53:55.033 INFO [kea-dhcp4.leases/1006.140669572348352] DHCP4_LEASE_ADVERT [hwtype=1 11:22:33:44:55:66], cid=[01:11:22:33:44:55:66], tid=0x4fb1ce6c: lease 10.11.12.13 will be advertised
4. Client not get any reply.
**Expected behavior**
Client get advertised IP address from Kea server.
**Environment:**
- Kea version: 2.2.0 from distro Debian Bookworm
- OS: Debian Bookworm amd64
**Additional Information**
**Request:**
```
21:31:30.293127 99:88:77:66:55:44 > 52:54:00:00:00:00, ethertype IPv4 (0x0800), length 342: (tos 0x0, ttl 64, id 14533, offset 0, flags [DF], proto UDP (17), length 328)
10.12.13.14.67 > 10.12.13.15.67: [udp sum ok] BOOTP/DHCP, Request from 11:22:33:44:55:66, length 300, hops 1, xid 0x7edf76f, secs 20667, Flags [none] (0x0000)
Gateway-IP 10.11.12.254
Client-Ethernet-Address 11:22:33:44:55:66
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Parameter-Request (55), length 8:
Subnet-Mask (1), Default-Gateway (3), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), BR (28), NTP (42), Vendor-Option (43)
MSZ (57), length 2: 576
Vendor-Class (60), length 4: "abcd"
Client-ID (61), length 7: ether 11:22:33:44:55:66
Agent-Information (82), length 16:
Circuit-ID SubOption 1, length 6: ^XM-}ttM-/(
Remote-ID SubOption 2, length 6: vlan10
```
**Reply:**
```
21:31:30.293740 52:54:00:00:00:00 > 99:88:77:66:55:44, ethertype IPv4 (0x0800), length 365: (tos 0x0, ttl 64, id 45947, offset 0, flags [DF], proto UDP (17), length 351)
10.12.13.15.67 > 10.12.13.14.67: [**bad udp cksum 0x247b -> 0x7b5a!**] BOOTP/DHCP, Reply, length 323, hops 1, xid 0x7edf76f, Flags [none] (0x0000)
Your-IP 10.11.12.13
Gateway-IP 10.11.12.254
Client-Ethernet-Address 11:22:33:44:55:66
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Offer
Subnet-Mask (1), length 4: 255.255.255.0
Default-Gateway (3), length 4: 10.11.12.254
Domain-Name-Server (6), length 8: 10.11.12.254
Hostname (12), length 4: "abcde"
Lease-Time (51), length 4: 31680
Server-ID (54), length 4: 10.12.13.15
RN (58), length 4: 15840
RB (59), length 4: 27720
Client-ID (61), length 7: ether 11:22:33:44:55:66
Agent-Information (82), length 16:
Circuit-ID SubOption 1, length 6: ^XM-}ttM-/(
Remote-ID SubOption 2, length 6: vlan10
```
**Contact**
You can contact me on my mail daemoncze@gmail.com or on jabber daemoncze@jabbim.cz .next-stable-2.6https://gitlab.isc.org/isc-projects/kea/-/issues/2932Kea HA issue with terminating connection2023-11-10T09:50:24ZNick HahnKea HA issue with terminating connectionWe recently migrated our DHCP setup from dhcpd to Kea. It runs on
two servers with hot standby and a memfile backend for the leases. Kea
assigns IP addresses for around 7000 pools.
Over the past few months the HA connection terminated...We recently migrated our DHCP setup from dhcpd to Kea. It runs on
two servers with hot standby and a memfile backend for the leases. Kea
assigns IP addresses for around 7000 pools.
Over the past few months the HA connection terminated in random intervals.
From looking at the logs on the passive node I can see a lot of
'ResourceBusy: IP address ... could not be updated' warnings prior to
the connection terminating. Since multithreading is enabled I suspected
this may be due to the threads encountering a resource lock on the memfile.
I suppose after the lease update fails a few times, the connection is terminated.
Is the 'ResourceBusy' warning the cause for the terminating HA connection and
is there any way to fix the underlying issue? Any ideas on the issue are greatly
appraciated.
Here are the logs from the primary server:
```
Jun 12 15:04:31 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625735366400] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1, cid=[], tid=0x0: failed to allocate an IPv4 lease in the subnet 123.123.123.123/30, subnet-id 30926, shared network (none)
Jun 12 15:04:31 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625735366400] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 address after 1 attempt(s)
Jun 12 15:04:31 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625735366400] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1], cid=[], tid=0x0: Failed to allocate an IPv4 address for client with classes: ALL, HA_primary-dhcp, VENDOR_CLASS_MSFT 5.0, UNKNOWN
Jun 12 15:04:39 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625726973696] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 lease in the subnet 123.123.123.123/30, subnet-id 30926, shared network (none)
Jun 12 15:04:39 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625726973696] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 address after 1 attempt(s)
Jun 12 15:04:39 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625726973696] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1], cid=[], tid=0x0: Failed to allocate an IPv4 address for client with classes: ALL, HA_primary-dhcp, VENDOR_CLASS_MSFT 5.0, UNKNOWN
Jun 12 15:04:45 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.ha-hooks.139625718580992] HA_LEASE_UPDATE_CONFLICT [hwtype=1], cid=[], tid=0x0: lease update to standby-dhcp (http://dhcp-2:8001/) returned conflict status code: ResourceBusy: IP address:123.123.123.123 could not be updated. (error code 4)
Jun 12 15:04:56 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625735366400] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 lease in the subnet 123.123.123.123/30, subnet-id 30926, shared network (none)
Jun 12 15:04:56 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625735366400] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 address after 1 attempt(s)
Jun 12 15:04:56 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625735366400] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1], cid=[], tid=0x0: Failed to allocate an IPv4 address for client with classes: ALL, HA_primary-dhcp, VENDOR_CLASS_MSFT 5.0, UNKNOWN
Jun 12 15:05:28 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625726973696] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 lease in the subnet 123.123.123.123/30, subnet-id 30926, shared network (none)
Jun 12 15:05:28 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625726973696] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 address after 1 attempt(s)
Jun 12 15:05:28 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625726973696] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1], cid=[], tid=0x0: Failed to allocate an IPv4 address for client with classes: ALL, HA_primary-dhcp, VENDOR_CLASS_MSFT 5.0, UNKNOWN
Jun 12 15:05:31 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625752151808] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 lease in the subnet 123.123.123.123/30, subnet-id 30926, shared network (none)
Jun 12 15:05:31 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625752151808] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1], cid=[], tid=0x0: failed to allocate an IPv4 address after 1 attempt(s)
Jun 12 15:05:31 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.alloc-engine.139625752151808] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1], cid=[], tid=0x0: Failed to allocate an IPv4 address for client with classes: ALL, HA_primary-dhcp, VENDOR_CLASS_MSFT 5.0, UNKNOWN
Jun 12 15:05:39 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.ha-hooks.139625701795584] HA_LEASE_UPDATE_CONFLICT [hwtype=1], cid=[], tid=0x0: lease update to standby-dhcp (http://dhcp-2:8001/) returned conflict status code: ResourceBusy: IP address:123.123.123.123 could not be updated. (error code 4)
Jun 12 15:05:39 dhcp-1 kea-dhcp4[564812]: WARN [kea-dhcp4.ha-hooks.139625718580992] HA_LEASE_UPDATE_CONFLICT [hwtype=1], cid=[], tid=0x0: lease update to standby-dhcp (http://dhcp-2:8001/) returned conflict status code: ResourceBusy: IP address:123.123.123.123 could not be updated. (error code 4)
Jun 12 15:05:39 dhcp-1 kea-dhcp4[564812]: ERROR [kea-dhcp4.ha-hooks.139625718580992] HA_LEASE_UPDATE_REJECTS_CAUSED_TERMINATION too many rejected lease updates cause the HA service to terminate
Jun 12 15:05:39 dhcp-1 kea-dhcp4[564812]: ERROR [kea-dhcp4.ha-hooks.139625718580992] HA_TERMINATED HA service terminated due to an unrecoverable condition. Check previous error message(s), address the problem and restart!
```
Here are the logs from the standby server:
```
Mar 12 19:25:06 dhcp-2 kea-dhcp4[203037]: WARN [kea-dhcp4.lease-cmds-hooks.139670034884352] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678688706, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 2907, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
Mar 12 19:25:06 dhcp-2 kea-dhcp4[203037]: WARN [kea-dhcp4.lease-cmds-hooks.139670009706240] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678688706, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 2907, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
Mar 12 19:27:28 dhcp-2 kea-dhcp4[203037]: WARN [kea-dhcp4.lease-cmds-hooks.139670009706240] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678688848, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 3812, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
Mar 12 19:32:05 dhcp-2 kea-dhcp4[203037]: WARN [kea-dhcp4.lease-cmds-hooks.139670018098944] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678689125, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 274, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
Mar 12 19:32:34 dhcp-2 kea-dhcp4[203037]: WARN [kea-dhcp4.lease-cmds-hooks.139670009706240] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678689154, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 113, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
Mar 12 19:32:36 dhcp-2 kea-dhcp4[203037]: ERROR [kea-dhcp4.ha-hooks.139670104323840] HA_TERMINATED HA service terminated due to an unrecoverable condition. Check previous error message(s), address the problem and restart!
Mar 12 22:11:09 dhcp-2 kea-dhcp4[203037]: ERROR [kea-dhcp4.packets.139670138794688] DHCP4_BUFFER_RECEIVE_FAIL error on attempt to receive packet: Truncated DHCPv4 packet (len=0) received, at least 236 is expected.
```
The relevant config is the following on both hosts, differing only in the "this-server-name" property.
```
"hooks-libraries": [{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so",
"parameters": {}
},
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_stat_cmds.so",
"parameters": {}
},
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [{
"this-server-name": "standby-dhcp",
"mode": "hot-standby",
"heartbeat-delay": 10000,
"max-response-delay": 60000,
"max-ack-delay": 5000,
"max-unacked-clients": 5,
"peers": [{
"name": "primary-dhcp",
"url": "http://dhcp-1:8001/",
"role": "primary",
"auto-failover": true
}, {
"name": "standby-dhcp",
"url": "http://dhcp-2:8001/",
"role": "standby",
"auto-failover": true
}]
}]
}
}]
```next-stable-2.6