ISC Open Source Projects issueshttps://gitlab.isc.org/groups/isc-projects/-/issues2024-03-28T16:58:13Zhttps://gitlab.isc.org/isc-projects/kea/-/issues/3295DDNS: addresses assigned from an arpa domain that is not configured can halt ...2024-03-28T16:58:13ZDarren AnkneyDDNS: addresses assigned from an arpa domain that is not configured can halt ddns processingKea Version tested: 2.4.0 with DHCPv4. Assumedly this same problem would exist in DHCPv6 but I didn't try that. The BIND version used in the test was 9.18.24, but I don't think it probably matters what version or brand of DNS server is...Kea Version tested: 2.4.0 with DHCPv4. Assumedly this same problem would exist in DHCPv6 but I didn't try that. The BIND version used in the test was 9.18.24, but I don't think it probably matters what version or brand of DNS server is used.
It has been discovered that it is possible that `kea-dhcp-ddns` can enter a state where no ddns updates are issued under certain circumstances. The circumstances required are only an intermittently unavailable DNS server, an address range in Kea that is not in the "reverse-ddns" portion of the DDNS configuration, a high rate of DHCP queries (I tested with 200 per second), and `"ddns-update-on-renew": true` in the `kea-dhcp4` configuration. Below is the test scenario (first with a working version of the ddns configuration):
<details><summary>Kea configuration and command line</summary>
Command: `kea-dhcp4 -c kea-dhcp4-test-ddns.json`
kea-dhcp4-test-ddns.json
```
{
"Dhcp4": {
"interfaces-config": {
"interfaces": [
"ens256"
]
},
"lease-database": {
"type": "memfile",
"persist": false
},
"calculate-tee-times": true,
"valid-lifetime": 7200,
"ddns-generated-prefix": "myhost",
"ddns-qualifying-suffix": "example.org",
"ddns-replace-client-name": "always",
"ddns-send-updates": true,
"ddns-override-client-update": true,
"ddns-override-no-update": true,
"ddns-update-on-renew": true,
"dhcp-ddns": {
"enable-updates": true,
"max-queue-size": 1024,
"ncr-format": "JSON",
"ncr-protocol": "UDP",
"sender-ip": "0.0.0.0",
"sender-port": 0,
"server-ip": "127.0.0.1",
"server-port": 53001
},
"shared-networks": [
{
"name": "my-secret-lair-level-1",
"interface": "ens256",
"subnet4": [
{
"subnet": "10.1.2.0/24",
"id": 1,
"option-data": [
{
"name": "routers",
"data": "10.1.2.1"
}
],
"pools": [
{
"pool": "10.1.2.100-10.1.2.200"
}
]
},
{
"subnet": "172.16.0.0/24",
"id": 2,
"option-data": [
{
"name": "routers",
"data": "172.16.0.1"
}
],
"pools": [
{
"pool": "172.16.0.100-172.16.0.200"
}
]
}
]
}
],
"loggers": [
{
"name": "kea-dhcp4",
"severity": "DEBUG",
"debuglevel": 99,
"output_options": [
{
"output": "stdout"
}
]
}
]
}
}
```
</details>
<details><summary>BIND configuration and command line</summary>
Command: `named -4 -g -c /tmp/named.conf`
named.conf
```
options {
directory "/tmp";
recursion no;
allow-update { any;};
dnssec-validation no;
};
zone "2.1.10.in-addr.arpa" {
type primary;
file "/tmp/2.1.10.in-addr.arpa";
};
zone "0.16.172.in-addr.arpa" {
type primary;
file "/tmp/0.16.172.in-addr.arpa";
};
zone "example.org" {
type primary;
file "/tmp/example.org";
};
```
example.org
```
$ORIGIN .
$TTL 86399 ; 23 hours 59 minutes 59 seconds
example.org IN SOA ns1.example.org. hostmaster.example.org. (
1 ; serial
43200 ; refresh (12 hours)
900 ; retry (15 minutes)
1814400 ; expire (3 weeks)
7200 ; minimum (2 hours)
)
NS ns1.example.org.
ns1.example.org A 192.168.20.114
```
2.1.10.in-addr.arpa
```
$ORIGIN .
$TTL 86400 ; 1 day
2.1.10.in-addr.arpa IN SOA 2.1.10.IN-ADDR.ARPA. . (
1 ; serial
30800 ; refresh (8 hours 33 minutes 20 seconds)
7200 ; retry (2 hours)
604800 ; expire (1 week)
300 ; minimum (5 minutes)
)
NS ns1.example.org.
```
0.16.172.in-addr.arpa
```
$ORIGIN .
$TTL 86400 ; 1 day
0.16.172.in-addr.arpa IN SOA 0.16.172.in-addr.arpa. . (
1 ; serial
30800 ; refresh (8 hours 33 minutes 20 seconds)
7200 ; retry (2 hours)
604800 ; expire (1 week)
300 ; minimum (5 minutes)
)
NS ns1.example.org.
```
</details>
<details><summary>Working DDNS configuration and command line</summary>
Command: `kea-dhcp-ddns -c kea-dhcp-ddns-test-ddns.json`
kea-dhcp-ddns-test-ddns.json
```
{
"DhcpDdns": {
"dns-server-timeout": 40000,
"forward-ddns": {
"ddns-domains": [
{
"dns-servers": [
{
"ip-address": "192.168.20.114",
"port": 53
}
],
"name": "example.org."
}
]
},
"reverse-ddns": {
"ddns-domains": [
{
"dns-servers": [
{
"ip-address": "192.168.20.114",
"port": 53
}
],
"name": "2.1.10.in-addr.arpa."
},
{
"dns-servers": [
{
"ip-address": "192.168.20.114",
"port": 53
}
],
"name": "0.16.172.in-addr.arpa."
}
]
},
"ip-address": "127.0.0.1",
"ncr-format": "JSON",
"ncr-protocol": "UDP",
"port": 53001,
"loggers": [
{
"severity": "DEBUG",
"debuglevel": 99,
"name": "kea-dhcp-ddns",
"output_options": [
{
"output": "stdout"
}
]
}
]
}
}
```
</details>
<details><summary>non-Working DDNS configuration and command line</summary>
Command: `kea-dhcp-ddns -c kea-dhcp-ddns-test-ddns.json`
kea-dhcp-ddns-test-ddns.json
```
{
"DhcpDdns": {
"dns-server-timeout": 40000,
"forward-ddns": {
"ddns-domains": [
{
"dns-servers": [
{
"ip-address": "192.168.20.114",
"port": 53
}
],
"name": "example.org."
}
]
},
"reverse-ddns": {
"ddns-domains": [
{
"dns-servers": [
{
"ip-address": "192.168.20.114",
"port": 53
}
],
"name": "2.1.10.in-addr.arpa."
}
//,
// {
// "dns-servers": [
// {
// "ip-address": "192.168.20.114",
// "port": 53
// }
// ],
// "name": "0.16.172.in-addr.arpa."
// }
]
},
"ip-address": "127.0.0.1",
"ncr-format": "JSON",
"ncr-protocol": "UDP",
"port": 53001,
"loggers": [
{
"severity": "DEBUG",
"debuglevel": 99,
"name": "kea-dhcp-ddns",
"output_options": [
{
"output": "stdout"
}
]
}
]
}
}
```
</details>
Perfdhcp was used to create the traffic for this test: `sudo perfdhcp -4 -r 200 -p 1800 -l ens256 -R 200`
BIND will, for some reason, stop responding intermittently during the test. The reason for that is not important for this issue. This was originally reported by a customer using some kind of off premise DNS servers that would intermittently be unavailable due to network issues. If all subnets are configured in the DDNS configuration, then DDNS will not become unresponsive when BIND becomes unresponsive.
This message might appear while BIND is unresponsive:
```DHCP_DDNS_AT_MAX_TRANSACTIONS application has 1024 queued requests but has reached maximum number of 32 concurrent transactions```
but DDNS will recover once BIND recovers.
Using the "non-Working DDNS configuration and command line", the DDNS server cannot recover and is unavailable for the remainder of the test.
The `kea-dhcp-ddns` service must be restarted before it will respond again.
I also tested this with `example.org` removed from the ddns configuration. `kea-dhcp-ddns` did not suffer a stop in processing with that zone removed. It appears to only be in the case of a missing `.arpa` zone.
<details><summary>When the `kea-dhcp-ddns` stops responding, it is during one of these failures to match reverse DNS zone</summary>
```
2024-03-19 16:56:18.347 WARN [kea-dhcp-ddns.dhcp-to-d2/1479.281473066429376] DHCP_DDNS_NO_MATCH No DNS servers match FQDN 149.0.16.172.in-addr.arpa.
2024-03-19 16:56:18.347 ERROR [kea-dhcp-ddns.dhcp-to-d2/1479.281473066429376] DHCP_DDNS_NO_REV_MATCH_ERROR Request ID 000101285974D2A2411A8BCED2CF77E9E97AD8582814F422CD88FD27E2B37B26969C5F: the configured list of reverse DDNS domains does not contain a match for: Type: 1 (CHG_REMOVE)
Forward Change: yes
Reverse Change: yes
FQDN: [myhost-172-16-0-149.example.org.]
IP Address: [172.16.0.149]
DHCID: [000101285974D2A2411A8BCED2CF77E9E97AD8582814F422CD88FD27E2B37B26969C5F]
Lease Expires On: 20240319173614
Lease Length: 2400
Conflict Resolution: yes
The request has been discarded.
```
No further logs are emitted by `kea-dhcp-ddns` until the process is restarted.
</details>
<details><summary>`kea-dhcp4` does not appear to realize anything is amiss</summary>
```
2024-03-19 16:58:41.932 DEBUG [kea-dhcp4.dhcpsrv/1487.281473627656064] DHCPSRV_QUEUE_NCR [hwtype=1 00:0c:01:02:03:23], cid=[01:00:0c:01:02:03:23]: Name change request to remove DNS entry queued: Type: 1 (CHG_REMOVE)
Forward Change: yes
Reverse Change: yes
FQDN: [myhost-10-1-2-131.example.org.]
IP Address: [10.1.2.131]
DHCID: [000101EC31CD9751563A5FD3586A0940AEDE3871AA5D6D952E92D3D5A21E173B5F146C]
Lease Expires On: 20240319173840
Lease Length: 2400
Conflict Resolution: yes
2024-03-19 16:58:41.932 DEBUG [kea-dhcp4.dhcpsrv/1487.281473669656592] DHCPSRV_DHCP_DDNS_NCR_SENT NameChangeRequest sent to kea-dhcp-ddns: Type: 1 (CHG_REMOVE)
Forward Change: yes
Reverse Change: yes
FQDN: [myhost-10-1-2-131.example.org.]
IP Address: [10.1.2.131]
DHCID: [000101EC31CD9751563A5FD3586A0940AEDE3871AA5D6D952E92D3D5A21E173B5F146C]
Lease Expires On: 20240319173840
Lease Length: 2400
Conflict Resolution: yes
2024-03-19 16:58:41.932 DEBUG [kea-dhcp4.dhcpsrv/1487.281473627656064] DHCPSRV_QUEUE_NCR [hwtype=1 00:0c:01:02:03:23], cid=[01:00:0c:01:02:03:23]: Name change request to add DNS entry queued: Type: 0 (CHG_ADD)
Forward Change: yes
Reverse Change: yes
FQDN: [myhost-10-1-2-131.example.org.]
IP Address: [10.1.2.131]
DHCID: [000101EC31CD9751563A5FD3586A0940AEDE3871AA5D6D952E92D3D5A21E173B5F146C]
Lease Expires On: 20240319173841
Lease Length: 2400
Conflict Resolution: yes
2024-03-19 16:58:41.932 DEBUG [kea-dhcp4.dhcpsrv/1487.281473669656592] DHCPSRV_DHCP_DDNS_NCR_SENT NameChangeRequest sent to kea-dhcp-ddns: Type: 0 (CHG_ADD)
Forward Change: yes
Reverse Change: yes
FQDN: [myhost-10-1-2-131.example.org.]
IP Address: [10.1.2.131]
DHCID: [000101EC31CD9751563A5FD3586A0940AEDE3871AA5D6D952E92D3D5A21E173B5F146C]
Lease Expires On: 20240319173841
Lease Length: 2400
Conflict Resolution: yes
```
as the log messages appear the same whether `kea-dhcp-ddns` is doing anything or not.
</details>
[SF1804](https://isc.lightning.force.com/lightning/r/Case/500S60000074PjmIAE/view)kea2.5.8https://gitlab.isc.org/isc-projects/kea/-/issues/3294Deletion of v6 reservation over API by address causes all v6 reservations to ...2024-03-22T09:52:31ZAndrew MulheirnDeletion of v6 reservation over API by address causes all v6 reservations to disappear---
name: Deletion of v6 reservation over API by address causes all v6 reservations to disappear
about: Kea 2.4.1
---
**Describe the bug**
I am using the API to add and remove reservations for v4 and v6.
I find that removing a v4 re...---
name: Deletion of v6 reservation over API by address causes all v6 reservations to disappear
about: Kea 2.4.1
---
**Describe the bug**
I am using the API to add and remove reservations for v4 and v6.
I find that removing a v4 reservation by IP address works as expected, but removing a v6 reservation results in all reservations being deleted.
**To Reproduce**
Steps to reproduce the behavior:
1. Create two v6 reservations in the database
2. send a 'reservation-del' to the dhcp6 process via the API to remove one address
3. run a 'reservation-get-all' and receive the message ```"text": "0 IPv6 host(s) found."```
4. Perform the same sequence with v4 and it succeeds
5. Check the postgres database contents and see that the v6 reservations table is empty
Note that if I do a deletion specifying the flex-id or DUID of a reservation, only that single reservation is removed.
**Expected behavior**
I am expecting that a single IPv6 reservation is deleted.
**Environment:**
- Kea version: 2.4.1
- OS: Ubuntu 22.04.4 LTS
- Installed from kea packages on cloudsmith
- Flex-id, HA, host commands and lease commands are in use.
**Additional Information**
reservation-get-all result over api:
```
[
{
"arguments": {
"hosts": [
{
"client-classes": [],
"flex-id": "67622D6C6F6E39382D7273773030313A78652D302F302F33",
"hostname": "123123.vorboss.lab",
"ip-addresses": [
"2a00:e340:1102::3"
],
"option-data": [],
"prefixes": [
"2a00:e340:1103:300::/56"
],
"subnet-id": 1
},
{
"client-classes": [],
"flex-id": "67622D6C6F6E39382D7273773030313A78652D302F302F34",
"hostname": "123124.vorboss.lab",
"ip-addresses": [
"2a00:e340:1102::4"
],
"option-data": [],
"prefixes": [
"2a00:e340:1103:400::/56"
],
"subnet-id": 1
}
]
},
"result": 0,
"text": "2 IPv6 host(s) found."
}
]
```
Deletion sent:
```
{
"command": "reservation-del",
"service": ["dhcp6"],
"arguments": {
"subnet-id": 1,
"ip-address": "2a00:e340:1102::4"
}
}
```
Debug from kea-ctrl-agent:
```
INFO COMMAND_RECEIVED Received command 'reservation-del'
DEBUG HOOKS_CALLOUTS_BEGIN begin all callouts for hook $reservation_del
INFO HOST_CMDS_RESERV_DEL reservation-del command called (parameters: { "ip-address": "2a00:e340:1102::4", "subnet-id": 1 })
INFO HOST_CMDS_RESERV_DEL_SUCCESS reservation-del command success (parameters: { "ip-address": "2a00:e340:1102::4", "subnet-id": 1 })
DEBUG HOOKS_CALLOUT_CALLED hooks library with index 2 has called a callout on hook $reservation_del that has address 0x7ff2f67cecb0 (callout duration: 3.743 ms)
DEBUG HOOKS_CALLOUTS_COMPLETE completed callouts for hook $reservation_del (total callouts duration: 3.743 ms)
```
Subsequent reservation-get-all:
```
[
{
"arguments": {
"hosts": []
},
"result": 3,
"text": "0 IPv6 host(s) found."
}
]
```
**Contacting you**
My work email is andrew.mulheirn@vorboss.comkea2.5.8https://gitlab.isc.org/isc-projects/kea/-/issues/3289DHCPv4: bad option 81 data (invalid FQDN) causes halt in further processing o...2024-03-14T15:00:06ZDarren AnkneyDHCPv4: bad option 81 data (invalid FQDN) causes halt in further processing of packetA packet with option 81 attached with an empty label causes further processing of the client's DHCPv4 packet to cease and the packet to be dropped.
This is very simple to reproduce with the following
<details><summary>Simple configurat...A packet with option 81 attached with an empty label causes further processing of the client's DHCPv4 packet to cease and the packet to be dropped.
This is very simple to reproduce with the following
<details><summary>Simple configuration</summary>
```
{
"Dhcp4": {
"interfaces-config": {
"interfaces": [
"ens256"
]
},
"lease-database": {
"type": "memfile",
"persist": false
},
"calculate-tee-times": true,
"option-data": [
{
"name": "domain-name-servers",
"data": "10.0.0.1"
}
],
"subnet4": [
{
"subnet": "10.1.2.0/24",
"id": 1,
"option-data": [
{
"name": "routers",
"data": "10.1.2.1"
}
],
"pools": [
{
"pool": "10.1.2.100-10.1.2.200"
}
]
}
],
"loggers": [
{
"name": "kea-dhcp4",
"severity": "DEBUG",
"debuglevel": 99,
"output_options": [
{
"output": "stdout"
}
]
}
]
}
}
```
</details>
and sending packets with malformed FQDN using `perfdhcp`:
```
perfdhcp -4 -r 1 -p 10 -l ens256 -R 1 -o 81,0100002E656D7074792E6C6162656C2E6578616D706C652E636F6D
```
<details><summary>Messages like this are logged</summary>
```
2024-03-13 11:21:28.124 DEBUG [kea-dhcp4.packets/52340.281473684041744] DHCP4_BUFFER_RECEIVED received buffer from 10.1.2.6:67 to 255.255.255.255:67 over interface ens256
2024-03-13 11:21:28.124 DEBUG [kea-dhcp4.options/52340.281473642041216] DHCP4_BUFFER_UNPACK parsing buffer received from 10.1.2.6 to 255.255.255.255 over interface ens256
2024-03-13 11:21:28.124 DEBUG [kea-dhcp4.bad-packets/52340.281473642041216] DHCP4_PACKET_DROP_0001 failed to parse packet from 10.1.2.6 to 255.255.255.255, received over interface ens256, reason: failed to parse the domain-name in DHCPv4 Client FQDN Option: non terminating empty label in .empty.label.example.com, hwaddr=00:0c:01:02:03:04
```
</details>
Clients with such incorrect FQDNs in option 81 are not able to get an IP address. Option 81 content from such clients is probably not useable and should perhaps be ignored but the client should still get an IP address possibly? This type of error in option 81 was allowed in ISC DHCP and so this is a problem for those migrating to Kea from ISC DHCP.
Attached a pcap of the DHCP packets generated by `perfdhcp`: [fqdn-test.pcap](/uploads/810d5aa88d78f58f1c4b39d6b1eec3d1/fqdn-test.pcap)
[SF1790](https://isc.lightning.force.com/lightning/r/Case/500S6000006lxqtIAA/view)kea2.5.8https://gitlab.isc.org/isc-projects/kea/-/issues/3287enable pylint and pycodestyle on all python files in kea repo2024-03-14T14:57:35ZWlodzimierz Wencelenable pylint and pycodestyle on all python files in kea repoextend kea pipeline with similar solution to what we are using in qa repo.extend kea pipeline with similar solution to what we are using in qa repo.kea2.5.8https://gitlab.isc.org/isc-projects/kea/-/issues/3283CableLabs option (RFC3495): Inconsistent "option_def" usage in client class d...2024-03-28T17:32:26ZPeter DaviesCableLabs option (RFC3495): Inconsistent "option_def" usage in client class definitionsInconsistent "option_def" usage in client class definitions:
Using Kea 2.5.6 - I have yet to try with 2.4.1.
The following "option-def" statement is accepted in the global scope of a
configuration file:
```
...
"opt...Inconsistent "option_def" usage in client class definitions:
Using Kea 2.5.6 - I have yet to try with 2.4.1.
The following "option-def" statement is accepted in the global scope of a
configuration file:
```
...
"option-def": [ {
"name": "Option_122", "code": 122, "type": "empty", "encapsulate": "Option_122_space" }, {
"name": "Option_122_1", "code": 1, "space": "Option_122_Space", "type": "ipv4-address" }, {
"name": "Option_122_2", "code": 2, "space": "Option_122_Space", "type": "ipv4-address" }
],
...
```
However, if this "option-def" is defined within a client class as:
```
...
"client-classes": [{
"name": "Docsis_Class", "test": "substring(option[60].hex,0,6) == 'docsis'",
"option-def": [ {
"name": "Option_122", "code": 122, "type": "empty", "encapsulate": "Option_122_space" }, {
"name": "Option_122_1", "code": 1, "space": "Option_122_Space", "type": "ipv4-address" }, {
"name": "Option_122_2", "code": 2, "space": "Option_122_Space", "type": "ipv4-address" } ],
"option-data": [ {
"name": "Option_122_1", "data": "10.0.0.68", "code": 1, "space": "Option_122_Space" }, {
"name": "Option_122_2", "data": "10.0.0.69", "code": 2, "space": "Option_122_Space" } ]
}],
...
```
The following message is generated, and Kea exits:
```
Error encountered: Not allowed option definition for code '122' in space 'dhcp4'
```
If the "Option_122" "option-def" is changed to private option "224" as:
```
...
"client-classes": [{
"name": "Docsis_Class", "test": "substring(option[60].hex,0,6) == 'docsis'",
"option-def": [ {
"name": "Option_122", "code": 224, "type": "empty", "encapsulate": "Option_122_space" }, {
"name": "Option_122_1", "code": 1, "space": "Option_122_Space", "type": "ipv4-address" }, {
"name": "Option_122_2", "code": 2, "space": "Option_122_Space", "type": "ipv4-address" } ],
"option-data": [ {
"name": "Option_122_1", "data": "10.0.0.68", "code": 1, "space": "Option_122_Space" }, {
"name": "Option_122_2", "data": "10.0.0.69", "code": 2, "space": "Option_122_Space" } ]
}],
...
```
Then the following error is generated, and Kea exits:
```
Error encountered: Not allowed option definition for code '1' in space 'Option_122_Space'
```
[SF00001775](https://isc.lightning.force.com/lightning/r/Case/500S6000006TLtj/view)kea2.5.8https://gitlab.isc.org/isc-projects/kea/-/issues/3281Follow-up from "Draft: Resolve "heap-use-after-free and invalid vptr on PingC...2024-03-29T08:36:56ZRazvan BecheriuFollow-up from "Draft: Resolve "heap-use-after-free and invalid vptr on PingCheckMgr destruction""The following discussion from !2197 should be addressed:
- [ ] @andrei started a [discussion](https://gitlab.isc.org/isc-projects/kea/-/merge_requests/2197#note_438320 'Draft: Resolve "heap-use-after-free and invalid vptr on PingCheckMg...The following discussion from !2197 should be addressed:
- [ ] @andrei started a [discussion](https://gitlab.isc.org/isc-projects/kea/-/merge_requests/2197#note_438320 'Draft: Resolve "heap-use-after-free and invalid vptr on PingCheckMgr destruction"'): (+3 comments)
> > To keep the members alive, they can be added to a lambda function which uses a smart pointer to capture the object, but does not use it. It then must be added to the IOService queue using the post function.
>
> I would take the `shared_from_this` alternative anytime if it gets rid of the posts.
>
> If you think that it is too much work for now although it shouldn't be, we can create a ticket, but can you at least add comments to say that they are posted only for extending lifetime?
>
> Core:
>
> ```plaintext
> + getIOService()->post(std::bind(f, queue_mgr_));
> + io_service_->post(std::bind(f, timer_));
> + io_service_->post(std::bind(f, timer_));
> + io_service_->post(std::bind(f, timer_));
> + io_service_->post(std::bind(f, timer_, tcp_socket_, tls_socket_));
> ```
>
> Premium:
>
> ```plaintext
> + main_io_service_->post(std::bind(f, expiration_timer_, channel_));
> ```kea2.5.8https://gitlab.isc.org/isc-projects/kea/-/issues/3278Perfmon-Hook-Task-4 Implement PerfMonMgr Basics - start up, configuration2024-03-26T19:39:49ZThomas MarkwalderPerfmon-Hook-Task-4 Implement PerfMonMgr Basics - start up, configurationComplete Hook Task 4: Implement PerfMonMgr Basics - start up, configuration.
This creates the initial PerfMonMgr class with stub functions. It should be able to parse configuration but not yet provide data processing.
See https://gitla...Complete Hook Task 4: Implement PerfMonMgr Basics - start up, configuration.
This creates the initial PerfMonMgr class with stub functions. It should be able to parse configuration but not yet provide data processing.
See https://gitlab.isc.org/isc-projects/kea/-/wikis/Designs/performance-monitor#perfmon-hook-taskskea2.5.8Thomas MarkwalderThomas Markwalderhttps://gitlab.isc.org/isc-projects/kea/-/issues/3276Kea primary server in "passive backup" freeze/crash on receiving ha-sync2024-03-28T10:34:28ZMarcin GodzinaKea primary server in "passive backup" freeze/crash on receiving ha-syncKea HA server set as `primary` freezes after receiving `ha-sync` command with proper arguments.
The backup server does NOT crash.
Freeze occurs only in `passive-backup` mode.
The problem exists in both v4 and v6. Also, in Memfile and m...Kea HA server set as `primary` freezes after receiving `ha-sync` command with proper arguments.
The backup server does NOT crash.
Freeze occurs only in `passive-backup` mode.
The problem exists in both v4 and v6. Also, in Memfile and mysql/psql lease database.
**Kea versions tested:**
- 2.5.7-git 8c1f22e3fb65225a0279606a8a65962850a5f881
- 2.4.0 release tarball
**Tested systems:**
- Fedora 38 in VM on my local setup.
- Ubuntu 22.04, Alpine 3.16, Fedora 36 on Jenkins build farm.
**To Reproduce**
Steps to reproduce the behavior:
1. Run Kea HA servers in **Passive backup** configuration (tested configuration provided)
2. Wait for servers to connect.
3. Optionally add leases (crashes either way)
4. Send the `ha-sync` command with proper arguments to the primary server. (`"server-name": "server2"` for provided configuration) (Invalid arguments respond with error)
The primary server freezes after receiving a response to the `dhcp-disable` command, sent automatically to the backup server. It does not respond to kea-ctrl agent, keyboard interrupts or SIGHUP
<details><summary>Commands tested to freeze provided config:</summary>
```
{
command": "ha-sync",
"arguments": {
"server-name": "server2"
}
}
```
```
{
command": "ha-sync",
"arguments": {
"server-name": "server1"
}
}
```
```
{
command": "ha-sync",
"arguments": {
"server-name": "server2",
"max-period": 60
}
}
```
</details>
**Configuration**
<details><summary>Primary</summary>
```
{
"Dhcp4": {
"option-data": [],
"hooks-libraries": [
{
"library": "/home/mgodzina/installed/keadev/lib/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [
{
"peers": [
{
"auto-failover": true,
"name": "server1",
"role": "primary",
"url": "http://192.168.56.102:8003/"
},
{
"auto-failover": true,
"name": "server2",
"role": "backup",
"url": "http://192.168.56.103:8003/"
}
],
"state-machine": {
"states": []
},
"mode": "passive-backup",
"this-server-name": "server1",
"multi-threading": {
"enable-multi-threading": true,
"http-dedicated-listener": true,
"http-listener-threads": 0,
"http-client-threads": 0
}
}
]
}
},
{
"library": "/home/mgodzina/installed/keadev/lib/kea/hooks/libdhcp_lease_cmds.so"
}
],
"shared-networks": [],
"subnet4": [
{
"subnet": "192.168.50.0/24",
"pools": [
{
"pool": "192.168.50.1-192.168.50.200"
}
],
"interface": "enp0s9"
}
],
"interfaces-config": {
"interfaces": [
"enp0s9"
]
},
"control-socket": {
"socket-type": "unix",
"socket-name": "/home/mgodzina/installed/keadev/var/run/kea/control_socket"
},
"renew-timer": 1000,
"rebind-timer": 2000,
"valid-lifetime": 4000,
"loggers": [
{
"name": "kea-dhcp4",
"output-options": [
{
"output": "/home/mgodzina/installed/keadev/var/log/kea.log"
}
],
"severity": "DEBUG",
"debuglevel": 99
}
],
"lease-database": {
"type": "memfile"
}
}
}
```
</details>
<details><summary>Backup</summary>
```
{
"Dhcp4": {
"option-data": [],
"hooks-libraries": [
{
"library": "/home/mgodzina/installed/keadev/lib/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [
{
"peers": [
{
"auto-failover": true,
"name": "server1",
"role": "primary",
"url": "http://192.168.56.102:8003/"
},
{
"auto-failover": true,
"name": "server2",
"role": "backup",
"url": "http://192.168.56.103:8003/"
}
],
"state-machine": {
"states": []
},
"mode": "passive-backup",
"this-server-name": "server2",
"multi-threading": {
"enable-multi-threading": true,
"http-dedicated-listener": true,
"http-listener-threads": 0,
"http-client-threads": 0
}
}
]
}
},
{
"library": "/home/mgodzina/installed/keadev/lib/kea/hooks/libdhcp_lease_cmds.so"
}
],
"shared-networks": [],
"subnet4": [
{
"subnet": "192.168.50.0/24",
"pools": [
{
"pool": "192.168.50.1-192.168.50.200"
}
],
"interface": "enp0s9"
}
],
"interfaces-config": {
"interfaces": [
"enp0s9"
]
},
"control-socket": {
"socket-type": "unix",
"socket-name": "/home/mgodzina/installed/keadev/var/run/kea/control_socket"
},
"renew-timer": 1000,
"rebind-timer": 2000,
"valid-lifetime": 4000,
"loggers": [
{
"name": "kea-dhcp4",
"output-options": [
{
"output": "/home/mgodzina/installed/keadev/var/log/kea.log"
}
],
"severity": "DEBUG",
"debuglevel": 99
}
],
"lease-database": {
"type": "memfile"
}
}
}
```
</details>
**Logs**
<details><summary>Primary server log tail</summary>
```
2024-02-28 16:20:13.417 DEBUG [kea-dhcp4.commands/2096.139741364354944] COMMAND_SOCKET_CONNECTION_OPENED Opened socket 38 for incoming command connection
2024-02-28 16:20:13.417 DEBUG [kea-dhcp4.commands/2096.139741364354944] COMMAND_SOCKET_READ Received 127 bytes over command socket 38
2024-02-28 16:20:13.417 INFO [kea-dhcp4.commands/2096.139741364354944] COMMAND_RECEIVED Received command 'ha-sync'
2024-02-28 16:20:13.417 DEBUG [kea-dhcp4.callouts/2096.139741364354944] HOOKS_CALLOUTS_BEGIN begin all callouts for hook $ha_sync
2024-02-28 16:20:13.417 DEBUG [kea-dhcp4.http/2096.139741364354944] HTTP_CLIENT_REQUEST_SEND sending HTTP request POST / HTTP/1.1 to http://192.168.56.103:8003/
2024-02-28 16:20:13.417 DEBUG [kea-dhcp4.http/2096.139741364354944] HTTP_CLIENT_REQUEST_SEND_DETAILS detailed information about request sent to http://192.168.56.103:8003/:
POST / HTTP/1.1
Host: 192.168.56.103
Content-Length: 86
Content-Type: application/json
{ "arguments": { "origin": 2000 }, "command": "dhcp-disable", "service": [ "dhcp4" ] }
2024-02-28 16:20:13.417 INFO [kea-dhcp4.ha-hooks/2096.139741364354944] HA_SYNC_START server1: starting lease database synchronization with server2
2024-02-28 16:20:13.417 DEBUG [kea-dhcp4.http/2096.139741364354944] HTTP_SERVER_RESPONSE_RECEIVED received HTTP response from http://192.168.56.103:8003/
2024-02-28 16:20:13.417 DEBUG [kea-dhcp4.http/2096.139741364354944] HTTP_SERVER_RESPONSE_RECEIVED_DETAILS detailed information about well-formed response received from http://192.168.56.103:8003/:
HTTP/1.1 200 OK
Content-Length: 54
Content-Type: application/json
Date: Wed, 28 Feb 2024 15:20:13 GMT
[ { "result": 0, "text": "DHCPv4 service disabled" } ]
```
</details>
<details><summary>Backup server log snippet with timeout:</summary>
```
2024-02-28 16:20:13.413 DEBUG [kea-dhcp4.http/20519.140151306917568] HTTP_REQUEST_RECEIVE_START start receiving request from 192.168.56.102 with timeout 10
2024-02-28 16:20:13.413 DEBUG [kea-dhcp4.http/20519.140151306917568] HTTP_DATA_RECEIVED received 179 bytes from 192.168.56.102
2024-02-28 16:20:13.413 DEBUG [kea-dhcp4.http/20519.140151306917568] HTTP_CLIENT_REQUEST_RECEIVED received HTTP request from 192.168.56.102
2024-02-28 16:20:13.413 DEBUG [kea-dhcp4.http/20519.140151306917568] HTTP_CLIENT_REQUEST_RECEIVED_DETAILS detailed information about well-formed request received from 192.168.56.102:
POST / HTTP/1.1
Host: 192.168.56.103
Content-Length: 86
Content-Type: application/json
{ "arguments": { "origin": 2000 }, "command": "dhcp-disable", "service": [ "dhcp4" ] }
2024-02-28 16:20:13.413 INFO [kea-dhcp4.commands/20519.140151306917568] COMMAND_RECEIVED Received command 'dhcp-disable'
2024-02-28 16:20:13.413 DEBUG [kea-dhcp4.callouts/20519.140151306917568] HOOKS_CALLOUTS_BEGIN begin all callouts for hook command_processed
2024-02-28 16:20:13.413 DEBUG [kea-dhcp4.callouts/20519.140151306917568] HOOKS_CALLOUT_CALLED hooks library with index 1 has called a callout on hook command_processed that has address 0x7f778767ffe0 (callout duration: 0.000 ms)
2024-02-28 16:20:13.413 DEBUG [kea-dhcp4.callouts/20519.140151306917568] HOOKS_CALLOUTS_COMPLETE completed callouts for hook command_processed (total callouts duration: 0.000 ms)
2024-02-28 16:20:13.413 DEBUG [kea-dhcp4.http/20519.140151306917568] HTTP_SERVER_RESPONSE_SEND sending HTTP response HTTP/1.1 200 OK to 192.168.56.102
2024-02-28 16:20:13.413 DEBUG [kea-dhcp4.http/20519.140151306917568] HTTP_SERVER_RESPONSE_SEND_DETAILS detailed information about response sent to 192.168.56.102:
HTTP/1.1 200 OK
Content-Length: 54
Content-Type: application/json
Date: Wed, 28 Feb 2024 15:20:13 GMT
[ { "result": 0, "text": "DHCPv4 service disabled" } ]
2024-02-28 16:20:17.831 DEBUG [kea-dhcp4.dhcpsrv/20519.140151383601024] DHCPSRV_TIMERMGR_RUN_TIMER_OPERATION running operation for timer: reclaim-expired-leases
2024-02-28 16:20:17.831 DEBUG [kea-dhcp4.alloc-engine/20519.140151383601024] ALLOC_ENGINE_V4_LEASES_RECLAMATION_START starting reclamation of expired leases (limit = 100 leases or 250 milliseconds)
2024-02-28 16:20:17.831 DEBUG [kea-dhcp4.dhcpsrv/20519.140151383601024] DHCPSRV_MEMFILE_GET_EXPIRED4 obtaining maximum 101 of expired IPv4 leases
2024-02-28 16:20:17.832 DEBUG [kea-dhcp4.alloc-engine/20519.140151383601024] ALLOC_ENGINE_V4_LEASES_RECLAMATION_COMPLETE reclaimed 0 leases in 0.033 ms
2024-02-28 16:20:17.832 DEBUG [kea-dhcp4.alloc-engine/20519.140151383601024] ALLOC_ENGINE_V4_NO_MORE_EXPIRED_LEASES all expired leases have been reclaimed
2024-02-28 16:20:17.832 DEBUG [kea-dhcp4.dhcpsrv/20519.140151383601024] DHCPSRV_TIMERMGR_START_TIMER starting timer: reclaim-expired-leases
2024-02-28 16:20:21.840 DEBUG [kea-dhcp4.dhcpsrv/20519.140151383601024] DHCPSRV_TIMERMGR_RUN_TIMER_OPERATION running operation for timer: flush-reclaimed-leases
2024-02-28 16:20:21.840 DEBUG [kea-dhcp4.alloc-engine/20519.140151383601024] ALLOC_ENGINE_V4_RECLAIMED_LEASES_DELETE begin deletion of reclaimed leases expired more than 3600 seconds ago
2024-02-28 16:20:21.840 DEBUG [kea-dhcp4.dhcpsrv/20519.140151383601024] DHCPSRV_MEMFILE_DELETE_EXPIRED_RECLAIMED4 deleting reclaimed IPv4 leases that expired more than 3600 seconds ago
2024-02-28 16:20:21.840 DEBUG [kea-dhcp4.alloc-engine/20519.140151383601024] ALLOC_ENGINE_V4_RECLAIMED_LEASES_DELETE_COMPLETE successfully deleted 0 expired-reclaimed leases
2024-02-28 16:20:21.840 DEBUG [kea-dhcp4.dhcpsrv/20519.140151383601024] DHCPSRV_TIMERMGR_START_TIMER starting timer: flush-reclaimed-leases
2024-02-28 16:20:27.852 DEBUG [kea-dhcp4.dhcpsrv/20519.140151383601024] DHCPSRV_TIMERMGR_RUN_TIMER_OPERATION running operation for timer: reclaim-expired-leases
2024-02-28 16:20:27.852 DEBUG [kea-dhcp4.alloc-engine/20519.140151383601024] ALLOC_ENGINE_V4_LEASES_RECLAMATION_START starting reclamation of expired leases (limit = 100 leases or 250 milliseconds)
2024-02-28 16:20:27.852 DEBUG [kea-dhcp4.dhcpsrv/20519.140151383601024] DHCPSRV_MEMFILE_GET_EXPIRED4 obtaining maximum 101 of expired IPv4 leases
2024-02-28 16:20:27.852 DEBUG [kea-dhcp4.alloc-engine/20519.140151383601024] ALLOC_ENGINE_V4_LEASES_RECLAMATION_COMPLETE reclaimed 0 leases in 0.032 ms
2024-02-28 16:20:27.852 DEBUG [kea-dhcp4.alloc-engine/20519.140151383601024] ALLOC_ENGINE_V4_NO_MORE_EXPIRED_LEASES all expired leases have been reclaimed
2024-02-28 16:20:27.852 DEBUG [kea-dhcp4.dhcpsrv/20519.140151383601024] DHCPSRV_TIMERMGR_START_TIMER starting timer: reclaim-expired-leases
2024-02-28 16:20:37.891 DEBUG [kea-dhcp4.dhcpsrv/20519.140151383601024] DHCPSRV_TIMERMGR_RUN_TIMER_OPERATION running operation for timer: reclaim-expired-leases
2024-02-28 16:20:37.892 DEBUG [kea-dhcp4.alloc-engine/20519.140151383601024] ALLOC_ENGINE_V4_LEASES_RECLAMATION_START starting reclamation of expired leases (limit = 100 leases or 250 milliseconds)
2024-02-28 16:20:37.892 DEBUG [kea-dhcp4.dhcpsrv/20519.140151383601024] DHCPSRV_MEMFILE_GET_EXPIRED4 obtaining maximum 101 of expired IPv4 leases
2024-02-28 16:20:37.892 DEBUG [kea-dhcp4.alloc-engine/20519.140151383601024] ALLOC_ENGINE_V4_LEASES_RECLAMATION_COMPLETE reclaimed 0 leases in 0.027 ms
2024-02-28 16:20:37.892 DEBUG [kea-dhcp4.alloc-engine/20519.140151383601024] ALLOC_ENGINE_V4_NO_MORE_EXPIRED_LEASES all expired leases have been reclaimed
2024-02-28 16:20:37.892 DEBUG [kea-dhcp4.dhcpsrv/20519.140151383601024] DHCPSRV_TIMERMGR_START_TIMER starting timer: reclaim-expired-leases
2024-02-28 16:20:43.433 DEBUG [kea-dhcp4.http/20519.140151315310272] HTTP_IDLE_CONNECTION_TIMEOUT_OCCURRED closing persistent connection with 192.168.56.102 as a result of a timeout
2024-02-28 16:20:43.433 DEBUG [kea-dhcp4.http/20519.140151315310272] HTTP_CONNECTION_STOP stopping HTTP connection from 192.168.56.102
```
</details>
[gdb.txt](/uploads/de79e56462885f7947eab90267f7a658/gdb.txt)kea2.5.8Marcin SiodelskiMarcin Siodelskihttps://gitlab.isc.org/isc-projects/kea/-/issues/3254Include git commit hash for premium in the config report2024-03-22T16:21:17ZAndrei Pavelandrei@isc.orgInclude git commit hash for premium in the config reportThe report includes the git commit hash for core in the config report that can be accessed with the `-W` parameter when Kea was built from sources. It would be nice to have the git commit hash for premium as well, if Kea was built with p...The report includes the git commit hash for core in the config report that can be accessed with the `-W` parameter when Kea was built from sources. It would be nice to have the git commit hash for premium as well, if Kea was built with premium. It could be mentioned under `Extended version:` or under `Premium hooks: yes`.kea2.5.8https://gitlab.isc.org/isc-projects/kea/-/issues/3250Unattended terminated state and a reboot2024-03-27T21:53:51ZMarcin SiodelskiUnattended terminated state and a rebootConsider the following case. The clocks on two HA-enabled servers diverge and the clock skew eventually exceeds 60 seconds. As a result, both servers transition to the terminated state. In this state, the servers continue serving DHCP cl...Consider the following case. The clocks on two HA-enabled servers diverge and the clock skew eventually exceeds 60 seconds. As a result, both servers transition to the terminated state. In this state, the servers continue serving DHCP clients but do not exchange the lease updates nor heartbeats. An administrator neglects to correct the clocks and one of the servers reboots. The server enters the `waiting` state and remains in this state hoping that the other server is restarted so they can continue the lease database synchronization and start normal operation. However, the server is unaware that its reboot was not triggered in the course of fixing the clocks, so it will wait for the partner endlessly (or until the administrator comes to work in the morning). The waiting server is not responding to the DHCP traffic until then.
This situation should not occur in a setup where NTP has been enabled. It also should not occur if there is a proper monitoring to detect that the clocks diverge early enough. However, there are chances this situation may happen when all of this is neglected.
The proposed solution is to apply a timeout (could even be several to 10 minutes long) for a server in the waiting state. If the transition of its partner does not occur until this timeout elapses, the server in the waiting state transitions back to the terminated state and continues serving the clients. The waiting server MUST NOT transition to the waiting state immediately after it detects that its partner is in the terminated state to allow enough time to the administrator to reboot the server sequentially after correcting the clocks.
[SF1598](https://isc.lightning.force.com/lightning/r/Case/500S6000003jBs3IAE/view)kea2.5.8https://gitlab.isc.org/isc-projects/kea/-/issues/3247changes in hammer for rocky linux2024-03-28T08:11:54ZMarcin Godzinachanges in hammer for rocky linuxWe need to extend Hammer to build on Rocky Linux 9. We have a business need for this.We need to extend Hammer to build on Rocky Linux 9. We have a business need for this.kea2.5.8Marcin GodzinaMarcin Godzinahttps://gitlab.isc.org/isc-projects/kea/-/issues/3246DHCPRELEASE and lease expiration in active-standby HA setup2024-03-27T12:53:48ZPeter DaviesDHCPRELEASE and lease expiration in active-standby HA setupDHCPvRELEASE lease expiration in active-standby HA setup
Kea 2.5.5
When a client sends a DHCPRELEASE message to a Kea primary HA server, the expired
lease processing settings are honoured.
However, the primary updates the...DHCPvRELEASE lease expiration in active-standby HA setup
Kea 2.5.5
When a client sends a DHCPRELEASE message to a Kea primary HA server, the expired
lease processing settings are honoured.
However, the primary updates the failover server with instructions to delete the
lease.
This leads to a divergence of lease data between the two servers.
[SF00001636](https://isc.lightning.force.com/lightning/r/Case/500S6000004XPRy/view)kea2.5.8https://gitlab.isc.org/isc-projects/kea/-/issues/3239New Global Counter assigned-addresses2024-03-27T12:53:11ZPeter DaviesNew Global Counter assigned-addressesFeature Request: New Global Counter assigned-addresses
Statistics return per subnet counters "assigned-addresses" and "cumulative-assigned-addresses".
However, globally, only the cumulative-assigned-addresses counter is retu...Feature Request: New Global Counter assigned-addresses
Statistics return per subnet counters "assigned-addresses" and "cumulative-assigned-addresses".
However, globally, only the cumulative-assigned-addresses counter is returned.
It may interest administrators to know the total number of assigned addresses per
server.
[SF00001629](https://isc.lightning.force.com/lightning/r/Case/500S6000004QXmC/view)kea2.5.8https://gitlab.isc.org/isc-projects/kea/-/issues/3196Thread sanitizer warnings with std::bind2024-03-22T13:39:27ZMarcin SiodelskiThread sanitizer warnings with std::bindI ran a DHCPv4 server compiled with a thread sanitizer, and I got the following sporadic failure:
```
==================
WARNING: ThreadSanitizer: data race (pid=95142)
Write of size 8 at 0x7b0c0000c220 by main thread:
#0 operator ...I ran a DHCPv4 server compiled with a thread sanitizer, and I got the following sporadic failure:
```
==================
WARNING: ThreadSanitizer: data race (pid=95142)
Write of size 8 at 0x7b0c0000c220 by main thread:
#0 operator delete(void*, unsigned long) ../../../../src/libsanitizer/tsan/tsan_new_delete.cpp:150 (libtsan.so.0+0x8e878)
#1 std::_Function_base::_Base_manager<std::_Bind<void (isc::dhcp::Dhcpv4Srv::*(isc::dhcp::Dhcpv4Srv*, boost::shared_ptr<isc::dhcp::Pkt4>))(boost::shared_ptr<isc::dhcp::Pkt4>&)> >::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) /usr/include/c++/11/bits/std_function.h:175 (kea-dhcp4+0xbe1ab)
#2 std::_Function_base::_Base_manager<std::_Bind<void (isc::dhcp::Dhcpv4Srv::*(isc::dhcp::Dhcpv4Srv*, boost::shared_ptr<isc::dhcp::Pkt4>))(boost::shared_ptr<isc::dhcp::Pkt4>&)> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /usr/include/c++/11/bits/std_function.h:203 (kea-dhcp4+0xbe1ab)
#3 std::_Function_handler<void (), std::_Bind<void (isc::dhcp::Dhcpv4Srv::*(isc::dhcp::Dhcpv4Srv*, boost::shared_ptr<isc::dhcp::Pkt4>))(boost::shared_ptr<isc::dhcp::Pkt4>&)> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /usr/include/c++/11/bits/std_function.h:282 (kea-dhcp4+0xbe1ab)
#4 std::_Function_base::~_Function_base() /usr/include/c++/11/bits/std_function.h:244 (kea-dhcp4+0xbccd2)
#5 std::function<void ()>::~function() /usr/include/c++/11/bits/std_function.h:334 (kea-dhcp4+0xbccd2)
#6 boost::detail::sp_ms_deleter<std::function<void ()> >::destroy() /usr/include/boost/smart_ptr/make_shared_object.hpp:59 (kea-dhcp4+0xbccd2)
#7 boost::detail::sp_ms_deleter<std::function<void ()> >::operator()(std::function<void ()>*) /usr/include/boost/smart_ptr/make_shared_object.hpp:93 (kea-dhcp4+0xbccd2)
#8 boost::detail::sp_counted_impl_pd<std::function<void ()>*, boost::detail::sp_ms_deleter<std::function<void ()> > >::dispose() /usr/include/boost/smart_ptr/detail/sp_counted_impl.hpp:169 (kea-dhcp4+0xbccd2)
#9 boost::detail::sp_counted_base::release() /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_atomic.hpp:120 (kea-dhcp4+0x5420a)
#10 boost::detail::shared_count::~shared_count() /usr/include/boost/smart_ptr/detail/shared_count.hpp:432 (kea-dhcp4+0xb845a)
#11 boost::shared_ptr<std::function<void ()> >::~shared_ptr() /usr/include/boost/smart_ptr/shared_ptr.hpp:335 (kea-dhcp4+0xb845a)
#12 isc::dhcp::Dhcpv4Srv::runOne() /home/marcin/devel/kea/src/bin/dhcp4/dhcp4_srv.cc:1113 (kea-dhcp4+0xb845a)
#13 isc::dhcp::Dhcpv4Srv::run() /home/marcin/devel/kea/src/bin/dhcp4/dhcp4_srv.cc:1023 (kea-dhcp4+0xb87b7)
#14 main /home/marcin/devel/kea/src/bin/dhcp4/main.cc:282 (kea-dhcp4+0x5082a)
Previous read of size 8 at 0x7b0c0000c220 by thread T6:
#0 boost::shared_ptr<isc::hooks::CalloutHandle> isc::dhcp::getCalloutHandle<boost::shared_ptr<isc::dhcp::Pkt4> >(boost::shared_ptr<isc::dhcp::Pkt4> const&) ../../../src/lib/dhcpsrv/callout_handle_store.h:50 (kea-dhcp4+0xb7f7d)
#1 isc::dhcp::Dhcpv4Srv::processPacketAndSendResponse(boost::shared_ptr<isc::dhcp::Pkt4>&) /home/marcin/devel/kea/src/bin/dhcp4/dhcp4_srv.cc:1139 (kea-dhcp4+0xb7f7d)
#2 isc::dhcp::Dhcpv4Srv::processPacketAndSendResponseNoThrow(boost::shared_ptr<isc::dhcp::Pkt4>&) /home/marcin/devel/kea/src/bin/dhcp4/dhcp4_srv.cc:1122 (kea-dhcp4+0xb88c7)
#3 void std::__invoke_impl<void, void (isc::dhcp::Dhcpv4Srv::*&)(boost::shared_ptr<isc::dhcp::Pkt4>&), isc::dhcp::Dhcpv4Srv*&, boost::shared_ptr<isc::dhcp::Pkt4>&>(std::__invoke_memfun_deref, void (isc::dhcp::Dhcpv4Srv::*&)(boost::shared_ptr<isc::dhcp::Pkt4>&), isc::dhcp::Dhcpv4Srv*&, boost::shared_ptr<isc::dhcp::Pkt4>&) /usr/include/c++/11/bits/invoke.h:74 (kea-dhcp4+0xba3c9)
#4 std::__invoke_result<void (isc::dhcp::Dhcpv4Srv::*&)(boost::shared_ptr<isc::dhcp::Pkt4>&), isc::dhcp::Dhcpv4Srv*&, boost::shared_ptr<isc::dhcp::Pkt4>&>::type std::__invoke<void (isc::dhcp::Dhcpv4Srv::*&)(boost::shared_ptr<isc::dhcp::Pkt4>&), isc::dhcp::Dhcpv4Srv*&, boost::shared_ptr<isc::dhcp::Pkt4>&>(void (isc::dhcp::Dhcpv4Srv::*&)(boost::shared_ptr<isc::dhcp::Pkt4>&), isc::dhcp::Dhcpv4Srv*&, boost::shared_ptr<isc::dhcp::Pkt4>&) /usr/include/c++/11/bits/invoke.h:96 (kea-dhcp4+0xba3c9)
#5 void std::_Bind<void (isc::dhcp::Dhcpv4Srv::*(isc::dhcp::Dhcpv4Srv*, boost::shared_ptr<isc::dhcp::Pkt4>))(boost::shared_ptr<isc::dhcp::Pkt4>&)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /usr/include/c++/11/functional:420 (kea-dhcp4+0xba3c9)
#6 void std::_Bind<void (isc::dhcp::Dhcpv4Srv::*(isc::dhcp::Dhcpv4Srv*, boost::shared_ptr<isc::dhcp::Pkt4>))(boost::shared_ptr<isc::dhcp::Pkt4>&)>::operator()<, void>() /usr/include/c++/11/functional:503 (kea-dhcp4+0xba3c9)
#7 void std::__invoke_impl<void, std::_Bind<void (isc::dhcp::Dhcpv4Srv::*(isc::dhcp::Dhcpv4Srv*, boost::shared_ptr<isc::dhcp::Pkt4>))(boost::shared_ptr<isc::dhcp::Pkt4>&)>&>(std::__invoke_other, std::_Bind<void (isc::dhcp::Dhcpv4Srv::*(isc::dhcp::Dhcpv4Srv*, boost::shared_ptr<isc::dhcp::Pkt4>))(boost::shared_ptr<isc::dhcp::Pkt4>&)>&) /usr/include/c++/11/bits/invoke.h:61 (kea-dhcp4+0xba3c9)
#8 std::enable_if<is_invocable_r_v<void, std::_Bind<void (isc::dhcp::Dhcpv4Srv::*(isc::dhcp::Dhcpv4Srv*, boost::shared_ptr<isc::dhcp::Pkt4>))(boost::shared_ptr<isc::dhcp::Pkt4>&)>&>, void>::type std::__invoke_r<void, std::_Bind<void (isc::dhcp::Dhcpv4Srv::*(isc::dhcp::Dhcpv4Srv*, boost::shared_ptr<isc::dhcp::Pkt4>))(boost::shared_ptr<isc::dhcp::Pkt4>&)>&>(std::_Bind<void (isc::dhcp::Dhcpv4Srv::*(isc::dhcp::Dhcpv4Srv*, boost::shared_ptr<isc::dhcp::Pkt4>))(boost::shared_ptr<isc::dhcp::Pkt4>&)>&) /usr/include/c++/11/bits/invoke.h:111 (kea-dhcp4+0xba3c9)
#9 std::_Function_handler<void (), std::_Bind<void (isc::dhcp::Dhcpv4Srv::*(isc::dhcp::Dhcpv4Srv*, boost::shared_ptr<isc::dhcp::Pkt4>))(boost::shared_ptr<isc::dhcp::Pkt4>&)> >::_M_invoke(std::_Any_data const&) /usr/include/c++/11/bits/std_function.h:290 (kea-dhcp4+0xba3c9)
#10 std::function<void ()>::operator()() const /usr/include/c++/11/bits/std_function.h:590 (libkea-util.so.80+0x3ef83)
#11 isc::util::ThreadPool<std::function<void ()>, std::deque<boost::shared_ptr<std::function<void ()> >, std::allocator<boost::shared_ptr<std::function<void ()> > > > >::run() ../../../src/lib/util/thread_pool.h:599 (libkea-util.so.80+0x3ef83)
Thread T6 (tid=95149, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xdc328)
#2 isc::dhcp::ControlledDhcpv4Srv::processCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::shared_ptr<isc::data::Element const>) /home/marcin/devel/kea/src/bin/dhcp4/ctrl_dhcp4_srv.cc:861 (kea-dhcp4+0x666f8)
#3 isc::dhcp::ControlledDhcpv4Srv::loadConfigFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /home/marcin/devel/kea/src/bin/dhcp4/ctrl_dhcp4_srv.cc:163 (kea-dhcp4+0x6732b)
#4 isc::dhcp::ControlledDhcpv4Srv::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /home/marcin/devel/kea/src/bin/dhcp4/ctrl_dhcp4_srv.cc:99 (kea-dhcp4+0x677fb)
#5 main /home/marcin/devel/kea/src/bin/dhcp4/main.cc:253 (kea-dhcp4+0x507e5)
SUMMARY: ThreadSanitizer: data race ../../../../src/libsanitizer/tsan/tsan_new_delete.cpp:150 in operator delete(void*, unsigned long)
==================
```
It looks that it may be related to using references to shared pointers in the functions passed to std::bind. Removing the reference seems to solve the problem. For example:
```patch
diff --git a/src/bin/dhcp4/dhcp4_srv.cc b/src/bin/dhcp4/dhcp4_srv.cc
index 612e8aa58b..f949015c04 100644
--- a/src/bin/dhcp4/dhcp4_srv.cc
+++ b/src/bin/dhcp4/dhcp4_srv.cc
@@ -1117,7 +1117,7 @@ Dhcpv4Srv::runOne() {
}
void
-Dhcpv4Srv::processPacketAndSendResponseNoThrow(Pkt4Ptr& query) {
+Dhcpv4Srv::processPacketAndSendResponseNoThrow(Pkt4Ptr query) {
try {
processPacketAndSendResponse(query);
} catch (const std::exception& e) {
diff --git a/src/bin/dhcp4/dhcp4_srv.h b/src/bin/dhcp4/dhcp4_srv.h
index 0ed55adfe5..0507d73ab5 100644
--- a/src/bin/dhcp4/dhcp4_srv.h
+++ b/src/bin/dhcp4/dhcp4_srv.h
@@ -352,7 +352,7 @@ public:
/// methods, generates appropriate answer, sends the answer to the client.
///
/// @param query A pointer to the packet to be processed.
- void processPacketAndSendResponseNoThrow(Pkt4Ptr& query);
+ void processPacketAndSendResponseNoThrow(Pkt4Ptr query);
/// @brief Process an unparked DHCPv4 packet and sends the response.
///
```
However, we should probably go over all similar cases, not only this one. I've seen similar issues with other functions.kea2.5.8https://gitlab.isc.org/isc-projects/kea/-/issues/3195Modifying pools using the subnetX_update commands does not update statistics2024-03-27T16:11:27ZMarcin SiodelskiModifying pools using the subnetX_update commands does not update statisticsI tested the latest addition to Stork - pools management. I had a subnet without pools initially. I added some address pools to the subnet and saved the changes. It results in sending `subnet4_update` followed by `config-write`. The stat...I tested the latest addition to Stork - pools management. I had a subnet without pools initially. I added some address pools to the subnet and saved the changes. It results in sending `subnet4_update` followed by `config-write`. The statistics (in particular `total-addresses` counter) has not been changed to reflect the new pools capacity. I am able to force recounting the statistics by sending the `config-reload` command, but that's not how it is supposed to work.kea2.5.8https://gitlab.isc.org/isc-projects/kea/-/issues/3179kea fails to log to syslog if run as non-root user2024-03-22T13:43:03ZLars Wendlerkea fails to log to syslog if run as non-root userWith the following config snippet
```
"loggers": [
{
"name": "kea-dhcp4",
"output_options": [ { "output": "syslog" } ],
"severity": "INFO",
...With the following config snippet
```
"loggers": [
{
"name": "kea-dhcp4",
"output_options": [ { "output": "syslog" } ],
"severity": "INFO",
"debuglevel": 0
}
]
```
kea won't log to syslog service once it's being started as non-root user. Simply starting kea as root makes it successfully log to syslog.
I am using the following syslogger: [sysklogd-2.4.4](https://github.com/troglobit/sysklogd)
I have found this problem being present in kea-2.4.1 and kea-2.5.4 and only tested the dhcp4 component of kea.kea2.5.8https://gitlab.isc.org/isc-projects/kea/-/issues/3163unstable unit tests2024-03-20T11:23:33ZAndrei Pavelandrei@isc.orgunstable unit testsThe following tests failed irregularly in the past fortnight:
* `AllocEngine6Test.reservedAddressInPoolReassignedOther` https://jenkins.aws.isc.org/job/kea-dev/job/ut-extended/1276/testReport/junit/(root)/AllocEngine6Test/all___debian_1...The following tests failed irregularly in the past fortnight:
* `AllocEngine6Test.reservedAddressInPoolReassignedOther` https://jenkins.aws.isc.org/job/kea-dev/job/ut-extended/1276/testReport/junit/(root)/AllocEngine6Test/all___debian_11_amd64___debian_11_amd64_results___reservedAddressInPoolReassignedOther/
```
test_utils.cc:88
Expected equality of these values:
first->cltt_
Which is: 1700075018
second->cltt_
Which is: 1700075019
```
* `HttpClientTest.unreachable` https://jenkins.aws.isc.org/job/kea-dev/job/ut-extended/1273/testReport/junit/(root)/HttpClientTest/all___alpine_3_16_amd64___alpine_3_16_amd64_results___unreachable/
```
server_client_unittests.cc:437
Failed
Timeout occurred while running the test!
```
* `MemfileLeaseQueryImpl6ProcessTest.queryByClientIdActiveLeases` https://jenkins.aws.isc.org/job/kea-dev/job/ut-extended/1274/testReport/junit/(root)/MemfileLeaseQueryImpl6ProcessTest/all___debian_12_amd64___debian_12_amd64_results___queryByClientIdActiveLeases/
```
lease_query_impl6_unittest.cc:1783
Expected equality of these values:
100
cltt_opt->getValue()
Which is: 101
```
* `MySQLBulkLeaseQuery6ProcessTest.queryByClientIdActiveLeases` https://jenkins.aws.isc.org/job/kea-dev/job/ut-extended/1280/testReport/junit/(root)/MySQLBulkLeaseQuery6ProcessTest/all___freebsd_13_0_amd64___freebsd_13_0_amd64_results___queryByClientIdActiveLeases/
```
bulk_lease_query6_unittest.cc:458
Expected equality of these values:
lease->valid_lft_ - elapsed
Which is: 3500
iaaddr_opt->getValid()
Which is: 3499
```
* `MySQLLeaseQueryImpl6ProcessTest.makeClientOptionSingleLink` https://jenkins.aws.isc.org/job/kea-dev/job/ut-extended/1265/testReport/junit/(root)/PgSQLLeaseQueryImpl6ProcessTest/all___alpine_3_15_amd64___alpine_3_15_amd64_results___makeClientOptionSingleLink/
```
lease_query_impl6_unittest.cc:880
Expected equality of these values:
100
cltt_opt->getValue()
Which is: 101
```
* `MySQLLeaseQueryImpl6ProcessTest.queryByClientIdActiveLeases` https://jenkins.aws.isc.org/job/kea-dev/job/ut-extended/1276/testReport/junit/(root)/MySQLLeaseQueryImpl6ProcessTest/all___debian_10_amd64___debian_10_amd64_results___queryByClientIdActiveLeases/
```
lease_query_impl6_unittest.cc:1783
Expected equality of these values:
100
cltt_opt->getValue()
Which is: 101
```
* `MySQLLeaseQueryImpl6ProcessTest.queryByIpAddressActiveLease` https://jenkins.aws.isc.org/job/kea-dev/job/ut-extended/1276/testReport/junit/(root)/MySQLLeaseQueryImpl6ProcessTest/all___ubuntu_20_04_amd64___ubuntu_20_04_amd64_results___queryByIpAddressActiveLease/
```
lease_query_impl6_unittest.cc:1581
Expected equality of these values:
100
cltt_opt->getValue()
Which is: 101
```
* `PgSQLBulkLeaseQuery6ProcessTest.queryByClientIdActiveLeases` https://jenkins.aws.isc.org/job/kea-dev/job/ut-extended/1275/testReport/junit/(root)/PgSQLBulkLeaseQuery6ProcessTest/all___fedora_38_amd64___fedora_38_amd64_results___queryByClientIdActiveLeases/
```
bulk_lease_query6_unittest.cc:458
Expected equality of these values:
lease->valid_lft_ - elapsed
Which is: 3500
iaaddr_opt->getValid()
Which is: 3499
```
* `PgSQLLeaseQueryImpl6ProcessTest.makeClientOptionSingleLink` https://jenkins.aws.isc.org/job/kea-dev/job/ut-extended/1276/testReport/junit/(root)/PgSQLLeaseQueryImpl6ProcessTest/all___fedora_38_amd64___fedora_38_amd64_results___makeClientOptionSingleLink/
```
lease_query_impl6_unittest.cc:880
Expected equality of these values:
100
cltt_opt->getValue()
Which is: 101
```
* `PgSQLLeaseQueryImpl6ProcessTest.queryByIpAddressActiveLease` https://jenkins.aws.isc.org/job/kea-dev/job/ut-extended/1270/testReport/junit/(root)/PgSQLLeaseQueryImpl6ProcessTest/all___rhel_9_amd64___rhel_9_amd64_results___queryByIpAddressActiveLease/
```
lease_query_impl6_unittest.cc:1581
Expected equality of these values:
100
cltt_opt->getValue()
Which is: 101
```kea2.5.8https://gitlab.isc.org/isc-projects/kea/-/issues/3133Read TSIG secret from file2024-03-27T22:49:58ZIulian MegheaRead TSIG secret from file---
name: Feature request
about: Suggest an idea for this project
---
Allow reading tsig-keys (at least the secret) from an external file. This way it can integrate better with secret management tools.
For example the configuration cou...---
name: Feature request
about: Suggest an idea for this project
---
Allow reading tsig-keys (at least the secret) from an external file. This way it can integrate better with secret management tools.
For example the configuration could use more relaxed permissions but the file containing the tsig secret can use very restrictive permissions.kea2.5.8https://gitlab.isc.org/isc-projects/kea/-/issues/3129extend hammer to prepare kea to test it with TSAN2024-03-28T08:12:03ZWlodzimierz Wencelextend hammer to prepare kea to test it with TSANWhile checking what is really needed to run fully automated system tests against KEA with TSAN enabled I came to a conclusion that different compilation procedure have to be incorporated in hammer.
In TSAN job in jenkins we are using:
`...While checking what is really needed to run fully automated system tests against KEA with TSAN enabled I came to a conclusion that different compilation procedure have to be incorporated in hammer.
In TSAN job in jenkins we are using:
```
CXX=clang++
CXXFLAGS="-g3 -ggdb -O0 -fsanitize=thread -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches"
TSAN_OPTIONS="detect_deadlocks=1 second_deadlock_stack=1"
```kea2.5.8Wlodzimierz WencelWlodzimierz Wencelhttps://gitlab.isc.org/isc-projects/kea/-/issues/3125HA ignored packets cause DROP statistics counter increment2024-03-27T12:58:00ZDarren AnkneyHA ignored packets cause DROP statistics counter incrementHA_BUFFER6_RECEIVE_NOT_FOR_US increments drop counters.
- This happens at least with a load balancing configuration.
- I think maybe not with hot-standby since I don't think the service logs anything or cares about incoming client pack...HA_BUFFER6_RECEIVE_NOT_FOR_US increments drop counters.
- This happens at least with a load balancing configuration.
- I think maybe not with hot-standby since I don't think the service logs anything or cares about incoming client packets unless it loses contact with the HA peer?
- I cite BUFFER6 above but I'm sure the same is true for DHCPv4.
Possible solutions:
- introduce a new drop status that could be discounted later or part of a different drop statistic?
- Could introduce new status that it is ignored or filtered instead of dropped?
[SF1374](https://isc.lightning.force.com/lightning/r/Case/5007V00002YkO0oQAF/view)kea2.5.8