crash on high request rate
Using forensic logging with a MySQL backend yields a segmentation fault if I send approximately 1000 requests per second with perfdhcp.
perfdhcp command:
perfdhcp -6 -l vethclient -R 4294967295 -r 1000 -t 1This is the Kea Config I used:
kea-dhcp6.conf
{
"Dhcp6": {
"config-control": {
"config-databases": [
{
"host": "127.0.0.1",
"max-reconnect-tries": 3,
"name": "keatest",
"password": "keatest",
"port": 3306,
"reconnect-wait-time": 3000,
"type": "mysql",
"user": "keatest"
}
],
"config-fetch-wait-time": 20
},
"control-socket": {
"socket-name": "/tmp/kea-dhcp6-ctrl.sock",
"socket-type": "unix"
},
"hooks-libraries": [
{
"library": "/opt/kea/lib/kea/hooks/libdhcp_legal_log.so",
"parameters": {
"base-name": "kea-forensic6",
"type": "mysql",
"name": "keatest",
"username": "keatest",
"password": "keatest"
}
},
{
"library": "/opt/kea/lib/kea/hooks/libdhcp_cb_cmds.so"
},
{
"library": "/opt/kea/lib/kea/hooks/libdhcp_lease_cmds.so"
},
{
"library": "/opt/kea/lib/kea/hooks/libdhcp_mysql_cb.so"
},
{
"library": "/opt/kea/lib/kea/hooks/libdhcp_stat_cmds.so"
}
],
"interfaces-config": {
"interfaces": [
"vethserver"
]
},
"lease-database": {
"name": "/tmp/kea-dhcp6.csv",
"persist": false,
"type": "memfile"
},
"loggers": [
{
"debuglevel": 99,
"name": "kea-dhcp6",
"output_options": [
{
"output": "stdout"
}
],
"severity": "DEBUG"
}
],
"multi-threading": {
"enable-multi-threading": true,
"thread-pool-size": 16,
"packet-queue-size": 0
},
"server-tag": "my-server",
"subnet6": [
{
"pd-pools": [
{
"delegated-len": 120,
"prefix": "2001:db8:1:0:2::",
"prefix-len": 80
}
],
"pools": [
{
"pool": "2001:db8:1:0:1::/80"
}
],
"subnet": "2001:db8:1::/64"
}
]
}
}
If I take out the libdhcp_legal_log hook library, the problem doesn't reproduce any more even.
The segmentation fault happened often in StatsMgr::add() or IOAddress::toBytes() but they probably have nothing to do with the actual memory corruption.
The 1000 requests per second is detrimental to reproducing this crash. Send too few and the race that probably causes this won't happen. If I send 100.000 it seems to happen less, but still happens.
premium fix in: https://gitlab.isc.org/isc-private/kea-premium/-/merge_requests/160
this MR needs to be backported to 1.8.x