Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • Kea Kea
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 554
    • Issues 554
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 66
    • Merge requests 66
  • Deployments
    • Deployments
    • Releases
  • Packages and registries
    • Packages and registries
    • Container Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • ISC Open Source ProjectsISC Open Source Projects
  • KeaKea
  • Issues
  • #1711
Closed
Open
Issue created Feb 18, 2021 by Andrei Pavel@andreiMaintainer

crash on high request rate

Using forensic logging with a MySQL backend yields a segmentation fault if I send approximately 1000 requests per second with perfdhcp.

perfdhcp command:

perfdhcp -6 -l vethclient -R 4294967295 -r 1000 -t 1

This is the Kea Config I used:

kea-dhcp6.conf { "Dhcp6": { "config-control": { "config-databases": [ { "host": "127.0.0.1", "max-reconnect-tries": 3, "name": "keatest", "password": "keatest", "port": 3306, "reconnect-wait-time": 3000, "type": "mysql", "user": "keatest" } ], "config-fetch-wait-time": 20 }, "control-socket": { "socket-name": "/tmp/kea-dhcp6-ctrl.sock", "socket-type": "unix" }, "hooks-libraries": [ { "library": "/opt/kea/lib/kea/hooks/libdhcp_legal_log.so", "parameters": { "base-name": "kea-forensic6", "type": "mysql", "name": "keatest", "username": "keatest", "password": "keatest" } }, { "library": "/opt/kea/lib/kea/hooks/libdhcp_cb_cmds.so" }, { "library": "/opt/kea/lib/kea/hooks/libdhcp_lease_cmds.so" }, { "library": "/opt/kea/lib/kea/hooks/libdhcp_mysql_cb.so" }, { "library": "/opt/kea/lib/kea/hooks/libdhcp_stat_cmds.so" } ], "interfaces-config": { "interfaces": [ "vethserver" ] }, "lease-database": { "name": "/tmp/kea-dhcp6.csv", "persist": false, "type": "memfile" }, "loggers": [ { "debuglevel": 99, "name": "kea-dhcp6", "output_options": [ { "output": "stdout" } ], "severity": "DEBUG" } ], "multi-threading": { "enable-multi-threading": true, "thread-pool-size": 16, "packet-queue-size": 0 }, "server-tag": "my-server", "subnet6": [ { "pd-pools": [ { "delegated-len": 120, "prefix": "2001:db8:1:0:2::", "prefix-len": 80 } ], "pools": [ { "pool": "2001:db8:1:0:1::/80" } ], "subnet": "2001:db8:1::/64" } ] } }

If I take out the libdhcp_legal_log hook library, the problem doesn't reproduce any more even.

The segmentation fault happened often in StatsMgr::add() or IOAddress::toBytes() but they probably have nothing to do with the actual memory corruption.

The 1000 requests per second is detrimental to reproducing this crash. Send too few and the race that probably causes this won't happen. If I send 100.000 it seems to happen less, but still happens. 🤷 This is supersistion, but I guess 1000 is close enough to Kea's limit on my machine, I get around 1% drops.

premium fix in: https://gitlab.isc.org/isc-private/kea-premium/-/merge_requests/160

this MR needs to be backported to 1.8.x

Edited Feb 21, 2021 by Razvan Becheriu
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking