dhcpd initialization race can leave a socket unread forever
name: Bug report
about: Create a report to help us improve
If you believe your bug report is a security issue (e.g. a packet that can kill the server), DO NOT REPORT IT HERE. Please use https://www.isc.org/community/report-bug/ instead or send mail to security-office(at)isc(dot)org. If you really need to report it here, please set the confidential field to true.
Describe the bug we are experiencing a race condition in dhcpd initialization that prevents leases from being handed out if a packet arrives during server init. note that in our case we use dhcpd 4.4.1 and libisc from bind 9.11.35 but i believe the issue still exists. our bind libraries are built using threads and epoll on linux.
in https://gitlab.isc.org/isc-projects/dhcp/-/blob/master/omapip/dispatch.c#L259 we register a socket into libisc with a callback. a little later, we insert into a linked list https://gitlab.isc.org/isc-projects/dhcp/-/blob/master/omapip/dispatch.c#L279.
in the callback, we check if the callback argument (the registered socket) is in the linked list https://gitlab.isc.org/isc-projects/dhcp/-/blob/master/omapip/dispatch.c#L137. if it is not, we return 0. this return 0 will disable re-arming of the socket in libisc epoll code in https://gitlab.isc.org/isc-projects/bind9/-/blob/v9_11_35/lib/isc/unix/socket.c#L4017.
if socket fails to be re-armed (because we returned 0), it never gets armed again and server will run without reading any DHCP packets. we can observe the socket receive queue increasing when this happens, and dhcp clients time out.
# ss -0ap | grep -E 'dhcpd|Netid'
Cannot open netlink socket: Protocol not supported
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
p_raw UNCONN 26880 0 *:br-lan * users:(("dhcpd",pid=13706,fd=11))
To Reproduce Steps to reproduce the behavior:
- run dhcpd.
- client sends dhcp request.
- if server race is triggered, it will not reply and packets pile up in receive queue.
to widen the race, you can try to place sleep(10); at https://gitlab.isc.org/isc-projects/dhcp/-/blob/master/omapip/dispatch.c#L278 after fdwatch creation but before linked list insertion, and send dhcp request in that sleep window.
Expected behavior server replies to dhcp requests.
Environment:
- ISC DHCP version: 4.4.1
- OS: yocto 3.1
- Which features were compiled in
Additional Information Add any other context about the problem here. In particular, feel free to share your config file and logs from around the time error occurred. Don't be shy to send more logs than you think are relevant. It is easy to grep large log files. It is tricky to guess what may have happened without any information.
Make sure you anonymize your config files (at the very lease make sure you obfuscate your database credentials, but you may also replace your actual IP addresses and host names with example.com and 10.0.0.0/8 or 2001:db8::/32).
Some initial questions
- Are you sure your feature is not already implemented in the latest ISC DHCP version?
- Are you sure your requrested feature is not already impemented in Kea? Perhaps it's a good time to consider migration?
- Are you sure what you would like to do is not possible using some other mechanisms?
- Have you discussed your idea on dhcp-users and/or dhcp-workers mailing lists?
Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] It is very important to describe what you would like to do and why?
Describe the solution you'd like A clear and concise description of what you want to happen.
Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.
Additional context Add any other context about the feature request here.
Funding its development ISC DHCP is run by ISC, which is a small non-profit organization without any government funding or any permanent sponsorship organizations. Are you able and willing to participate financially in the development costs?
Participating in development Are you willing to participate in the feature development? ISC team always tries to make a feature as generic as possible, so it can be used in wide variety of situations. That means the proposed solution may be a bit different that you initially thought. Are you willing to take part in the design discussions? Are you willing to test an unreleased engineering code?
Contacting you How can ISC reach you to discuss this matter further? If you do not specify any means such as e-mail, jabber id or a telephone, we may send you a message on github with questions when we have them.