DHCP6_DDNS_GENERATED_FQDN_UPDATE_FAIL during allocation of lease to client that became UNKNOWN

We encountered this bit of code in dhcp6_serv.cc within Dhcpv6Srv::generateFqdn()

    try {
        // The lease has been acquired but the FQDN for this lease hasn't
        // been updated in the lease database. We now have new FQDN
        // generated, so the lease database has to be updated here.
        // However, never update lease database for Advertise, just send
        // our notion of client's FQDN in the Client FQDN option.
        if (answer->getType() != DHCPV6_ADVERTISE) {
            Lease6Ptr lease;
            for (auto const& l : ctx.new_leases_) {
                if ((l->type_ == Lease::TYPE_NA) && (l->addr_ == addr)) {
                    lease = l;
                    break;
                }
            }
            if (lease) {
                lease->hostname_ = generated_name;
                lease->reuseable_valid_lft_ = 0;
                LeaseMgrFactory::instance().updateLease6(lease);

            } else {
                isc_throw(isc::Unexpected, "there is no lease in the database "
                          " for address " << addr << ", so as it is impossible"
                          " to update FQDN data. This is a programmatic error"
                          " as the given address is now being handed to the"
                          " client");
            }
        }

Clearly this was unexpected (the warning/error message being logged says so!)

This is what was logged (with the server name, IPv6 addresses and client MAC replaced/obscured for privacy)

2024-04-23 14:49:53.610 DEBUG [kea-dhcp6.dhcp6/4182503.139983248578304] DHCP6_CLASS_ASSIGNED duid=[CLIENT MAC ADDRESS HERE], tid=0x134984: client packet has been assigned to the following class(es): ALL, HA_vm-kea-testbig.customer.name, UNKNOWN 2024-04-23 14:49:53.610 WARN [kea-dhcp6.alloc-engine/4182503.139983248578304] ALLOC_ENGINE_V6_ALLOC_FAIL_CLASSES duid=[CLIENT MAC ADDRESS HERE], tid=0x134984: Failed to allocate an IPv6 address for client with classes: ALL, HA_vm-kea-testbig.customer.name, UNKNOWN 2024-04-23 14:49:53.610 ERROR [kea-dhcp6.ddns/4182503.139983248578304] DHCP6_DDNS_GENERATED_FQDN_UPDATE_FAIL duid=[CLIENT MAC ADDRESS HERE], tid=0x134984: failed to update the lease using address , after generating FQDN for a client, reason: there is no lease in the database for address , so as it is impossible to update FQDN data. This is a programmatic error as the given address is now being handed to the client

Note that the address IS being handed out the client, and indeed does appear as a lease when this was checked up on by the operator:

{
"cltt": 1713877713,
"duid": "CLIENT MAC ADDRESS HERE",
"fqdn-fwd": false,
"fqdn-rev": false,
"hostname": "THIS IS AN OBSCURED HOSTNAME",
"hw-address": "CLIENT MAC ADDRESS HERE",
"iaid": 0,
"ip-address": "SAME VALID IPv6 ADDRESS",
"preferred-lft": 2253,
"state": 0,
"subnet-id": 173703168,
"type": "IA_NA",
"valid-lft": 3606
},

To Reproduce

I don't have a reliable reproducer and the problem was transient, but perhaps the DDNS configuration and the circumstances under which it occurred could be helpful.

The DHCPv6 DDNS settings are significant because we're not actually using DDNS, but we ARE using the DDNS logic to create the FQDN:

"Dhcp6": {
"allocator": "iterative",
"calculate-tee-times": true,
"control-socket": {
"socket-name": "/var/lib/kea/kea-dhcp6.socket",
"socket-type": "unix"
},
"ddns-generated-prefix": "host",
"ddns-override-client-update": true,
"ddns-override-no-update": true,
"ddns-qualifying-suffix": "",
"ddns-replace-client-name": "always",  <<<<<<<<< THIS IS WHAT CAUSES THE FQDN TO BE COMPUTED
"ddns-send-updates": true, <<<<<<<<< GLOBAL SETTING - but overridden by "enable-updates" below.
"ddns-update-on-renew": false,
"ddns-use-conflict-resolution": false,
"decline-probation-period": 86400,
"dhcp-ddns": {
"enable-updates": false, <<<<<<<<< DDNS-specific setting
"max-queue-size": 1024,
"ncr-format": "JSON",
"ncr-protocol": "UDP",
"sender-ip": "0.0.0.0",
"sender-port": 0,
"server-ip": "127.0.0.1",
"server-port": 53001
},

So there is no actual DDNS happening here, per the table found here:

https://kea.readthedocs.io/en/latest/arm/dhcp6-srv.html#ddns-for-dhcpv6

This warning only started to be logged when a bunch of clients, previously 'known' became UNKNOWN, likely because they were migrating between VMs and they were using tunnels between hypervisors - details a bit vague, but could be further researched.

My hypothesis is that there was something in this scenario causing the code to take an unexpected path in which although a new lease/address was being handed out to the client, we somehow expected that we were going to be updating a lease that was already in the database. Possibly these clients might have had host reservations before (??).

The problem subsequently went away, and it doesn't looks as if anything actually went operationally wrong, although I don't know if the hostname that was put in the (new?) lease was the one calculated as the FQDN per the configuration option "ddns-replace-client-name": "always"

Environment:

• Kea version: 2.4.1
• OS: unknown, although this is probably irrelevant
• Which features were compiled in: unknown, but I could find out (we know this customer environment very well)
• If/which hooks were loaded in: unknown (but again ... but not sure it's relevant anyway?)

Additional Information Full details can be found in Support case SF1909