Add option(s) to make use of an additional Kea backup server (with HA) non-blocking
This is from Support ticket #15378 and is a follow-up feature request to make a Kea HA pair's operation more independent of any backup Kea server that they're also sending lease updates to.
The question of whether or not this would be a good idea arose during the investigation of #964 (closed) - although this request is orthogonal to the work on that issue.
The question asked was, how important is it that the active server(s) in the HA configuration, wait for an acknowledgement to their lease updates from the backup server?
Notably, the backup server does not participate in any HA monitoring - so the HA pair will not know if it becomes unreachable or goes down - they will just end up waiting indefinitely (or possibly failing on a network send) in that case. This could significantly affect the production environment.
Therefore, this is a proposal that we add a configuration 'knob' to make waiting on the acknowledgement of the lease updates to the backup server, an optional thing. (IF the system administrator prefers for the HA pair to wait, then the onus is on them to ensure the availability and reachability of the backup servers, or else...)
Ahem, but that then leads to another 'what if?'. What if some of the lease updates go astray? This probably doesn't matter too much - 'almost right' is going to be a better bet for recovery using a backup server (in case both of the HA pair for some reason disappear entirely), than 'nothing at all'. As part of this feature request therefore, could we also consider implementing (on the backup server) a period lease sync (as if it had just been rebooted and was coming online and starting to participate afresh)?
--- And finally, specifically for backup servers where the main HA pair are not waiting on acknowledgements on lease updates, should/could we also add an option to batch lease updates every x seconds (or so), to optimise the process for keeping the backup server up to date?