improve performance of NETCONF operations by retrieving items in bulk
Data often needs to be retrieved from a sysrepo datastore. Such an action is done, among other opportunities, every time a change is brought to the configuration. kea-netconf currently calls Session::get_item()
for all leaves and Session::get_items()
for all leaf lists. The repeated calls to these functions hinder performance as shown by the call graph below[0] which is collected after a minor change is done through NETCONF to a configuration with 114 subnets. This action takes 10 seconds on my laptop which might seem reasonable, but it can take significantly more on lesser provisioned systems. The performance also seems to worsen quadratically. See RT#20676 for a distribution of measurements over configuration sizes.
The performance could be improved by doing a single run to the sysrepo datastore and retrieving the entire configuration data in one go. Retrieving the entire configuration seems to be a requirement because we validate it before comitting any minor change. If this had not been a requirement, we could only retrieve the changed node and apply it to the already committed configuration.
As such, the bulk retrieval could be done via sr_get_items()
/ Session::get_items()
on the top-level node. The efficiency of this approach is also mentioned in Sysrepo code:
https://github.com/sysrepo/sysrepo/blob/v1.4.140/src/sysrepo.h#L749-L751
* @see Use ::sr_get_items for retrieving larger chunks
* of data from the datastore. Since it retrieves the data from datastore in
* larger chunks, it can work much more efficiently than multiple ::sr_get_item calls.
Or, in the case that it only works on leaf lists, then sr_get_data()
/ Session::get_data()
would be the alternative:
https://github.com/sysrepo/sysrepo/blob/v1.4.140/src/sysrepo.h#L805-L806
* Top-level trees are always returned so if an inner node is selected, all of its descendants
* and its direct parents (lists also with keys) are returned.
[0] Click and zoom. See how most of the time is spent in sr_get_item
and lyb_parse_subtree
.