Skip to content

High memory usage by long-running Stork agent process

The issue was reported on our mailing list: link.

The user reported a utilization of 1.8 GB RAM by the long-running Stork agent process. We should analyze if it is an expected memory usage and reasonableness and possibilities to decrease it.

Message 1

I have been observing very high memory utilization for Kea VMs. In ~4 days, the memory utilization has grown from ~2 GB to close to 8GB for no good reason.

image

I seem to be running multiple copies of processes for some reason, with stork-agent likely being the primary culprit, and Kea DHCP servers being likely the second culprit. I have the very same situation on both lab nodes. I have to restart the server / stork agent periodically to eliminate the memory bloat.

    634 stork-age  20   0 1866M 26240 15824 S  0.0  0.3  7:35.77 /usr/bin/stork-agent

    709 stork-age  20   0 1866M 26240 15824 S  0.0  0.3  0:18.34 /usr/bin/stork-agent

    710 stork-age  20   0 1866M 26240 15824 S  0.0  0.3  0:34.27 /usr/bin/stork-agent

    711 stork-age  20   0 1866M 26240 15824 S  0.0  0.3  0:00.00 /usr/bin/stork-agent

    713 stork-age  20   0 1866M 26240 15824 S  0.0  0.3  0:00.01 /usr/bin/stork-agent

    727 stork-age  20   0 1866M 26240 15824 S  0.0  0.3  0:00.00 /usr/bin/stork-agent

    740 stork-age  20   0 1866M 26240 15824 S  0.0  0.3  1:29.63 /usr/bin/stork-agent

    894 stork-age  20   0 1866M 26240 15824 S  0.0  0.3  1:24.96 /usr/bin/stork-agent

   1921 stork-age  20   0 1866M 26240 15824 S  0.0  0.3  1:29.49 /usr/bin/stork-agent

  15986 stork-age  20   0 1866M 26240 15824 S  0.0  0.3  0:53.26 /usr/bin/stork-agent

   1340 _kea       20   0  507M 23864 16784 S  0.0  0.3  4:54.19 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf

   1341 _kea       20   0  507M 23864 16784 S  0.0  0.3  0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf

   1342 _kea       20   0  507M 23864 16784 S  0.0  0.3  0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf

   1343 _kea       20   0  507M 23864 16784 S  0.0  0.3  0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf

   1344 _kea       20   0  507M 23864 16784 S  0.0  0.3  0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf

   1352 _kea       20   0  507M 23864 16784 S  0.0  0.3  0:05.43 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf

   1353 _kea       20   0  507M 23864 16784 S  0.0  0.3  0:05.49 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf

   1354 _kea       20   0  507M 23864 16784 S  0.0  0.3  0:08.22 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf

   1355 _kea       20   0  507M 23864 16784 S  0.0  0.3  0:08.25 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf

  20544 _kea       20   0  507M 23864 16784 S  0.0  0.3  0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf

  20545 _kea       20   0  507M 23864 16784 S  0.0  0.3  0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf

    679 root       20   0  107M 20736 12596 S  0.0  0.3  0:00.04 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal

    732 root       20   0  107M 20736 12596 S  0.0  0.3  0:00.00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal

   1889 _kea       20   0  507M 19292 14552 S  0.0  0.2  3:05.41 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf

   1890 _kea       20   0  507M 19292 14552 S  0.0  0.2  0:00.00 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf

   1891 _kea       20   0  507M 19292 14552 S  0.0  0.2  0:00.00 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf

   1892 _kea       20   0  507M 19292 14552 S  0.0  0.2  0:00.00 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf

   1893 _kea       20   0  507M 19292 14552 S  0.0  0.2  0:00.00 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf

   1897 _kea       20   0  507M 19292 14552 S  0.0  0.2  0:05.47 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf

   1898 _kea       20   0  507M 19292 14552 S  0.0  0.2  0:05.15 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf

   1899 _kea       20   0  507M 19292 14552 S  0.0  0.2  0:10.02 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf

   1900 _kea       20   0  507M 19292 14552 S  0.0  0.2  0:09.95 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf

  20586 _kea       20   0  507M 19292 14552 S  0.0  0.2  0:00.09 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf

 20587 _kea       20   0  507M 19292 14552 S  0.0  0.2  0:00.08 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf

Message 2

Looking at the historic utilization over the week

image

you can clearly see a rather dramatic increase in memory utilization starting 5/10, with no specific changes to what the node is doing.

Message 3

These were taken from node2, which is a backup node. And no, I have not done anything to the processes, not even restarted them over the course of a last week or so. I looked through the logs and I do not see any explanation for the behavior.

Node1 is where I constantly change the config (it is the lab after all) and the process is restarted over and over again (“service xxx restart” from shell) but even there I would not expect a process hanging for any reason

Message 4

This is what I see on node1 over the last 24 hours – yesterday around 10:45 am, I was doing configuration changes on dhcpv6 side, so the process would be restarted a few times, it would run, ramping up memory utilization is seems, and then got restarted again and has been running since. No changes were made to dhcpv4 process or stork agent

image

I am happy to share the config file(s) with you to see whether there is any issue with them and any potential ways of improving memory utilization.