isc-stork-server consumes all memory on Debian 12

name: isc-stork-server suspected to leak memory
about: it consumes all available memory on Debian 12

When running isc-stork-server > 1.11 on Debian 12, CPU usage of the process /usr/bin/stork-server jumps to 100% and the memory consumption steadily increases after displaying the "DHCP" -> "Subnets" Page in the frontend. After 1-2 minutes, all available memory is consumed by the /usr/bin/stork-server process and the frontend becomes laggy and starts printing errors. This problem appears with the versions:

1.12
1.13
1.14

but not with:

1.11

To Reproduce Steps to reproduce the behavior:

Install isc-stork-server 1.14
Visit the page "DHCP" -> "Subnets"
Observe memory usage of /usr/bin/stork-server
When it reaches maximum usage, frontend becomes unresponsive

Expected behavior The /usr/bin/stork-server should consume a sane amount of memory and the frontend should stay responsive.

Environment:

Kea version: 2.2.1
Stork: 1.14
OS:
- Debian 12 amd64 for Kea and stork-agent (VM)
- Debian 12 amd64 for stork-server (separate system) with 8 GB RAM and 2 CPU cores (VM)
Database for stork-server:
- PostgreSQL 15 (Debian 12 amd64)
Kea configuration:
- Subnets in configuration (1384 subnets)
- Host reservations in configuration
- Leases stored in memfile
Kea hooks loaded:
- libdhcp_lease_cmds.so
- libdhcp_host_cmds.so
- libdhcp_stat_cmds.so
- libdhcp_ha.so
Kea setup: 3 Kea instances in HA configuration with roles:
- primary
- secondary
- backup

Additional Information

Stork server configuration server.env:

### database settings
### the address of a PostgreSQL database
STORK_DATABASE_HOST=ddi-db-prod.example.org
### the port of a PostgreSQL database
# STORK_DATABASE_PORT=
### the name of a database
STORK_DATABASE_NAME=stork
### the username for connecting to the database
STORK_DATABASE_USER_NAME=stork
### the SSL mode for connecting to the database
### possible values: disable, require, verify-ca, or verify-full
STORK_DATABASE_SSLMODE=require
### the location of the SSL certificate used by the server to connect to the database
# STORK_DATABASE_SSLCERT=
### the location of the SSL key used by the server to connect to the database
# STORK_DATABASE_SSLKEY=
### the location of the root certificate file used to verify the database server's certificate
# STORK_DATABASE_SSLROOTCERT=
### the password for the username connecting to the database
### empty password is set to avoid prompting a user for database password
STORK_DATABASE_PASSWORD=secretpassword

### REST API settings
### the IP address on which the server listens
STORK_REST_HOST=0.0.0.0
### the port number on which the server listens
STORK_REST_PORT=8443
### the file with a certificate to use for secure connections
STORK_REST_TLS_CERTIFICATE=/etc/ssl/localcerts/dhcp-mon-prod.example.org.crt
### the file with a private key to use for secure connections
STORK_REST_TLS_PRIVATE_KEY=/etc/ssl/localcerts/dhcp-mon-prod.example.org.key
### the certificate authority file used for mutual TLS authentication
# STORK_REST_TLS_CA_CERTIFICATE=
### the directory with static files served in the UI
STORK_REST_STATIC_FILES_DIR=/usr/share/stork/www

### enable Prometheus /metrics HTTP endpoint for exporting metrics from
### the server to Prometheus. It is recommended to secure this endpoint
### (e.g. using HTTP proxy).
STORK_SERVER_ENABLE_METRICS=false

### Logging parameters

### Set logging level. Supported values are: DEBUG, INFO, WARN, ERROR
STORK_LOG_LEVEL=WARN
### disable output colorization
CLICOLOR=false

Kea-DHCP4 configuration kea-dhcp4.conf (leaving out subnets and reservations):

{
  "Dhcp4": {
    "cache-max-age": 120,
    "cache-threshold": 0.5,
    "control-socket": {
      "socket-name": "/tmp/kea4-ctrl-socket",
      "socket-type": "unix"
    },
    "ddns-generated-prefix": "",
    "ddns-override-client-update": false,
    "ddns-override-no-update": false,
    "ddns-qualifying-suffix": "example.org.",
    "ddns-replace-client-name": "never",
    "ddns-send-updates": true,
    "ddns-update-on-renew": true,
    "ddns-use-conflict-resolution": true,
    "decline-probation-period": 3600,
    "dhcp-ddns": {
      "enable-updates": true,
      "max-queue-size": 1024,
      "ncr-format": "JSON",
      "ncr-protocol": "UDP",
      "server-ip": "127.0.0.1",
      "server-port": 53001
    },
    "hooks-libraries": [
      {
        "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so"
      },
      {
        "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_host_cmds.so"
      },
      {
        "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_stat_cmds.so"
      },
      {
        "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
        "parameters": {
          "high-availability": [
            {
              "cert-file": "/etc/ssl/localcerts/dhcp-pri1-prod.example.org.crt",
              "delayed-updates-limit": 100,
              "heartbeat-delay": 1000,
              "key-file": "/etc/ssl/localcerts/dhcp-pri1-prod.example.org.key",
              "max-ack-delay": 5000,
              "max-response-delay": 4000,
              "max-unacked-clients": 0,
              "mode": "load-balancing",
              "multi-threading": {
                "enable-multi-threading": true,
                "http-client-threads": 0,
                "http-dedicated-listener": true,
                "http-listener-threads": 0
              },
              "peers": [
                {
                  "auto-failover": true,
                  "name": "dhcp-pri1-prod.example.org",
                  "role": "primary",
                  "url": "https://10.158.0.68:8001/"
                },
                {
                  "auto-failover": true,
                  "name": "dhcp-sec1-prod.example.org",
                  "role": "secondary",
                  "url": "https://10.158.0.168:8001/"
                },
                {
                  "auto-failover": true,
                  "name": "dhcp-bac1-prod.example.org",
                  "role": "backup",
                  "url": "https://10.158.0.169:8001/"
                }
              ],
              "require-client-certs": true,
              "send-lease-updates": true,
              "sync-leases": true,
              "this-server-name": "dhcp-pri1-prod.example.org",
              "trust-anchor": "/usr/share/ca-certificates/root_ad-example.org.2018.crt"
            }
          ]
        }
      }
    ],
    "hostname-char-replacement": "",
    "hostname-char-set": "",
    "interfaces-config": {
      "interfaces": [
        "eth0"
      ]
    },
    "lease-database": {
      "name": "/var/lib/kea/kea-leases4.csv",
      "type": "memfile"
    },
    "loggers": [
      {
        "debuglevel": 99,
        "name": "*",
        "severity": "DEBUG"
      },
      {
        "debuglevel": 99,
        "name": "kea-dhcp4.commands",
        "severity": "DEBUG"
      }
    ],
    "multi-threading": {
      "enable-multi-threading": true,
      "packet-queue-size": 64,
      "thread-pool-size": 0
    },
    "option-data": [
      {
        "always-send": false,
        "data": "10.158.0.153, 10.158.1.53",
        "name": "domain-name-servers"
      },
      {
        "always-send": false,
        "data": "example.org",
        "name": "domain-name"
      },
      {
        "always-send": false,
        "data": "example.org",
        "name": "domain-search"
      },
      {
        "always-send": false,
        "data": "10.144.128.12",
        "name": "cisco-autoinstall"
      }
    ],
    "option-def": [
      {
        "array": false,
        "code": 150,
        "name": "cisco-autoinstall",
        "type": "ipv4-address"
      }
    ],
    "rebind-timer": 2000,
    "renew-timer": 1000,
    "reservations-global": true,
    "reservations-in-subnet": true,
    "reservations-out-of-pool": false,
    "valid-lifetime": 4000
  }
}

Edited Dec 12, 2023 by Roman Haefeli