stork issueshttps://gitlab.isc.org/isc-projects/stork/-/issues2024-03-25T17:24:07Zhttps://gitlab.isc.org/isc-projects/stork/-/issues/1335Not possible to run DHCPv6 only and collect stats with Prometheus2024-03-25T17:24:07ZRinse KloekNot possible to run DHCPv6 only and collect stats with Prometheus - Kea version: 2.4.1
- Stork: which version? 1.15.0
- OS: Debian 11
I have an issue with the prometheus exporter. I had enabled DHCPv4/DHCPv6/D2 in my kea-ctrl-agent as well as installed the KEA DHCPv4 server. However this server on... - Kea version: 2.4.1
- Stork: which version? 1.15.0
- OS: Debian 11
I have an issue with the prometheus exporter. I had enabled DHCPv4/DHCPv6/D2 in my kea-ctrl-agent as well as installed the KEA DHCPv4 server. However this server only serves DHCPv6 request. If I remove the DHCPv6/D2 entries in the Kea-ctrl-agent.conf and deinstall KEA DHCPv4 server, I am not able to see my DHCPv6 PD stats (kea_dhcp6_pd_assigned_total) anymore.
I get this error message in the log on the server
stork-agent[929757]: time="2024-03-19 11:54:23" level="error" msg="Problem parsing DHCPv4 labels from Kea: problem with content of DHCP labels response from Kea: forwarding socket is not configured for the server type dhcp4\nisc.org/stork/agent.(*SubnetList).UnmarshalJSON\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:82\nencoding/json.(*decodeState).array\n\t/builds/isc-projects/stork/tools/golang/go/src/encoding/json/decode.go:507\nencoding/json.(*decodeState).value\n\t/builds/isc-projects/stork/tools/golang/go/src/encoding/json/decode.go:364\nencoding/json.(*decodeState).unmarshal\n\t/builds/isc-projects/stork/tools/golang/go/src/encoding/json/decode.go:181\nencoding/json.Unmarshal\n\t/builds/isc-projects/stork/tools/golang/go/src/encoding/json/decode.go:108\nisc.org/stork/agent.(*lazySubnetNameLookup).fetchAndCacheNames\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:274\nisc.org/stork/agent.(*lazySubnetNameLookup).getName\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:291\nisc.org/stork/agent.(*lazySubnetNameLookup).getNameOrDefault\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:303\nisc.org/stork/agent.(*PromKeaExporter).setDaemonStats\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:772\nisc.org/stork/agent.(*PromKeaExporter).collectStats\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:877\nisc.org/stork/agent.(*PromKeaExporter).statsCollectorLoop\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:724\nruntime.goexit\n\t/builds/isc-projects/stork/tools/golang/go/src/runtime/asm_amd64.s:1650" file=" promkeaexporter.go:276 "
`
After installing the KEA DHCPv4 server again and reeneabling the KEA DHCPv4 server in kea-ctrl-agent these log message don't appear anymore and the Prometheus stats collection works again.
Is it possible to remove the DHCPv4 server and keep the stats working for DHCPv6
regads,
Rinsehttps://gitlab.isc.org/isc-projects/stork/-/issues/1323Attach more labels to the Prometheus samples2024-03-19T14:58:05ZSlawek FigielAttach more labels to the Prometheus samplesCurrently, if you haven't the subnet_cmds hook, the metrics are labeled with the subnet ID, and if you have the subnet_cmds hook, the metrics are labeled with the subnet name prefix if provided, otherwise with the subnet ID.
Our custome...Currently, if you haven't the subnet_cmds hook, the metrics are labeled with the subnet ID, and if you have the subnet_cmds hook, the metrics are labeled with the subnet name prefix if provided, otherwise with the subnet ID.
Our customer needs samples labeled by subnet ID regardless of the subnet_cmds presence. It would also be helpful to attach the shared network name.
[SF#1762](https://isc.lightning.force.com/lightning/r/Case/500S6000006AQSSIA4/view)1.17https://gitlab.isc.org/isc-projects/stork/-/issues/1322Upgrade Grafana and its dashboards2024-03-05T15:02:12ZSlawek FigielUpgrade Grafana and its dashboardsYou cannot import the example Grafana dashboards to a modern Grafana instance.
![image](/uploads/a95f33a1aa876ae18d92057fd933fc5f/image.png)
The Stork demo uses the `8.3.7` Grafana version. The latest version is `10.3.3`.
We should up...You cannot import the example Grafana dashboards to a modern Grafana instance.
![image](/uploads/a95f33a1aa876ae18d92057fd933fc5f/image.png)
The Stork demo uses the `8.3.7` Grafana version. The latest version is `10.3.3`.
We should upgrade the Grafana (and maybe Prometheus) used in the demo and migrate the example dashboards to a modern format.1.16https://gitlab.isc.org/isc-projects/stork/-/issues/1315Prometheus exporters should fetch the data on demand2024-03-05T14:56:38ZSlawek FigielPrometheus exporters should fetch the data on demandCurrently, we periodically aggregate the metrics for Prometheus purposes using internal Stork intervals. It means the Prometheus collector always pulls a bit of outdated cached values.
We should avoid using the internal collecting loop ...Currently, we periodically aggregate the metrics for Prometheus purposes using internal Stork intervals. It means the Prometheus collector always pulls a bit of outdated cached values.
We should avoid using the internal collecting loop and our own intervals. The Prometheus collector should be the one to decide when and how often the data are fetched.
- [ ] The Stork agent should send the `statistics-get-all` command and process the response on request to the `/metrics` endpoint
- [ ] The Stork server should retrieve the adjusted utilizations and machine counters on request to the `/metrics` endpointbackloghttps://gitlab.isc.org/isc-projects/stork/-/issues/1289Fix metrics endpoint in demo2024-03-05T10:31:36ZSlawek FigielFix metrics endpoint in demoThe server's `/metrics` endpoint is unavailable in the demo environment due to the misconfiguration of Nginx.
This issue contains the changes made during the analysis of #1214.The server's `/metrics` endpoint is unavailable in the demo environment due to the misconfiguration of Nginx.
This issue contains the changes made during the analysis of #1214.1.16Slawek FigielSlawek Figielhttps://gitlab.isc.org/isc-projects/stork/-/issues/1214Shared network address utilization not consistent2024-02-27T10:03:39ZVictor PetrescuShared network address utilization not consistentHi everyone,
I've encounter an issue related to the values of the Shared Network Address Utilization. It seems that the values from the /metrics of the Stork Server frequently not showing same values as in the Stork Web Application.
Fo...Hi everyone,
I've encounter an issue related to the values of the Shared Network Address Utilization. It seems that the values from the /metrics of the Stork Server frequently not showing same values as in the Stork Web Application.
For example:
Information from Stork Web App:
![Screenshot_1205](/uploads/0d06970d5bdb83da790e7521dcde773e/Screenshot_1205.png)
Information from Stork Server /metrics:
storkserver_shared_network_address_utilization{name="1"} 0.005
storkserver_shared_network_address_utilization{name="2"} 0.154
storkserver_shared_network_address_utilization{name="3"} 0.004
storkserver_shared_network_address_utilization{name="4"} 0.003
storkserver_shared_network_address_utilization{name="5"} 0.003
storkserver_shared_network_address_utilization{name="6"} 0
As you can see the values don't match. Strange is that sometimes values do match.
Stork Web App is showing the correct values, the problem is with the ones from /metrics.
Thank you !1.16https://gitlab.isc.org/isc-projects/stork/-/issues/1211Stork subnet view does not account for reservations-out-of pool in 'Used Addr...2024-01-23T14:31:44Zfue36Stork subnet view does not account for reservations-out-of pool in 'Used Addresses' pool calculationIn our environment we have a subnet with 0 in-pool leases and 13 in use reservations-out-of-pool, however in Stork _/dhcp/subnets/xx_ URL it appears that 13 in pool leases are being utilized. Using Stork 1.12.0, Kea DHCP-4 2.2.0 on Alma...In our environment we have a subnet with 0 in-pool leases and 13 in use reservations-out-of-pool, however in Stork _/dhcp/subnets/xx_ URL it appears that 13 in pool leases are being utilized. Using Stork 1.12.0, Kea DHCP-4 2.2.0 on Alma 9.2.1.15https://gitlab.isc.org/isc-projects/stork/-/issues/1210Stork metrics are not up-to-date2023-11-24T15:30:50ZVictor PetrescuStork metrics are not up-to-dateHi everyone,
Is there any refresh rate for the statistics exported under /metrics ?
I’m asking this because I’ve integrated Stork Server with Prometheus and Grafana. The issue is that in Stork Server the DHCP Lease usage adjusts accord...Hi everyone,
Is there any refresh rate for the statistics exported under /metrics ?
I’m asking this because I’ve integrated Stork Server with Prometheus and Grafana. The issue is that in Stork Server the DHCP Lease usage adjusts accordingly to the ISC KEA DHCP server, but when pulling the metrics the value is not the same, seems to be an old one.
The fix to get /metrics up-to-date is to restart isc-stork-server which I don't consider a permanent fix.
Thank you !1.14Marcin SiodelskiMarcin Siodelskihttps://gitlab.isc.org/isc-projects/stork/-/issues/1177subnet stats should display a warning message if they are out of date2023-10-10T13:29:01ZRazvan Becheriusubnet stats should display a warning message if they are out of datelast time the stats have been received is stored in the database, so if stats are older than `current time - fetch time interval`, a warning should be displayed that stats are out of datelast time the stats have been received is stored in the database, so if stats are older than `current time - fetch time interval`, a warning should be displayed that stats are out of datebackloghttps://gitlab.isc.org/isc-projects/stork/-/issues/1153add pool statistics to prometheus2023-09-12T13:48:16ZRazvan Becheriuadd pool statistics to prometheusnew stats have been added in 2.4.0:
v4:
```
"subnet[1].pool[0].assigned-addresses": [
[
0,
"2023-06-13 20:42:46.836205"
]
],
"subnet[1...new stats have been added in 2.4.0:
v4:
```
"subnet[1].pool[0].assigned-addresses": [
[
0,
"2023-06-13 20:42:46.836205"
]
],
"subnet[1].pool[0].cumulative-assigned-addresses": [
[
0,
"2023-06-13 20:42:46.836137"
]
],
"subnet[1].pool[0].declined-addresses": [
[
0,
"2023-06-13 20:42:46.836213"
]
],
"subnet[1].pool[0].reclaimed-declined-addresses": [
[
0,
"2023-06-13 20:42:46.836225"
]
],
"subnet[1].pool[0].reclaimed-leases": [
[
0,
"2023-06-13 20:42:46.836236"
]
],
"subnet[1].pool[0].total-addresses": [
[
11010049,
"2023-06-13 20:42:46.836128"
]
],
```
v6:
```
"subnet[1].pd-pool[0].assigned-pds": [
[
0,
"2023-06-13 21:28:57.196785"
]
],
"subnet[1].pd-pool[0].cumulative-assigned-pds": [
[
0,
"2023-06-13 21:28:57.196744"
]
],
"subnet[1].pd-pool[0].reclaimed-leases": [
[
0,
"2023-06-13 21:28:57.196789"
]
],
"subnet[1].pd-pool[0].total-pds": [
[
256,
"2023-06-13 21:28:57.196741"
]
],
"subnet[1].pool[0].assigned-nas": [
[
0,
"2023-06-13 21:28:57.196773"
]
],
"subnet[1].pool[0].cumulative-assigned-nas": [
[
0,
"2023-06-13 21:28:57.196739"
]
],
"subnet[1].pool[0].declined-addresses": [
[
0,
"2023-06-13 21:28:57.196775"
]
],
"subnet[1].pool[0].reclaimed-declined-addresses": [
[
0,
"2023-06-13 21:28:57.196779"
]
],
"subnet[1].pool[0].reclaimed-leases": [
[
0,
"2023-06-13 21:28:57.196783"
]
],
"subnet[1].pool[0].total-nas": [
[
281474976710656,
"2023-06-13 21:28:57.196736"
]
],
```
```backloghttps://gitlab.isc.org/isc-projects/stork/-/issues/896Question: Query hit ratio = ?2022-11-10T23:12:01ZmikygeeQuestion: Query hit ratio = ?Hello,
I can see that query hit ratio is ?
![image](/uploads/0ef6506abbe4ff6d763b53ee016c1a6f/image.png)
When I rollover the triangle I see Hits: 0 , Misses: 0
When should I start my troubleshooting to understand why it's ? or 0 ?
R...Hello,
I can see that query hit ratio is ?
![image](/uploads/0ef6506abbe4ff6d763b53ee016c1a6f/image.png)
When I rollover the triangle I see Hits: 0 , Misses: 0
When should I start my troubleshooting to understand why it's ? or 0 ?
Regardshttps://gitlab.isc.org/isc-projects/stork/-/issues/871Unsupported stat info messages should be a lower log level2023-03-07T13:44:37Zvps-ericUnsupported stat info messages should be a lower log levelThese log statements are, in my opinion, not relevant enough to warrant being INFO level:
```
INFO[2022-09-26 21:45:51] promkeaexporter.go:683 Encountered unsupported stat: v4-allocation-fail-no-pools
INFO[2022-09-26 21:45:51] prom...These log statements are, in my opinion, not relevant enough to warrant being INFO level:
```
INFO[2022-09-26 21:45:51] promkeaexporter.go:683 Encountered unsupported stat: v4-allocation-fail-no-pools
INFO[2022-09-26 21:45:51] promkeaexporter.go:683 Encountered unsupported stat: v4-allocation-fail-subnet
INFO[2022-09-26 21:45:51] promkeaexporter.go:680 Encountered unsupported stat: v4-reservation-conflicts
INFO[2022-09-26 21:45:51] promkeaexporter.go:683 Encountered unsupported stat: declined-addresses
INFO[2022-09-26 21:45:51] promkeaexporter.go:683 Encountered unsupported stat: reclaimed-leases
INFO[2022-09-26 21:45:51] promkeaexporter.go:683 Encountered unsupported stat: reclaimed-declined-addresses
INFO[2022-09-26 21:45:51] promkeaexporter.go:683 Encountered unsupported stat: v4-allocation-fail
INFO[2022-09-26 21:45:51] promkeaexporter.go:683 Encountered unsupported stat: v4-reservation-conflicts
INFO[2022-09-26 21:45:51] promkeaexporter.go:683 Encountered unsupported stat: v4-allocation-fail-shared-network
INFO[2022-09-26 21:45:51] promkeaexporter.go:683 Encountered unsupported stat: v4-allocation-fail-classes
INFO[2022-09-26 21:45:51] promkeaexporter.go:683 Encountered unsupported stat: cumulative-assigned-addresses
INFO[2022-09-26 21:45:51] promkeaexporter.go:680 Encountered unsupported stat: v4-reservation-conflicts
...
```
I propose that they be moved to DEBUG log level. Adding support for an unsupported stat is not something expected of a normal system administrator.
See #839.backloghttps://gitlab.isc.org/isc-projects/stork/-/issues/856Server metrics doesn't work2022-10-24T18:53:58ZSlawek FigielServer metrics doesn't workThe issue was found during 1.6.0 sanity checks. [Source](https://gitlab.isc.org/isc-projects/stork/-/issues/850#note_312543)
The Grafana reports invalid authorized/unauthorized machine numbers.
![image](https://gitlab.isc.org/isc-proje...The issue was found during 1.6.0 sanity checks. [Source](https://gitlab.isc.org/isc-projects/stork/-/issues/850#note_312543)
The Grafana reports invalid authorized/unauthorized machine numbers.
![image](https://gitlab.isc.org/isc-projects/stork/uploads/5549b98bc2c267349f897f62ed2a3868/image.png)
The metrics endpoint returns HTML instead of the data in Prometheus format.
![image](https://gitlab.isc.org/isc-projects/stork/uploads/fe3c4ff3be24de6e762ce45e49fc2f0d/image.png)
The server metrics endpoint returns an inappropriate response.1.8https://gitlab.isc.org/isc-projects/stork/-/issues/834NaN metrics values2023-01-30T13:13:47ZSlawek FigielNaN metrics valuesReported by @ray - [Source](https://mattermost.isc.org/isc/pl/n5hqa4gzmigjj87p6c4exbs8zc)
I don't yet have a Prometheus server polling this, but the NaN from the raw /metrics pull here seems wrong:
```
# TYPE bind_traffic_incoming_requ...Reported by @ray - [Source](https://mattermost.isc.org/isc/pl/n5hqa4gzmigjj87p6c4exbs8zc)
I don't yet have a Prometheus server polling this, but the NaN from the raw /metrics pull here seems wrong:
```
# TYPE bind_traffic_incoming_requests_udp4_size histogram
bind_traffic_incoming_requests_udp4_size_bucket{le="47"} 2
bind_traffic_incoming_requests_udp4_size_bucket{le="+Inf"} 2
bind_traffic_incoming_requests_udp4_size_sum NaN
bind_traffic_incoming_requests_udp4_size_count 2
```backloghttps://gitlab.isc.org/isc-projects/stork/-/issues/833Avoid creating separate metrics per transport2022-10-25T13:36:55ZSlawek FigielAvoid creating separate metrics per transportReported by @ray - [Source](https://mattermost.isc.org/isc/pl/n5hqa4gzmigjj87p6c4exbs8zc):
> \# TYPE bind_traffic_incoming_requests_udp4_size histogram
> bind_traffic_incoming_requests_udp4_size_bucket{le="47"} 2
> bind_traffic_inco...Reported by @ray - [Source](https://mattermost.isc.org/isc/pl/n5hqa4gzmigjj87p6c4exbs8zc):
> \# TYPE bind_traffic_incoming_requests_udp4_size histogram
> bind_traffic_incoming_requests_udp4_size_bucket{le="47"} 2
> bind_traffic_incoming_requests_udp4_size_bucket{le="+Inf"} 2
> bind_traffic_incoming_requests_udp4_size_sum NaN
> bind_traffic_incoming_requests_udp4_size_count 2
I'd also like to suggest that the udp4 part of these metrics should be a label e.g. {transport="udp4" } and not separate metrics per transport
the rationale is that as an operator, I want to be able to graph these things (udp4, tcp6, etc) in aggregate, and also in isolation, and IIUC, labels are the Prometheus way to accomplish thatoutstandinghttps://gitlab.isc.org/isc-projects/stork/-/issues/710DHCP dashbord statitistic doubles the values2022-08-16T15:54:59ZSlawek FigielDHCP dashbord statitistic doubles the values[Source](https://lists.isc.org/pipermail/stork-users/2022-February/000093.html)
Hello,
I made the maj to stork 1.1 today and since the values (Addresses and Prefixes) displayed on the Dashboard in the statistics section are doubled.
T...[Source](https://lists.isc.org/pipermail/stork-users/2022-February/000093.html)
Hello,
I made the maj to stork 1.1 today and since the values (Addresses and Prefixes) displayed on the Dashboard in the statistics section are doubled.
Thanks to the correction of the bug [#676] I could add my second server which is in standby mode. But as there is a synchronization of the leases between the master and the standby these are doubled and I think that is where the problem comes from.
So instead of having for example 5000 prefixes that are really assigned, the dashboard displays 10000. However when I add the values obtained via the "host reservations", "subnets" and "shared-network" I have only 5000 prefixes that have been assigned.
I think that the dashboard does not take into account that my two servers are in Hot-Standby and displays the sum of the leases obtained by the two.
Is there any way to get the real value?
Thanks,
PO1.5Slawek FigielSlawek Figielhttps://gitlab.isc.org/isc-projects/stork/-/issues/439tests for kea statistics from #413 should be enabled2021-05-31T15:29:31ZMichal Nowikowskitests for kea statistics from #413 should be enabledCurrently there is a problem with GitLab CI and VM with LXD that runs only in IPv6. These tests and especially Kea needs IPv4.
Problem is probably connected with configuration of VM. It needs to be fixed, recreated and uploaded do regis...Currently there is a problem with GitLab CI and VM with LXD that runs only in IPv6. These tests and especially Kea needs IPv4.
Problem is probably connected with configuration of VM. It needs to be fixed, recreated and uploaded do registry.0.18Michal NowikowskiMichal Nowikowskihttps://gitlab.isc.org/isc-projects/stork/-/issues/254Update templates for new Kea cumulative assigned statistics2022-11-16T11:54:50ZFrancis DupontUpdate templates for new Kea cumulative assigned statisticsKea 1.7.7 now has a new statistic that shows an ever increasing number of assigned addresses. The customer wanted to observe how many devices were provisioned in his network each day (he'll be resetting this every midnight).Kea 1.7.7 now has a new statistic that shows an ever increasing number of assigned addresses. The customer wanted to observe how many devices were provisioned in his network each day (he'll be resetting this every midnight).backloghttps://gitlab.isc.org/isc-projects/stork/-/issues/252Stork needs to show LPS statistic2020-08-03T13:00:02ZTomek MrugalskiStork needs to show LPS statisticWith the completion of #226 we now have a dashboard that shows subnets and pool utilization. We should extend it with leases per second statistic. Here's a mockup made by @godfryd how this could possibly look like.
![dashboard-mockup](/...With the completion of #226 we now have a dashboard that shows subnets and pool utilization. We should extend it with leases per second statistic. Here's a mockup made by @godfryd how this could possibly look like.
![dashboard-mockup](/uploads/15b7c7f88b7c8291949b58f32526fd8a/dashboard-mockup.png)
The ability to show it in the last 5 or 15 mins and some larger timescale (24h maybe) would be very useful.
This will require some small design. Here's couple caveats that's worth considering:
- should we keep the historic (last 24h) data in stork or in kea?
- what if Kea is restarted?
- if we implement something new in Kea, how can this work with older Kea releases?
- Kea doesn't have explicit LPS statistic, but has the ability to store multiple observations of addresses-assigned with timestamp. This can be used to estimate LPS. But is it precise enough? If there's a lot of traffic, the observations are only split second apart. If there's no traffic at all, there are no observations.
- Kea has a mechanism for storing multiple observations. If it's useful, perhaps we could use it. If not, maybe we could get rid of it altogether?0.10Thomas MarkwalderThomas Markwalderhttps://gitlab.isc.org/isc-projects/stork/-/issues/211Periodically pull metrics in batches2020-04-13T14:30:15ZMatthijs Mekkingmatthijs@isc.orgPeriodically pull metrics in batchesBIND 9: Cache hit ratio will only be updated now when detecting the app and when explicitly refresh. It needs a statistics puller similar to Kea stats puller.
Pulling a lot of stats perhaps need to be done as a batch, per machine. This ...BIND 9: Cache hit ratio will only be updated now when detecting the app and when explicitly refresh. It needs a statistics puller similar to Kea stats puller.
Pulling a lot of stats perhaps need to be done as a batch, per machine. This needs group discussion. This discussion takes place elsewehere and the work is out of scope for this issue.0.7Matthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.org