|
|
# Introduction to Stork Development
|
|
|
|
|
|
This document describes the steps to join Stork development. There are more detailed documents aimed at developers available on the project's wiki pages. The Developer's Guide is a part of the Stork ARM. This document is not their replacement. It is a quick start guide for people joining the project.
|
|
|
|
|
|
## Project Location
|
|
|
Gitlab page: https://gitlab.isc.org/isc-projects/stork/
|
|
|
Stork ARM: https://stork.readthedocs.io/en/v1.7.0/index.html
|
|
|
Stork Developer's Guide: https://stork.readthedocs.io/en/v1.7.0/devel.html
|
|
|
|
|
|
## Project Meetings
|
|
|
Stork team meetings occur on Tuesdays at 15:00 CET. During these meetings, we discuss current development issues, plan new release milestones and triage GL issues. Contact Tomek if you wish to attend.
|
|
|
|
|
|
## Release Cycles
|
|
|
Stork releases occur every first Wednesday of the month. We currently do not differentiate between stable and development releases. Stork is still a relatively new project, with primarily new code coming in on each release.
|
|
|
|
|
|
## Development Environment
|
|
|
Stork should compile on a variety of different operating systems. We recently improved builds on FreeBSD and Alpine. However, the developers typically work on Ubuntu and macOS. The latter is not officially supported for Stork use in production. However, we ensure that Stork runs on macOS because some developers use this system.
|
|
|
|
|
|
## Main Components
|
|
|
|
|
|
### Stork Agent
|
|
|
|
|
|
stork-agent is a program written in Go (part of the Stork backend) that must be installed on each machine monitored by the stork-server. It serves the following purposes.
|
|
|
|
|
|
| Feature | Code Locations |
|
|
|
| ------ | ------ |
|
|
|
| Detect Kea and Bind9 instances on the machine | |
|
|
|
| Establish secure communication with the server | |
|
|
|
| Forward commands from the stork-server to Kea and Bind9 instances. Pass the responses back to the server | |
|
|
|
| Intercept Kea commands and responses with the ability to modify them before forwarding (man in the middle) | |
|
|
|
| Provide offline access to the Kea logs to the server | |
|
|
|
| Export Bind9 statistics to Prometheus | |
|
|
|
| Export Kea statistics to Prometheus | |
|
|
|
|
|
|
### Stork Server
|
|
|
stork-server is a program written in Go (part of the Stork backend) that is a central instance of the Stork deployment. It manages and monitors remote systems (via the agents). It provides the middleware serving UI files downloaded to a user's browser. The browser then communicates with the server over the REST API. The API can also be used directly. Stork system tests use the server's REST API to interact with it. The following is the list of the server's features.
|
|
|
|
|
|
| Feature | Code Locations |
|
|
|
| ------ | ------ |
|
|
|
| Middleware serving UI files | backend/server/restservice/middleware.go |
|
|
|
| Expose REST API for communication with the server | backend/server/restservice |
|
|
|
| Establish and maintain connectivity with the PostgreSQL database | backend/server/database |
|
|
|
| Run database queries with ORM | backend/server/database/model |
|
|
|
| Run periodic pullers fetching data from the monitored machines | backend/server/restservice/pullers.go, backend/server/apps/kea |
|
|
|
| Generate events and push them over the SSE connection | backend/server/eventcenter |
|
|
|
| Communicate with Kea and Bind9 instances via the Stork agents | backend/server/agentcomm |
|
|
|
| Store and maintain information about the monitored machines, apps (including their configurations), services, configuration review results, events, etc., in the database | backend/server/database/model, backend/server/apps, backend/server/apps/kea |
|
|
|
| Take part in the secure connection negotiations with the agents | backend/server/certs, backend/pki |
|
|
|
| Expose server metrics to Prometheus | |
|
|
|
| Generate packages with the troubleshooting files and return them as tar.gz via the middleware | backend/server/dumper |
|
|
|
| Apply Kea configuration changes (host_cmds only for now) using the Kea command channel | backend/server/config |
|
|
|
| Run Kea configuration reviews | backend/server/configreview |
|
|
|
|
|
|
### Stork Tool
|
|
|
stork-tool is a standalone program running database migrations, setting up the database schema, and importing custom certificates into Stork. Note that the stork-server can also automatically run necessary database migrations upon startup. However, stork-tool can do more than that.
|
|
|
|
|
|
### PostgreSQL Database
|
|
|
stork-server uses PostgreSQL database to hold all necessary information. It includes the server's information (e.g., settings, events) and the information fetched from the monitored servers or resulting from processing the fetched data. Most of the REST API calls perform some database operations (select, delete, insert data). The database is typically collocated with the server but can also be installed on a remote system.
|
|
|
|
|
|
### Prometheus
|
|
|
Prometheus is an open-source monitoring and alerting toolkit. It includes a multi-dimensional data model with time series data identified by metrics names. stork-agent works as an exporter of various Kea metrics to Prometheus. The exporter periodically sends statistics queries to Kea and makes them available to Prometheus using the Go client packages shipped with the Prometheus project.
|
|
|
|
|
|
### Grafana
|
|
|
Grafana is an open-source observability and data visualization platform that can display the time series data stored in Prometheus. Stork includes Grafana templates reflecting Kea and Bind9 monitoring views with various interesting metrics. Grafana and Prometheus can't be configured via Stork. However, links to the running Prometheus and Grafana services can be specified in the Stork WebUI. In that case, Stork can display links to the specific subnets in Grafana.
|
|
|
|
|
|
## Languages and Technologies
|
|
|
|
|
|
### Go
|
|
|
The Stork backend is implemented in Go (https://go.dev). Among other reasons, we chose this language because of its built-in concurrency (with goroutines), ease of building programs, large number of available packages, and growing community. We currently use Go 1.18 and often upgrade to new versions when desired language features become available.
|
|
|
|
|
|
### Typescript
|
|
|
The WebUI in Stork is implemented in Typescript (https://www.typescriptlang.org). It is a strongly typed programming language that builds on JavaScript. We currently use Typescript (4.8.2).
|
|
|
|
|
|
The ECMAScript is a JavaScript standard intended to ensure the interoperability of web pages across different browsers. Stork UI uses ECMAScript 2015 (ES2015), with extensions from later standards: es2018, es2020.string, ES2020.BigInt.
|
|
|
|
|
|
### Angular
|
|
|
Angular (https://angular.io) is a development platform for creating sophisticated single-page applications. A component is a functional piece of the UI comprising the .ts file with the component logic in Typescript, .html file with an HTML template of the component, a style file for the component, and the tests. Components can be small or big. Bigger components embed smaller components in them.
|
|
|
|
|
|
### PrimeNG
|
|
|
PrimeNG is a rich library of Angular components, such as form components, dynamic tables, data panels, and many more. Stork currently uses PrimeNG 13.3.3.
|
|
|
|
|
|
### Docker
|
|
|
The Stork demo setup comprises different inter-connected Docker containers simulating the production environment. It includes Kea and Bind9 instances with Stork agents, the Stork server instance, databases, Grafana, and Prometheus. The instances contain pre-populated data (e.g., DHCP leases, host reservations).
|
|
|
|
|
|
Docker containers are also used in our system tests to run Stork components the tests can interact with.
|
|
|
|
|
|
### Rake Build System
|
|
|
A Ruby version of the Make program uses Ruby syntax to define build tasks in Stork. Our build system includes a set of Rakefiles segregating build task categories into several name spaces.
|
|
|
|
|
|
### Protocol Buffers
|
|
|
It is a language and platform-neutral mechanism for serializing structured data. Data structures are created once, and the client-server code is generated from them. Stork uses this mechanism for communication between the server and the agents.
|
|
|
|
|
|
### Swagger
|
|
|
It is a rich toolset for defining REST API specifications. The Swagger editor helps define REST API endpoints by visualizing the output and highlighting syntax and errors. The Swagger UI can interact with existing API by selecting endpoints from the list and specifying static parameters.
|
|
|
|
|
|
Swagger 2.0 is the API specification version currently used in Stork. OpenAPI 3.0 is the next specification version with a more sophisticated data model. We currently don't use OpenAPI 3.0 because the Go-swagger code generator does not support it.
|
|
|
|
|
|
### PostgreSQL
|
|
|
Stork server uses PostgreSQL as a central database. Go-pg and Go-pg/orm are used as client libraries. These libraries are now in maintenance mode. The Bun project offers similar functionality but also works for other databases. We are considering migration to this project. |
|
|
\ No newline at end of file |