... | ... | @@ -55,24 +55,24 @@ These are the values terms used in this document. |
|
|
* **ANA**: Authorization and Authentication component of Stork. It provides mechanism to verify that the user has permissions to access the system and specifies what the user can do in the system. **ANA** component also provides means for the super user to create new users and define their permissions.
|
|
|
<<<<<<< HEAD
|
|
|
* **Application**: A **Program** or a group of **Programs** providing some functionality. For example, the Kea **Application** comprises of several **Daemons**, including the control agent and the dhcpv4 daemon. An **Application** of the same type can not run more than once on the same **Machine**.
|
|
|
* **Daemon**: A software component that is part of an **Application**. For example, the *named* deamon is part of the BIND9 **Application**. In the case of Kea, a *kea-dhcp4* is also a *Daemon*. The deamon may be *active* (running) or *inactive* (not running). The deamon may be inactive on purpose (because it is optional or not needed in the current configuration) or may be inactive because of a failure.
|
|
|
* **Daemon**: A software component that is part of an **Application**. For example, the *named* daemon is part of the BIND9 **Application**. In the case of Kea, a *kea-dhcp4* is also a *Daemon*. The daemon may be *active* (running) or *inactive* (not running). The daemon may be inactive on purpose (because it is optional or not needed in the current configuration) or may be inactive because of a failure.
|
|
|
=======
|
|
|
* **Application**: A **Program** or a group of **Programs** providing some functionality. For example, the Kea application comprises of several daemons, including the control agent and the dhcpv4 daemon. An **Application** of the same type can not run more than once on the same **Machine**.
|
|
|
* **Current State**: is a collection of runtime information about the application, deamon or service which provides the details about its operation and allows for making determination whether it is operating as desired or with issues. It also includes the information about the issues.
|
|
|
* **Current State**: is a collection of runtime information about the application, daemon or service which provides the details about its operation and allows for making determination whether it is operating as desired or with issues. It also includes the information about the issues.
|
|
|
>>>>>>> dfd80568315a6ea01a649550bb7da1a8f0b254cb
|
|
|
* **Component**: A functional part of the system. It can be a third party software integrated with Stork (e.g. Prometheus, Logstash instance, database instance) or it can be a part of Stork distributed on remote machines, e.g. **STAG**.
|
|
|
* **Condition**: is a boolean-like information about the deamon, application or service indicating if it is in the desired state or not. The two possible values are: `healthy` and `unhealthy`. The determination whether it is in one of the two is made by examining the current state vs desired state. The condition may be accompanied by a brief description of the problem which gives a hint to the user why the deamon, application or service is unhealthy.
|
|
|
* **Daemon**: A software component that is part of an application. For example, the *named* deamon is part of a BIND9 application. In the case of Kea: a *kea-dhcp4* is also a deamon. The deamon may have the `running` or `not running` status. The deamon may have `not running` status on purpose (because it is optional or not needed in the current configuration) or because of a failure. Whether it is not running on purpose or because of the failure can be determined by examining the desired state of the deamon.
|
|
|
* **Desired State**: is a collection of information about an application, deamon or service which define how the administrator wants the application, deamon or service to behave. A set of parameters belonging to the desired state may not be the same as the set of parameters describing the current state, because the current state may include some diagnostic information, e.g. CPU utilization or application uptime. However, it must be always possible to determine whether the application, service or deamon is in the desired state by examining the current state.
|
|
|
* **Condition**: is a boolean-like information about the daemon, application or service indicating if it is in the desired state or not. The two possible values are: `healthy` and `unhealthy`. The determination whether it is in one of the two is made by examining the current state vs desired state. The condition may be accompanied by a brief description of the problem which gives a hint to the user why the daemon, application or service is unhealthy.
|
|
|
* **Daemon**: A software component that is part of an application. For example, the *named* daemon is part of a BIND9 application. In the case of Kea: a *kea-dhcp4* is also a daemon. The daemon may have the `running` or `not running` status. The daemon may have `not running` status on purpose (because it is optional or not needed in the current configuration) or because of a failure. Whether it is not running on purpose or because of the failure can be determined by examining the desired state of the daemon.
|
|
|
* **Desired State**: is a collection of information about an application, daemon or service which define how the administrator wants the application, daemon or service to behave. A set of parameters belonging to the desired state may not be the same as the set of parameters describing the current state, because the current state may include some diagnostic information, e.g. CPU utilization or application uptime. However, it must be always possible to determine whether the application, service or daemon is in the desired state by examining the current state.
|
|
|
* **LAR**: Locally Applicable Resource, a resource which is stored in the **StorkDB** and not on the remote machine.
|
|
|
* **Program**: Synonym of deamon.
|
|
|
* **Program**: Synonym of daemon.
|
|
|
* **Machine**: A machine running one or more **Services** which Stork connects to.
|
|
|
* **RAR**: Remotely Applicable Resource, a resource which is used and stored on the remote machine besides being stored in **StorkDB**.
|
|
|
* **Resource**: Representative piece of information typically stored in the StorkDB, upon which CRUD operations can be performed. Example of resources: Stork user, subnet, DNS zone.
|
|
|
* **Service**: A set of **Applications** that interact together to provide something useful for the end user. For example, the DHCP service may have several kea **Applications** to provide High Availability. A **Service** is managed by Stork via an API, e.g. DHCP service, DNS service, Prometheus instance.
|
|
|
* **STAG**: Stork Agent, a deamon running on Machine which manages Services local to this machine. Example functions of **STAG**: report software versions, perform **Service** upgrades.
|
|
|
* **STAG**: Stork Agent, a daemon running on Machine which manages Services local to this machine. Example functions of **STAG**: report software versions, perform **Service** upgrades.
|
|
|
* **StorkDB**: PostgreSQL database used by Stork to store critical information about Stork itself (e.g. configuration information) and about other components of the system, such as managed servers, network topology, users with their credentials and many more.
|
|
|
* **Status**: is a boolean-like information indicating whether a deamon or service has been started and hasn't terminated. There are two `status` values defined: `running` or `not running`.
|
|
|
* **Status**: is a boolean-like information indicating whether a daemon or service has been started and hasn't terminated. There are two `status` values defined: `running` or `not running`.
|
|
|
* **StorkCLI**: Stork command line interface, a command line tool which can be used instead of the **StorkUI** to manage the system.
|
|
|
* **StorkUI**: The graphical user interface provided by Stork and available to the user via web browser.
|
|
|
* **Supported Software**: The software installed on **Machine** and providing services that Stork has integration with, e.g. Kea software, BIND software, Prometheus software etc.
|
... | ... | @@ -101,7 +101,7 @@ Despite Python and Flask being more mature we have chosen Golang because we beli |
|
|
|
|
|
In addition, some of the team members expressed their good experience with writing applications in Go.
|
|
|
|
|
|
Finally, we have chosen PostreSQL as a database for Stork. It is considered to be one of the most advanced relational open source databases. One of the nice PostgreSQL features, not available in MySQL, is the ability to write stored procedures in C (or Go compiled as C library). Finally, there are performant drivers and ORM libraries available in Golang for PostgreSQL, e.g. [go-pg](https://github.com/go-pg/pg).
|
|
|
Finally, we have chosen PostreSQL as a database for Stork. It is considered to be one of the most advanced relational open source databases. One of the nice PostgreSQL features, not available in MySQL, is the ability to write stored procedures in C (or Go compiled as C library). Finally, there are performant drivers and ORM libraries available in Golang for PostgreSQL, e.g. [go-pg](https://github.com/go-pg/pg).
|
|
|
|
|
|
The decision about selecting these technologies was made on the regular Stork call on September 24th, 2019 (see the appendix).
|
|
|
|
... | ... | @@ -189,9 +189,9 @@ STAG is functionally going to be a service performing certain tasks on Stork's r |
|
|
|
|
|
There are many different approaches to implementing RPC. The JSON-RPC 1.0 codec is provided with the standard *net/rpc/json* library. However, JSON-RPC 1.0 lacks the mechanism to send notifications, i.e. the requests for which the client expects no response. JSON-RPC 2.0 introduces bulk commands (multiple commands sent within a single request), which can be processed concurrently. However, the available go libraries implementing JSON-RPC 2.0 often lack some part of the functionality, e.g. bulk updates.
|
|
|
|
|
|
The good alternative to JSON-RPC is the [gRPC](https://grpc.io) using the [Protocol Buffers](https://developers.google.com/protocol-buffers) for data serialization. It is mature and battle tested solution which meets our requirements.
|
|
|
The good alternative to JSON-RPC is the [gRPC](https://grpc.io) using the [Protocol Buffers](https://developers.google.com/protocol-buffers) for data serialization. It is mature and battle tested solution which meets our requirements.
|
|
|
|
|
|
We have implemented a simple PoC using gRPC. It is available on the *experiments/grpc* branch of the Stork project.
|
|
|
We have implemented a simple PoC using gRPC. It is available on the *experiments/grpc* branch of the Stork project.
|
|
|
|
|
|
The following are the major strengths of the gRPC:
|
|
|
* It is possible to generate both the client and the server code from the `.proto` file. This file contains the API definition.
|
... | ... | @@ -250,7 +250,7 @@ It looks that introducing some control over the flow of resources between Stork |
|
|
|
|
|
The major issue with duplication of the RARs between Stork and remote systems is that the conflicts may arise when the RAR is modified on the remote system directly. Another use case is to modify the RAR in Stork without populating it to the remote system, perhaps because the system is temporarily offline. In this section of the document we're trying to deal with the resolution of the conflicts between the RARs.
|
|
|
|
|
|
### Use Case: Adding New Server
|
|
|
### Use Case: Adding New Server
|
|
|
|
|
|
The following sequence diagram shows interactions between the user, StorkUI, Stork Backend, Stork Database and the Kea server being added to the system. Kea is used here as an example. Any type of server would fit into this diagram.
|
|
|
|
... | ... | @@ -262,7 +262,7 @@ When the new server is being added to the system, the information about this ser |
|
|
|
|
|
## Daemons, Applications, and Services
|
|
|
|
|
|
What exactly is a **Service**? A **Service** is one or more **Applications** running on the network that provides some capability. An **Application** is one ore more **Daemons** that together provide the desired functionality. For example, the DHCP service may have multiple kea **Applications** running in different modes in order to provide High Availability. A kea **Application** constists of multiple **Daemons**, such as the Control Agent, the DHCPv4 daemon, the DHCPv6 daemon and the DDNS daemon.
|
|
|
What exactly is a **Service**? A **Service** is one or more **Applications** running on the network that provides some capability. An **Application** is one ore more **Daemons** that together provide the desired functionality. For example, the DHCP service may have multiple kea **Applications** running in different modes in order to provide High Availability. A kea **Application** consists of multiple **Daemons**, such as the Control Agent, the DHCPv4 daemon, the DHCPv6 daemon and the DDNS daemon.
|
|
|
|
|
|
It is important to acknowledge that there are different presentations of an **Application**. One view is that an **Application** is something that is running on a **Machine** and has a particular state (Active or not, configuration).
|
|
|
Another way to look at it is that the **Application** itself dictates the configuration and the **Daemons** running on a **Machine** represent an instance of that **Application**. That instance also has a state and hopefully matches
|
... | ... | @@ -282,11 +282,11 @@ There is a need for having a table that would hold the list of the services that |
|
|
```
|
|
|
postgres=# \d service_api;
|
|
|
Table "public.service_api"
|
|
|
Column | Type | Collation | Nullable | Default
|
|
|
Column | Type | Collation | Nullable | Default
|
|
|
-------------+---------+-----------+----------+-----------------------------------------
|
|
|
id | integer | | not null | nextval('service_api_id_seq'::regclass)
|
|
|
name | text | | |
|
|
|
description | text | | |
|
|
|
name | text | | |
|
|
|
description | text | | |
|
|
|
Indexes:
|
|
|
"service_api_pkey" PRIMARY KEY, btree (id)
|
|
|
"name_unique" UNIQUE CONSTRAINT, btree (name)
|
... | ... | @@ -298,7 +298,7 @@ and the predefined API types will be inserted into the table: |
|
|
|
|
|
```
|
|
|
postgres=# SELECT * FROM service_api;
|
|
|
id | name | description
|
|
|
id | name | description
|
|
|
----+-------------------+--------------------------------------
|
|
|
1 | rndc | BIND rndc
|
|
|
2 | kea-dhcp4-over-ca | Kea DHCPv4 server over Control Agent
|
... | ... | @@ -313,15 +313,15 @@ Finally, the service table containing the services that Stork should connect to |
|
|
```
|
|
|
postgres=# \d service;
|
|
|
Table "public.service"
|
|
|
Column | Type | Collation | Nullable | Default
|
|
|
Column | Type | Collation | Nullable | Default
|
|
|
-------------+---------+-----------+----------+-------------------------------------
|
|
|
id | integer | | not null | nextval('service_id_seq'::regclass)
|
|
|
name | text | | |
|
|
|
address | inet | | |
|
|
|
hostname | text | | |
|
|
|
port | integer | | not null |
|
|
|
api_type | integer | | not null |
|
|
|
description | text | | |
|
|
|
name | text | | |
|
|
|
address | inet | | |
|
|
|
hostname | text | | |
|
|
|
port | integer | | not null |
|
|
|
api_type | integer | | not null |
|
|
|
description | text | | |
|
|
|
Indexes:
|
|
|
"service_pkey" PRIMARY KEY, btree (id)
|
|
|
"service_api_type_idx" btree (api_type)
|
... | ... | @@ -439,7 +439,7 @@ The respective lines have the following meaning: |
|
|
|
|
|
Note that the policy file both defines user permissions and also associates some of the users with groups (roles). Neither the user names nor the resource names are validated by casbin enforcer. It is up to the application to validate the user names (also perform authentication) and the resource names. The policy file is read (typically when the application is launched) and the enforcer uses this data for authorization. Adding the new policy entries to the file will not take effect until the application is reloaded. Therefore, casbin provides a simple API to add new policy entries. The updated policy can be saved into the file (or other storage).
|
|
|
|
|
|
There are [storage adapters](https://casbin.org/docs/en/adapters) available, contributed by casbin users. One of them is [casbin-pg-adapter](https://github.com/MonedaCacao/casbin-pg-adapter) - which works with the [go-pg adapter](https://github.com/go-pg/pg), considered in this design as the ORM library for Stork. The casbin policy file uses CSV format. The way that the adapters seem to store the policy in the database reflects the CSV file structure. The policy is stored in the single table and this table is dropped and recreated (with updated policy) upon the policy update.
|
|
|
There are [storage adapters](https://casbin.org/docs/en/adapters) available, contributed by casbin users. One of them is [casbin-pg-adapter](https://github.com/casbin/casbin-pg-adapter) - which works with the [go-pg adapter](https://github.com/go-pg/pg), considered in this design as the ORM library for Stork. The casbin policy file uses CSV format. The way that the adapters seem to store the policy in the database reflects the CSV file structure. The policy is stored in the single table and this table is dropped and recreated (with updated policy) upon the policy update.
|
|
|
|
|
|
The [casbin API documentation](https://casbin.org/docs/en/management-api) describes all API calls being supported. It is possible to fetch policies for the particular user or group. It is also possible to check whether the user has certain permissions based on grouping policy or user specific policy.
|
|
|
|
... | ... | @@ -507,13 +507,13 @@ We already have the table `stork_users` which holds the information about the us |
|
|
```sql
|
|
|
postgres=# \d user_permission;
|
|
|
Table "public.user_permission"
|
|
|
Column | Type | Collation | Nullable | Default
|
|
|
Column | Type | Collation | Nullable | Default
|
|
|
---------------+---------+-----------+----------+-----------------------------------
|
|
|
id | integer | | not null | nextval('perms_id_seq'::regclass)
|
|
|
resource_name | text | | not null |
|
|
|
resource_id | integer | | |
|
|
|
user_id | integer | | |
|
|
|
access_list | json | | |
|
|
|
resource_name | text | | not null |
|
|
|
resource_id | integer | | |
|
|
|
user_id | integer | | |
|
|
|
access_list | json | | |
|
|
|
Foreign-key constraints:
|
|
|
"perms_user_id_fkey" FOREIGN KEY (user_id) REFERENCES stork_user(id)
|
|
|
```
|
... | ... | @@ -525,11 +525,11 @@ The ability to specify fain grained permissions for individual users is flexible |
|
|
```sql
|
|
|
postgres=# \d stork_user_group;
|
|
|
Table "public.stork_user_group"
|
|
|
Column | Type | Collation | Nullable | Default
|
|
|
Column | Type | Collation | Nullable | Default
|
|
|
-------------+---------+-----------+----------+----------------------------------------------
|
|
|
id | integer | | not null | nextval('stork_user_group_id_seq'::regclass)
|
|
|
name | text | | |
|
|
|
description | text | | |
|
|
|
name | text | | |
|
|
|
description | text | | |
|
|
|
Indexes:
|
|
|
"stork_user_group_pkey" PRIMARY KEY, btree (id)
|
|
|
Referenced by:
|
... | ... | @@ -541,13 +541,13 @@ The group is associated with a user specified set of permissions via the `group_ |
|
|
```sql
|
|
|
postgres=# \d group_permission;
|
|
|
Table "public.group_permission"
|
|
|
Column | Type | Collation | Nullable | Default
|
|
|
Column | Type | Collation | Nullable | Default
|
|
|
---------------+---------+-----------+----------+----------------------------------------------
|
|
|
id | integer | | not null | nextval('group_permission_id_seq'::regclass)
|
|
|
resource_name | text | | not null |
|
|
|
resource_id | integer | | |
|
|
|
group_id | integer | | |
|
|
|
access_list | json | | |
|
|
|
resource_name | text | | not null |
|
|
|
resource_id | integer | | |
|
|
|
group_id | integer | | |
|
|
|
access_list | json | | |
|
|
|
Indexes:
|
|
|
"group_permission_pkey" PRIMARY KEY, btree (id)
|
|
|
"group_id_idx" btree (group_id)
|
... | ... | @@ -562,10 +562,10 @@ Finally, individual users will be associated with the groups using the following |
|
|
```sql
|
|
|
postgres=# \d user_group_assoc;
|
|
|
Table "public.user_group_assoc"
|
|
|
Column | Type | Collation | Nullable | Default
|
|
|
Column | Type | Collation | Nullable | Default
|
|
|
----------+---------+-----------+----------+---------
|
|
|
user_id | integer | | not null |
|
|
|
group_id | integer | | not null |
|
|
|
user_id | integer | | not null |
|
|
|
group_id | integer | | not null |
|
|
|
Indexes:
|
|
|
"user_group_assoc_pkey" PRIMARY KEY, btree (user_id, group_id)
|
|
|
Foreign-key constraints:
|
... | ... | @@ -581,7 +581,7 @@ The organization of the ACL should facilitate the following typical use cases: |
|
|
- check the permissions of the user and/or group with respect to this particular resource?
|
|
|
- find the users (and groups) that have any privileges with respect to this resource and what these privileges are?
|
|
|
- check the permissions of that user in the system?
|
|
|
|
|
|
|
|
|
Below, we provide example queries for these use cases.
|
|
|
|
|
|
#### Use Case 1
|
... | ... | @@ -590,8 +590,8 @@ The following query selects all user specific permissions of the user `jkowal@ex |
|
|
|
|
|
```sql
|
|
|
SELECT u.email, s.content ->> 'prefix' AS subnet, p.access_list
|
|
|
FROM stork_user AS u
|
|
|
INNER JOIN user_permission AS p
|
|
|
FROM stork_user AS u
|
|
|
INNER JOIN user_permission AS p
|
|
|
ON u.id = p.user_id
|
|
|
INNER JOIN subnet AS s
|
|
|
ON CAST(s.content ->> 'subnet_id' AS INTEGER) = p.resource_id
|
... | ... | @@ -599,7 +599,7 @@ SELECT u.email, s.content ->> 'prefix' AS subnet, p.access_list |
|
|
```
|
|
|
|
|
|
```
|
|
|
email | subnet | access_list
|
|
|
email | subnet | access_list
|
|
|
--------------------+-----------+------------------------------------------
|
|
|
jkowal@example.org | 192.0.2.5 | [ "create", "read", "update", "delete" ]
|
|
|
```
|
... | ... | @@ -626,7 +626,7 @@ SELECT u.email, p.access_list |
|
|
```
|
|
|
|
|
|
```
|
|
|
email | access_list
|
|
|
email | access_list
|
|
|
--------------------+------------------------------------------
|
|
|
jkowal@example.org | [ "create", "read", "update", "delete" ]
|
|
|
```
|
... | ... | @@ -651,7 +651,7 @@ SELECT p.resource_table, p.resource_id, p.access_list |
|
|
```
|
|
|
|
|
|
```
|
|
|
resource_table | resource_id | access_list
|
|
|
resource_table | resource_id | access_list
|
|
|
----------------+-------------+------------------------------------------
|
|
|
subnet | 1 | [ "create", "read", "update", "delete" ]
|
|
|
```
|
... | ... | |