Skip to content
GitLab

HA connection with MT support · Changes

Page history
rename designs to Designs authored by Andrei Pavel's avatar Andrei Pavel
Hide whitespace changes
Inline Side-by-side
Designs/HA-connection-with-MT-support.md 0 → 100644
View page @ e3c1f9fd
**NOTE**: This page is obsolete. It was an early discussion. For the current approach for multithreading support in HA, see [this page](https://gitlab.isc.org/isc-projects/kea/-/wikis/designs/HA-MT-Design-for-Multi-threaded-Http-HA-traffic).
This document presents a design for a HA connection that can be used with MT.
Fell free to add comments, corrections and other ideas to the design.
### Current implementation and limitations.
```
HA lib <JSON over HTTP> CA <JSON over UNIX socket> Kea
one thread either one thread one thread
main thread in ST main thread main thread
or one of the
thread pool's
threads in MT
1 -> 2-3 -> 4
8 <- 6-7 <- 5
1 code and send HTTP request
2 receive and decode HTTP request
3 send json command
4 receive and execute command
5 send json response
6 receive json response
7 code and send HTTP response
8 receive and decode HTTP response
```
The HTTP connection remains opened for multiple pairs of <request|reply>, but each pair is handled sequential.
The referenced class is: HttpClient.
This is the first limitation, as new requests need to wait for already queued request to finish, before even being sent to the CA.
The HTTP connection is opened by the HA hook on demand, when a request is created for a specific endpoint.
The related code can be found in HttpClient::asyncSendRequest which uses a pool of connections, one for each endpoint. As mentioned, each connection is kept alive (will be recreated only on connection failure or lost connection events). Each connection from the pool calls queueRequestInternal which handles the creation of the new HTTP connection or just the queuing of the new request.
The CA basically just decodes the HTTP request and sends it as JSON command over the UNIX socket to the Kea server.
It does this using one single thread, the IO thread (also main thread).
Each response from the Kea server is packed inside a HTTP response and send back. Only at this stage, the CA can handle another HTTP request.
The CA HTTP listener is created on: CtrlAgentProcess::configure which uses CtrlAgentResponseCreatorFactory as a factory to create responses using CtrlAgentResponseCreator and the overloaded function CtrlAgentResponseCreator::
createDynamicHttpResponse. This function will call CtrlAgentCommandMgr::handleCommand which actually calls CtrlAgentCommandMgr::forwardCommand which communicates with the Kea process.
The HTTP connection is created on HttpListenerImpl::accept.
The CA opens a connection over UNIX socket to Kea on each command (ref: CtrlAgentCommandMgr::forwardCommand), and closes the connection after receiving the response (lambda function in ref: CtrlAgentCommandMgr::forwardCommand). This also happen on the Kea side (ref: Connection::sendHandler).
The Kea server listens on a UNIX socket and handles each request (ref: CommandMgrImpl::openCommandSocket). On each new connection, the lambda in CommandMgrImpl::doAccept is called and the new connection is added to the connection pool. Each request causes a command to be run (ref: Connection::receiveHandler) on the IO thread in the Kea process (main thread) and also closes the connection after sending the response (ref: Connection::sendHandler).
Kea is also a bottleneck here as there is only one thread handling the requests on the UNIX connection. Even if there are multiple connection to the UNIX socket, they are handled in sequence.
### Possible parallelization improvements.
There are several ways to improve the connectivity and processing of the HA requests which require remote Kea server lease updates.
##### 1. Parallel handling of the commands in Kea
The most basic change that needs to be addressed is the Kea ability to handle multiple commands in parallel. Currently, all connections are handled by the main thread:
```
CommandMgr::instance().setIOService(getIOService());
```
There is a need to create multiple IO services, one per thread, or at least move the handling of the CommandMgr::instance().processCommand(cmd) in Connection::receiveHandler on a different thread with the continuation of Connection::doSend.
FD comment: the problem is not in the I/O itself which is asynchronous but in the handling of commands. And BTW creating multiple IO services do no work as the socket can't be attached to more than one IO service.
This will result in:
```
HA lib <JSON over HTTP> CA <JSON over UNIX socket> Kea <JSON over UNIX socket> socat
one thread either one thread multiple threads
main thread in ST main thread one for each
or one of the command, or one
thread pool's from a thread pool
threads in MT
1 -> 2-3 -> 4
8 <- 6-7 <- 5
2' <- 1'
3' -> 4'
1 code and send HTTP request
2 receive and decode HTTP request
3 send json command
4 receive and execute command
5 send json response
6 receive json response
7 code and send HTTP response
8 receive and decode HTTP response
1' send json command
2' receive and execute command
3' send json response
4' receive json response
```
(4, 5), and (2', 3') can be executed in parallel as opposed to in sequence in current implementation.
This would mean that the Kea server could handle multiple connections to the UNIX socket in parallel, and out of order, as each request is handled on a different connection.
FD changed into: This would mean that the Kea server could execute multiple commands from the UNIX socket in parallel, and out of order, as each request is handled on a different connection.
##### 2. Parallel send and receive of HTTP requests/replies
Now, looking at the other end, all HA threads use one HTTP connection to send requests to CA.
This can be parallelized, so that each connection uses it's own internal state and could send requests in parallel.
This would mean that HttpClient::asyncSendRequest would push the request on one of the connections available for the endpoint, not only one per url, but one per thread per url.
This would mean that the CA can also handle multiple incoming HTTP requests, one for each corresponding HA thread in Kea. This would require creating an IO service per thread, so that each HTTP connection can change internal state independently and in parallel.
FD comment: the IO service supports multi-threading so there is no need for an IO service per thread. The only requirement is to protect request queue(s) against simultaneous push. Note this does not give full parallelism
because each HTTP connection is sequential: HTTP 1.x does not provide a way to match requests and responses. To summary connections are in parallel but not out-of-order inside a connections. Another point: even this trivially can lead to more contention one queue is better than a queue per connection.
##### 3. Asynchronous send and receive of HTTP requests/replies
Another way to add some kind of parallelism is to simply keep one connection and each HTTP request/response would use a sequence ID for each request and add that to the resolved response, so that the asynchronous property of MT processing is kept (out of order execution), even if using a single connection. This means that requests and responses could be send and received in parallel/interleaved, without waiting, and be matched by their ID on their path way:
```
HA lib <JSON over HTTP> CA
send 1, 2, 3, 4 -> receive and queue 1, 2, 3, 4
receive 3, 2, 4, 1 <- process and send 3, 2, 4, 1
```
In this case, commands would be processed in parallel, but the communication would be serialized.
To support asynchronous exchanges between HA and CA would also mean sending asynchronous exchanges between CA and Kea, but this is already handled by the fact that Connection::receiveHandler would create a thread for each connection or an IO service per thread, and the fact that Kea uses one thread per UNIX connection.
Responses for commands would be available as soon as possible, the only thing to do is to match the response with the request ID, and send the response over an available HTTP connection.
FD comment: this is a supported by nobody extension of the HTTP 1.x protocol. So it is not an available option.
##### 4. Direct connection between HA thread and Kea using TCP/IP
The simplest way to accelerate the handling of HA requests with remote Kea lease updates would imply the implementation of `1. Parallel handling of the commands in Kea` and adding TCP/IP sockets to communicate between HA and Kea.
FD comment: my comment about commands vs connections even more applies here. So the connection handling will be asynchronous and by the main thread, and command execution will be in parallel using for instance a thread pool.
```
HA lib <JSON over TCP/IP> Kea
one thread either multiple threads
main thread in ST one for each
or one of the command, or one
thread pool's from a thread pool
threads in MT
1 -> 2
4 <- 3
1' -> 2'
4' <- 3'
1 send json command
2 receive and execute command
3 send json response
4 receive json response
1' send json command
2' receive and execute command
3' send json response
4' receive json response
```
All steps can be executed in parallel, independent of each other.
The obvious change would be the replacement of the HTTP connection in HA and the UnixDomainSocketEndpoint in Kea with the TCPEndpoint to handle remote control channel commands for lease updates.
FD comment: I removed UDPEndpoint because a request or response size is unbound.
The code from CA to communicate with Kea can be used in HA also:
```
// Forward command and receive reply.
IOServicePtr io_service(new IOService());;
ClientConnection conn(*io_service);
boost::system::error_code received_ec;
ConstJSONFeedPtr received_feed;
conn.start(ClientConnection::SocketPath(socket_name),
ClientConnection::ControlCommand(command->toWire()),
[&io_service, &received_ec, &received_feed]
(const boost::system::error_code& ec, ConstJSONFeedPtr feed) {
// Capture error code and parsed data.
received_ec = ec;
received_feed = feed;
// Got the IO service so stop IO service. This causes to
// stop IO service when all handlers have been invoked.
io_service->stopWork();
}, ClientConnection::Timeout(TIMEOUT_AGENT_FORWARD_COMMAND));
io_service->run();
```
FD comment about this piece of code: it is the place where the CA was made sequential: everything is done in the local IO service inside the call of its run() method (the caller is blocked until it returns eventually on timeout).
If this needs to be asynchronous/nonblocking, adding the corresponding lambda as a task to a thread pool would be sufficient.
This implementation would require a configuration option on the HA lib and also on the Kea side to be able to specify the socket connection endpoint (TCP/IP - basically IP + port). (FD comment: can't parse the parenthesis content: I believe it is IP address and TCP port).
This implementation would also bypass the CA and less effort would be spend on implementing this future.
This would also keep current functionality, and Kea and CA can work just as before, but not for HA lease updates. This would also not require HTTP protocol (coding/encoding), maintaining the HTTP connection and internal state thread safe, and would not require providing any 'asynchronous' property to the HTTP connection.
FD comment: (this assumes that connections are kept after sending a response) it allows to send requests and responses in any order: the only constraint for a simple code is to have the whole request or response in one block. It does not require parallel connections too: in fact the only advantage of parallel connections is more transport throughput: it is not a required property and if the network itself is slow it won't really help too.
The fact that Kea closes connections after each command channel connection means that there can be spawned as many threads as there are commands, but a thread pool with a fixed size can also be used to handle incoming commands, which can this way be easily queued and processed.
The security concerns about the connection between HA and Kea peer can be solved by using other security methods (openvpn, IPsec, isolated network, etc) and are not discussed in this document.
FD comments: openvpn is a product (I'll stay polite and I shan't put my opinion about it here), IPsec (its spelling
is mandated in its RFC) is infrastructure protection.
##### 5. Direct connection between HA thread and Kea using HTTP
Another approach consists in keeping the HA HTTP connection, but instead of connecting to the CA, the HTTP listener would be created on Kea side. This implies implementing `2. Parallel send and receive of HTTP requests/replies` and also implementing `1. Parallel handling of the commands in Kea` with a HTTP listener instead.
```
HA lib <JSON over HTTP> Kea
one thread either multiple threads
main thread in ST one for each
or one of the command, or one
thread pool's from a thread pool
threads in MT
1 -> 2
4 <- 3
1' -> 2'
4' <- 3'
1 code and send HTTP request
2 receive, decode and execute command
3 code and send HTTP response
4 receive and decode HTTP response
1' code and send HTTP request
2' receive, decode and execute command
3' code and send HTTP response
4' receive and decode HTTP response
```
Just as before, all steps can be executed in parallel, independent of each other.
The drawback of this implementation would be the use of HTTP, which needs packing/unpacking, and having no real value beside being able to redirect it through a HTTP proxy with HTTPS (secure transport). However, adding another element in the connection path (the HTTP/HTTPS proxy) will slow down the connection, possibly diminishing the actual performance gain.
FD comment: HTTP has the big advantage to go through all firewalls and middle boxes. And it can be made secure on the network up to (but not including) the node where Kea runs. I am afraid it is its only advantages but they are very important...
This is a WIP.
Fell free to add comments, corrections and other ideas to the design.