Agent registration appears to ignore --skip-tls-cert-verification with self-signed server certificate
Description: Stork agent appears to ignore both the STORK_AGENT_SKIP_TLS_CERT_VERIFICATION
environment variable and the --skip-tls-cert-verification
flag during registration with a server token. Regardless of these options, registration fails:
stork-agent@P:/$ env
STORK_AGENT_SERVER_URL=https://10.0.7.11:443/
AGENT_PORT=8080
STORK_AGENT_SERVER_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXX
STORK_AGENT_SKIP_TLS_CERT_VERIFICATION=true
AGENT_HOSTNAME=10.0.7.11
stork-agent@P:/$ STORK_AGENT_HOST="$AGENT_HOSTNAME:$AGENT_PORT" stork-agent register
INFO[2022-09-09 19:39:15] register.go:156 Agent token stored in /var/lib/stork-agent/tokens/agent-token.txt
INFO[2022-09-09 19:39:15] register.go:157 Agent key, agent token, and CSR (re)generated
INFO[2022-09-09 19:39:15] register.go:445 =============================================================================
INFO[2022-09-09 19:39:15] register.go:446 AGENT TOKEN: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
INFO[2022-09-09 19:39:15] register.go:447 =============================================================================
INFO[2022-09-09 19:39:15] register.go:452 Machine will be automatically registered using the server token
INFO[2022-09-09 19:39:15] register.go:453 Agent token is printed above for informational purposes only
INFO[2022-09-09 19:39:15] register.go:454 User does not need to copy or verify the agent token during registration via the server token
INFO[2022-09-09 19:39:15] register.go:455 It will be sent to the server but it is not directly used in this type of machine registration
INFO[2022-09-09 19:39:15] register.go:467 Try to register agent in Stork Server
ERRO[2022-09-09 19:39:15] register.go:470 problem registering machine: Post "https://10.0.7.11:443/api/machines": x509: certificate signed by unknown authority
FATA[2022-09-09 19:39:15] main.go:101 Registration failed
Note 1: I found that with these variables, STORK_AGENT_HOST
requires a port ONLY during registration, and requires the port to NOT be specified when running normally. This is not clear in the documentation.
Note 2: The stork server is running with a self-signed certificate. Relevant environment variables are STORK_REST_PORT=443
, STORK_REST_TLS_CERTIFICATE=/path/to/certificate.crt
, and STORK_REST_TLS_PRIVATE_KEY=/path/to/cert_priv.key
. STORK_REST_TLS_CA_CERTIFICATE
is unset to disable mutual TLS authentication.
Note 3: The Stork documentation says here that the default for STORK_AGENT_SKIP_TLS_CERT_VERIFICATION
is false
, but says here to set it to 1
to disable verification. I see here that it is parsed with strconv and that it can accept any of "1
, t
, T
, TRUE
, true
, True
, 0
, f
, F
, FALSE
, false
, False
". I suggest that the documentation chooses one of these and standardizes on it to avoid confusion, instead of using false
and 1
.
TLS certificate verification should not occur when disabled, which should suppress the error (or log without failure).
To Reproduce
Steps to reproduce the behavior:
- Build Stork server and agent from version 1.5.0 commit
75c7280a20682e02296da6bed71a4367fc4c0230
. - Configure Stork server for TLS as detailed in note 2. Be sure to allow execution by the unprivileged (undocumented except for one code snippet) stork-agent user (UID/GID 375 in my case) with
setcap 'cap_net_bind_service=+ep' $(which stork-server)
. Run the server - I do so inside ubuntu:latest (current Image ID2dc39ba059dc
) in host network mode. - Get the generated server token. I am using postgres with image ID
b37c2a6c1506
, so I usedocker exec -it stork-db psql -U stork stork -c "SELECT content FROM secret WHERE name = 'srvtkn';"
instead of the web interface. - Set the environment variables in the above snippet and attempt to register the agent. I have also been unable to register when I set
STORK_AGENT_SKIP_TLS_CERT_VERIFICATION
to any oftrue
,1
,false
,0
, or when I run the command as:STORK_AGENT_HOST="$AGENT_HOSTNAME:$AGENT_PORT" stork-agent register
,STORK_AGENT_HOST="$AGENT_HOSTNAME:$AGENT_PORT" stork-agent --skip-tls-cert-verification register
,STORK_AGENT_HOST="$AGENT_HOSTNAME:$AGENT_PORT" stork-agent register --skip-tls-cert-verification
, and any of these commands in combination with any of those values of the environment variable. I also see that the environment variable is picked up, as runningstork-agent --help
shows the "default" astrue
:
...
--skip-tls-cert-verification Skip TLS certificate verification when the Stork Agent connects to Kea over TLS and Kea uses self-signed certificates (default: true) [$STORK_AGENT_SKIP_TLS_CERT_VERIFICATION]
...
Expected behavior
The agent should not abort registration as a result of a self-signed x509 certificate in use by the Stork server.
Additional Information Logs from Stork server when I attempt registration:
2022/09/09 19:50:21 http: TLS handshake error from 10.0.7.11:57284: remote error: tls: bad certificate
2022/09/09 19:50:21 http: TLS handshake error from 10.0.7.11:57292: remote error: tls: bad certificate
cURL, showing that the Stork server webpage works:
[root@P ~]# curl -v --cacert /path/to/certificates/services_ca_cert.crt https://10.0.7.11
* Rebuilt URL to: https://10.0.7.11/
* Trying 10.0.7.11...
* TCP_NODELAY set
* Connected to 10.0.7.11 (10.0.7.11) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /path/to/certificates/services_ca_cert.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=Utah; L=Salt Lake City; O=Unknown; CN=10.0.7.11; emailAddress=Unknown
* start date: Sep 9 20:50:55 2022 GMT
* expire date: Sep 9 20:50:55 2023 GMT
* subjectAltName: host "10.0.7.11" matched cert's IP address!
* issuer: C=US; ST=Utah; L=Salt Lake City; O=Unknown; CN=localhost; emailAddress=Unknown
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* Using Stream ID: 1 (easy handle 0x55d2d14926b0)
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET / HTTP/2
> Host: 10.0.7.11
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/2 200
< accept-ranges: bytes
< content-type: text/html; charset=utf-8
< last-modified: Wed, 07 Sep 2022 22:51:23 GMT
< content-length: 7786
< date: Fri, 09 Sep 2022 20:58:41 GMT
<
* TLSv1.3 (IN), TLS app data, [no content] (0):
... Page body ...
* Connection #0 to host 10.0.7.11 left intact
Some initial questions
- Are you sure your feature is not already implemented in the latest Kea version? Yes.
- Are you sure what you would like to do is not possible using some other mechanisms? It may be, by adding my certificate to the system.
- Have you discussed your idea on kea-users or kea-dev mailing lists? No.
Describe the solution you'd like
A patch, explanation of what I am doing wrong, or otherwise resolution that allows me to register agents with self-signed certificates.
Describe alternatives you've considered
Hard-coding the Stork code to always skip TLS verification (though I don't see any glaring issues with the code after a cursory check), or adding the certificate to the system.
Participating in development Are you willing to participate in the feature development? Yes, tell me how I can help.