Commit 5af66c86 authored by Michal Nowikowski's avatar Michal Nowikowski
Browse files

[#481] agent server TLS - part 2

- added code for registering a machine in the server and
  performing key and certs exchange but ut us not used fully yet
- added server-token and agent-token based agent authorizations
  but for now it is disabled
- added API for presenting and regenerating server token but it
  is not used in UI yet - updated content of reference agent.env
  agent config file
parent deaf7a4d
* 126 [func] godfryd
This is the second part of secured agent-server channel
implementation. Added code for registering a machine in the server
and performing key and certs exchange but it is not used fully
yet. Added server-token and agent-token based agent
authorizations. Added REST API for presenting and regenerating
server token, but it is not used in UI yet. Updated content of
reference agent.env agent config file.
(Gitlab #481)
* 125 [func] marcin
Assign friendly names to the apps monitored in Stork. The apps'
......
......@@ -482,7 +482,11 @@ task :unittest_backend => [GO, RICHGO, MOCKERY, MOCKGEN, :build_server, :build_a
# Those two are tested in backend/server/server_test.go, in TestCommandLineSwitches*
# However, due to how it's executed (calling external binary), it's not detected
# by coverage.
'ParseArgs', 'NewStorkServer']
'ParseArgs', 'NewStorkServer',
# TODO: DISABLED FOR NOW
'getAgentAddrAndPortFromUser',
]
if ENV['short'] == 'true'
ignore_list.concat(['setupRootKeyAndCert', 'setupServerKeyAndCert', 'SetupServerCerts'])
end
......
......@@ -43,6 +43,51 @@
readOnly: true
type: string
NewMachineReq:
type: object
required:
- address
- agentCSR
properties:
address:
type: string
agentPort:
type: integer
agentCSR:
type: string
description: Agent Certificate Signing Request.
serverToken:
type: string
description: >-
A token that is issued by the Stork server. It can be taken
from Machines page. If it is provided then an agent will
be immediately authorized in the server and will be operational.
If it is empty then agentToken must be provided.
agentToken:
type: string
description: >-
A token that is generated by an agent. An agent traces it in
the logs during startup and stores it in
/var/lib/stork-agent/tokens/agent-token.txt. A machine added
this way to the Stork server requires separate authorization
that can be made in the Stork server UI or using server API.
If it is empty then serverToken must be provided.
NewMachineResp:
type: object
properties:
id:
type: integer
description: The machine ID.
serverCACert:
type: string
readOnly: true
description: Server's CA certificate.
agentCert:
type: string
readOnly: true
description: Signed agent's certificate.
Machine:
type: object
required:
......@@ -55,6 +100,10 @@
type: string
agentPort:
type: integer
authorized:
type: boolean
agentToken:
type: string
agentVersion:
type: string
readOnly: true
......
......@@ -18,6 +18,10 @@
in: query
description: Limit returned list of machines to these which provide given app, possible values 'bind' or 'kea'.
type: string
- name: authorized
in: query
description: Indicate if authorized or unauthorized machines should be returned.
type: boolean
responses:
200:
description: List of machines
......@@ -30,22 +34,27 @@
post:
summary: Add new machine.
description: >-
To create new machine at least one parameter is needed: address.
There are optional parameters as well.
Register new machine in the server. It requires two
parameters: address and agentCSR. It also requires one of two
other parameters: serverToken or agentToken depending or the
registration method selected.
operationId: createMachine
# security disabled because anyone can add machine but it still requires
# either server token or manual authorization in web ui
security: []
tags:
- Services
parameters:
- name: machine
in: body
description: Machine
description: New machine basic information including CSR.
schema:
$ref: '#/definitions/Machine'
$ref: '#/definitions/NewMachineReq'
responses:
200:
description: Machine information
description: Registration information
schema:
$ref: "#/definitions/Machine"
$ref: '#/definitions/NewMachineResp'
default:
description: generic error response
schema:
......@@ -118,6 +127,42 @@
schema:
$ref: "#/definitions/ApiError"
/machines/{id}/ping:
post:
summary: Check connectivity with machine.
operationId: pingMachine
security: []
tags:
- Services
parameters:
- in: path
name: id
type: integer
required: true
description: Machine ID.
- in: body
name: ping
description: >-
Body should contain proper server or agent token. If none
of them match the values stored by the server, the ping is
rejected.
schema:
type: object
properties:
serverToken:
type: string
description: Server access token.
agentToken:
type: string
description: Agent token.
responses:
200:
description: The response is empty.
default:
description: generic error response
schema:
$ref: "#/definitions/ApiError"
/machines/{id}/state:
get:
summary: Get machine's runtime state.
......@@ -140,6 +185,49 @@
schema:
$ref: "#/definitions/ApiError"
/machines-server-token:
get:
summary: Get server token for registering machines.
description: >-
The server token is used in server token machine registration.
operationId: getMachinesServerToken
tags:
- Services
responses:
200:
description: Current server token.
schema:
type: object
properties:
token:
type: string
description: Current server token.
default:
description: generic error response
schema:
$ref: "#/definitions/ApiError"
put:
summary: Regenerate server token.
description: >-
When there is probability that current server token
leaked then it should be regenerated.
operationId: regenerateMachinesServerToken
tags:
- Services
responses:
200:
description: Regenerated server token.
schema:
type: object
properties:
token:
type: string
description: Regenerated server token.
default:
description: generic error response
schema:
$ref: "#/definitions/ApiError"
/apps:
get:
summary: Get list of apps.
......
......@@ -4,6 +4,8 @@ import (
"bytes"
"compress/gzip"
"context"
"crypto/tls"
"crypto/x509"
"fmt"
"io/ioutil"
"net"
......@@ -11,22 +13,19 @@ import (
"runtime"
"strings"
"github.com/pkg/errors"
"github.com/shirou/gopsutil/host"
"github.com/shirou/gopsutil/load"
"github.com/shirou/gopsutil/mem"
log "github.com/sirupsen/logrus"
"github.com/urfave/cli/v2"
"google.golang.org/grpc"
"google.golang.org/grpc/security/advancedtls"
"isc.org/stork"
agentapi "isc.org/stork/api"
)
// Stork Agent settings.
type Settings struct {
Host string `long:"host" description:"the IP or hostname to listen on for incoming Stork server connection." env:"STORK_AGENT_ADDRESS"`
Port int `long:"port" description:"the TCP port to listen on for incoming Stork server connection." default:"8080" env:"STORK_AGENT_PORT"`
}
// Global Stork Agent state.
type StorkAgent struct {
Settings *cli.Context
......@@ -50,8 +49,6 @@ func NewStorkAgent(settings *cli.Context, appMonitor AppMonitor) *StorkAgent {
httpClient := NewHTTPClient()
server := grpc.NewServer()
logTailer := newLogTailer()
sa := &StorkAgent{
......@@ -59,7 +56,6 @@ func NewStorkAgent(settings *cli.Context, appMonitor AppMonitor) *StorkAgent {
AppMonitor: appMonitor,
HTTPClient: httpClient,
RndcClient: rndcClient,
server: server,
logTailer: logTailer,
keaInterceptor: newKeaInterceptor(),
}
......@@ -69,6 +65,101 @@ func NewStorkAgent(settings *cli.Context, appMonitor AppMonitor) *StorkAgent {
return sa
}
// Read the latest root CA cert from file for Stork server's cert verification.
func getRootCertificates(params *advancedtls.GetRootCAsParams) (*advancedtls.GetRootCAsResults, error) {
certPool := x509.NewCertPool()
ca, err := ioutil.ReadFile(RootCAFile)
if err != nil {
err = errors.Wrapf(err, "could not read CA certificate: %s", RootCAFile)
log.Errorf("%+v", err)
return nil, err
}
// append the client certificates from the CA
if ok := certPool.AppendCertsFromPEM(ca); !ok {
err = errors.New("failed to append client certs")
log.Errorf("%+v", err)
return nil, err
}
log.Printf("loaded CA cert: %s\n", RootCAFile)
return &advancedtls.GetRootCAsResults{
TrustCerts: certPool,
}, nil
}
// Read the latest Stork agent's cert from file for presenting its identity to the Stork server.
func getIdentityCertificatesForServer(info *tls.ClientHelloInfo) ([]*tls.Certificate, error) {
keyPEM, err := ioutil.ReadFile(KeyPEMFile)
if err != nil {
err = errors.Wrapf(err, "could not load key PEM file: %s", KeyPEMFile)
log.Errorf("%+v", err)
return nil, err
}
certPEM, err := ioutil.ReadFile(CertPEMFile)
if err != nil {
err = errors.Wrapf(err, "could not load cert PEM file: %s", CertPEMFile)
log.Errorf("%+v", err)
return nil, err
}
certificate, err := tls.X509KeyPair(certPEM, keyPEM)
if err != nil {
err = errors.Wrapf(err, "could not setup TLS key pair")
log.Errorf("%+v", err)
return nil, err
}
log.Printf("loaded server cert: %s and key: %s\n", CertPEMFile, KeyPEMFile)
return []*tls.Certificate{&certificate}, nil
}
// Prepare gRPC server with configured TLS.
func newGRPCServerWithTLS() (*grpc.Server, error) {
// Prepare structure for advanced TLS. It defines hook functions
// that dynamically load key and cert from files just before establishing
// connection. Thanks to this if these files changed in meantime then
// always latest version for new connections is used.
// Beside that there is enabled client authentication and forced
// cert and host verification.
options := &advancedtls.ServerOptions{
// pull latest root CA cert for stork server cert verification
RootOptions: advancedtls.RootCertificateOptions{
GetRootCertificates: getRootCertificates,
},
// pull latest stork agent cert for presenting its identity to stork server
IdentityOptions: advancedtls.IdentityCertificateOptions{
GetIdentityCertificatesForServer: getIdentityCertificatesForServer,
},
// force stork server cert verification
RequireClientCert: true,
// check cert and if it matches host IP
VType: advancedtls.CertAndHostVerification,
}
creds, err := advancedtls.NewServerCreds(options)
if err != nil {
return nil, errors.Wrapf(err, "cannot create server credentials for TLS")
}
srv := grpc.NewServer(grpc.Creds(creds))
return srv, nil
}
// Setup the agent as gRPC server endpoint.
func (sa *StorkAgent) Setup() error {
// server, err := newGRPCServerWithTLS() // TODO: NOT USED FOR NOW
// if err != nil {
// return err
// }
server := grpc.NewServer()
sa.server = server
return nil
}
// Respond to ping request from the server. It assures the server that
// the connection from the server to client is established. It is used
// in server token registration procedure.
func (sa *StorkAgent) Ping(ctx context.Context, in *agentapi.PingReq) (*agentapi.PingRsp, error) {
rsp := agentapi.PingRsp{}
return &rsp, nil
}
// Get state of machine.
func (sa *StorkAgent) GetState(ctx context.Context, in *agentapi.GetStateReq) (*agentapi.GetStateRsp, error) {
vm, _ := mem.VirtualMemory()
......
......@@ -4,11 +4,13 @@ import (
"bytes"
"compress/gzip"
"context"
"crypto/tls"
"fmt"
"io/ioutil"
"math/rand"
"os"
"os/exec"
"path"
"strings"
"testing"
......@@ -16,7 +18,9 @@ import (
log "github.com/sirupsen/logrus"
"github.com/stretchr/testify/require"
"github.com/urfave/cli/v2"
"google.golang.org/grpc/security/advancedtls"
"gopkg.in/h2non/gock.v1"
"isc.org/stork"
agentapi "isc.org/stork/api"
)
......@@ -67,6 +71,7 @@ func setupAgentTest(rndc CommandExecutor) (*StorkAgent, context.Context) {
logTailer: newLogTailer(),
keaInterceptor: newKeaInterceptor(),
}
sa.Setup()
ctx := context.Background()
return sa, ctx
}
......@@ -91,6 +96,7 @@ func makeAccessPoint(tp, address, key string, port int64) (ap []AccessPoint) {
})
}
// Check if NewStorkAgent can be invoked and sets SA members.
func TestNewStorkAgent(t *testing.T) {
fam := &FakeAppMonitor{}
var settings cli.Context
......@@ -100,6 +106,16 @@ func TestNewStorkAgent(t *testing.T) {
require.NotNil(t, sa.RndcClient)
}
// Check if an agent returns a response to a ping message..
func TestPing(t *testing.T) {
sa, ctx := setupAgentTest(mockRndc)
args := &agentapi.PingReq{}
rsp, err := sa.Ping(ctx, args)
require.NoError(t, err)
require.NotNil(t, rsp)
}
// Check if GetState works.
func TestGetState(t *testing.T) {
sa, ctx := setupAgentTest(mockRndc)
......@@ -578,3 +594,203 @@ func TestCommandLineVersion(t *testing.T) {
require.Equal(t, ver, stork.Version)
}
}
// Checks if getRootCertificates:
// - returns an error if the cert file doesn't exist,
// - returns an error if the cert contents are invalid,
// - reads and returns correct certificate successfully.
func TestGetRootCertificates(t *testing.T) {
params := &advancedtls.GetRootCAsParams{}
// missing cert file error
_, err := getRootCertificates(params)
require.EqualError(t, err, "could not read CA certificate: /var/lib/stork-agent/certs/ca.pem: open /var/lib/stork-agent/certs/ca.pem: no such file or directory")
// prepare temp dir for cert files
tmpDir, err := ioutil.TempDir("", "reg")
require.NoError(t, err)
defer os.RemoveAll(tmpDir)
os.Mkdir(path.Join(tmpDir, "certs"), 0755)
RootCAFile = path.Join(tmpDir, "certs/ca.pem")
// store bad cert
err = ioutil.WriteFile(RootCAFile, []byte("CACertPEM"), 0600)
require.NoError(t, err)
_, err = getRootCertificates(params)
require.EqualError(t, err, "failed to append client certs")
// store correct cert
var CACertPEM []byte = []byte(`-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIBATANBgkqhkiG9w0BAQsFADAzMQswCQYDVQQGEwJVUzES
MBAGA1UEChMJSVNDIFN0b3JrMRAwDgYDVQQDEwdSb290IENBMB4XDTIwMTIwODA4
MDc1M1oXDTMwMTIwODA4MDgwM1owMzELMAkGA1UEBhMCVVMxEjAQBgNVBAoTCUlT
QyBTdG9yazEQMA4GA1UEAxMHUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIP
ADCCAgoCggIBALgcYkndHQGFmLk8yi8+yetaCBI1cLG/nm+hwjh5C2rh3lqqDziG
qRmcITxkEbCFujbxJzlaXop1MeXwg2YJMky3WM1GWomVKv3jOVR+GkQG70pp0qpt
JmU2CuXoNhwMFA0H22CG8pPRiilUGPI7RLXaLWpA8D+AslfPHR9TG00HbJ86Bi3g
m4/uPiGdcHS6Q+wmKQRsKs6wAKSmlCrvmaKfmVOkxpuKyuKgjmIKoCwY3gYL1T8L
idvVePvbP/Z2SRQOVbSV8eMaYuk+uFwGKq8thLHs8bIEKhrIGlzDss6ZlPotTi2V
I6e6lb06oFuCSfhBaiHPw2sldwYvE/I8MkWUAuWtBgNvVE/e64FgJb1lGIzJpYMj
5jUp9Z13INsXy9zA8nKyZAK4fI6vlQGRg3bERn+S4Q6HXQor9Ma8QWxsqbdiC9dt
pxpzyx11tWg0jEgzCEBfk9IZjlGqyCdX5Z9pshHkQZ9VeK+DG0s6tYEm7BO1ssQD
+qbJS2PJq4Cwe82a6gO+lDz8A+xiXk8dJeTb8hf/c1NY192rqSLewI8oaHOLKEQg
XNSPEEkQqtIqn92Y5oKhLYKmYkwfOgldpj0XQQ3YwUnsOCfy2wRVNRg6VYnbjca8
rSy58t2MfovKWz9UcKhpnXefSdMgR7VhGv0ekDddGIfONn153uyjN/LpAgMBAAGj
NTAzMBIGA1UdEwEB/wQIMAYBAf8CAQEwHQYDVR0OBBYEFILkrDPZAlboeF+nav7C
Rf7nN1W+MA0GCSqGSIb3DQEBCwUAA4ICAQCDfvIgo70Y0Mi+Rs0mF6114z2gGQ7a
7/VnxV9w9uIjuaARq42E2DemFs5e72tPIfT9UWncgs5ZfyO5w2tjRpUOaVCSS5VY
93qzXBfTsqgkrkVRwec4qqZxpNqpsL9u2ZIfsSJ3BJWFV3Zq/3cOrDulfR5bk0G4
hYo/GDyLHjNalBFpetJSIk7l0VOkr2CBUvxKBOP0U1IQGXd+NL/8zW6UB6OitqNL
/tO+JztOpjo6ZYKJGZvxyL/3FUsiHmd8UwqAjnFjQRd3w0gseyqWDgILXQaDXQ5D
vs2oK+HheJv4h6CdrcIdWlWRKoZP3odZyWB0l31kpMbgYC/tMPYebG6mjPx+/S4m
7L+K27zmm2wItUaWI12ky2FPgeW78ALoKDYWmQ+CnpBNE1iFUf4qRzmypu77DmmM
bLgLFj8Bb50j0/zciPO7+D1h6hCPxwXdfQk0tnWBqjImmK3enbkEsw77kF8MkNjr
Hka0EeTt0hyEFKGgJ7jVdbjLFnRzre63q1GuQbLkOibyjf9WS/1ljv1Ps82aWeE+
rh78iXtpm8c/2IqrI37sLbAIs08iPj8ULV57RbcZI7iTYFIjKwPlWL8O2U1mopYP
RXkm1+W4cMzZS14MLfmacBHnI7Z4mRKvc+zEdco/l4omlszafmUXxnCOmqZlhqbm
/p0vFt1oteWWSQ==
-----END CERTIFICATE-----`)
err = ioutil.WriteFile(RootCAFile, CACertPEM, 0600)
require.NoError(t, err)
// all should be ok
result, err := getRootCertificates(params)
require.NoError(t, err)
require.NotNil(t, result)
require.NotNil(t, result.TrustCerts)
}
// Checks if getIdentityCertificatesForServer:
// - returns an error if the key file doesn't exist,
// - returns an error if the key or cert contents are invalid,
// - reads and returns correct key and certificate pair successfully.
func TestGetIdentityCertificatesForServer(t *testing.T) {
info := &tls.ClientHelloInfo{}
// missing key files
_, err := getIdentityCertificatesForServer(info)
require.EqualError(t, err, "could not load key PEM file: /var/lib/stork-agent/certs/key.pem: open /var/lib/stork-agent/certs/key.pem: no such file or directory")
// prepare temp dir for cert files
tmpDir, err := ioutil.TempDir("", "reg")
require.NoError(t, err)
defer os.RemoveAll(tmpDir)
os.Mkdir(path.Join(tmpDir, "certs"), 0755)
os.Mkdir(path.Join(tmpDir, "tokens"), 0755)
KeyPEMFile = path.Join(tmpDir, "certs/key.pem")
CertPEMFile = path.Join(tmpDir, "certs/cert.pem")
// store bad content to files
err = ioutil.WriteFile(KeyPEMFile, []byte("KeyPEMFile"), 0600)
require.NoError(t, err)
err = ioutil.WriteFile(CertPEMFile, []byte("CertPEMFile"), 0600)
require.NoError(t, err)
_, err = getIdentityCertificatesForServer(info)
require.EqualError(t, err, "could not setup TLS key pair: tls: failed to find any PEM data in certificate input")
// store proper content
var certPEM []byte = []byte(`-----BEGIN CERTIFICATE-----
MIIGLTCCBBWgAwIBAgIBAjANBgkqhkiG9w0BAQsFADAzMQswCQYDVQQGEwJVUzES
MBAGA1UEChMJSVNDIFN0b3JrMRAwDgYDVQQDEwdSb290IENBMB4XDTIwMTIwODA4
MDc1NloXDTMwMTIwODA4MDgwNlowRjELMAkGA1UEBhMCVVMxEjAQBgNVBAoTCUlT
QyBTdG9yazEPMA0GA1UECxMGc2VydmVyMRIwEAYDVQQDEwlsb2NhbGhvc3QwggIi
MA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDR8yndmAonFo0dKWS3WQ3r60lI
wKPOZwsdJy+2+eNrmZixYJ+CdlvH3/AVSBRJfYx14NFrHcRUsbW+hn63kUwT3XHl
uLTs+QJWSaWa1zTLTJqiaEiPZI/xliQrTYoAV00jJip7CDWr0xpAPpBwJmhJLrlw
nxxZ6XlYLlGjyp+aImugYVQ+3xs4p18LcAmwf/+CyCPdp0rs6bUIEmo99DwLvI1a
vWDkbzT3JAVgk3Kc4Jp3eZ/gGRWBBa0eSM5zr11G3xOouPFpe++epMsLdjrYgnt1
PZBy8DPi5hL/7ltdfdWGvGkIeq1Y0n987P482nOizYoHhrSPKbz+dL3e0ifIvxUU
VrGmCnSefm4cwxW7GzDAZzUwZGa/qk24oEPeAi4zDrUeSdK6WTjFev+g6PTQDjif
L1jYZHjxyn0+itDzcHqU9lIZUT5CzdenOhEEu3StUskoHOlq3tz2bkG1hxnHX/CT
bczbx1ave9XNSnZw3lCoAPGiL+Ra9Zaov+VVfhTTMv4uxGrjOV4dDUTveu6mc+75
E5mLBjmkGjtsD3H3e/xHTIdiOZd0emgr4PD8yXQqbDKybcvOOhLZuNFQPIE4gqzy
GMb9BniOkECASLNBKgZcmtibkdwIghQATh+WbYhhOx+DyY+Dd/tGLE1Q+Wf5sd2O
c/C59W6zDOKfmXmNXwIDAQABo4IBNzCCATMwHQYDVR0OBBYEFBI3C/apKHAgS+U6
S29CoHJIZ80kMB8GA1UdIwQYMBaAFILkrDPZAlboeF+nav7CRf7nN1W+MIHwBgNV
HREEgegwgeWCCWxvY2FsaG9zdIIWbG9jYWxob3N0LmxvY2FsZG9tYWluLoIKbG9j
YWxob3N0NIIYbG9jYWxob3N0NC5sb2NhbGRvbWFpbjQuhwR/AAABhwTAqACXhwQK
lfcBhwQKLToBhwQKAAMBhwTAqHoBhwSsHQABhwSsEwABhwSsGwABhwSsGgABhwSs
EQABhwSsEgABhwTAqDIBhwSsHAABhxD+gAAAAAAAANrXH7RB719lhxAgAQ24AAEA
AAAAAAAAAAABhxD+gAAAAAAAAABCov/+0+/QhxD+gAAAAAAAAAAAAAAAAAABMA0G
CSqGSIb3DQEBCwUAA4ICAQCDcQhC1ecL28xcDhpJZULO66MwYesT9NmcpHL9VlG2
9tFcgo4Tyac+OT4BaQVwp9w/CCuGKbzUzY+EOaIF8OufoXeRJsf0g31hDqB/V/yv
BuxTH+q6S+9SrYV1Hf+mHfr36/MKH6Zwd8uEwjphhkIaq9y/m8gGLMHQ9a4u/pBx
2+GO9awT/9ZAtgO75kW7QB3GKJP6rd43DZ7+ypsiD39oVjTbOA7ET5wqNtzeB/nR
VD2OtZcXIUhWpgZWUl3+++PXrIB0N+jDAhWTyexhb2djCCfI6WRB7SY+59dX8pta
zmtwmadl7Z2nVDSTPRBBdQQ1dZwwKWDN4omfXmuGk6jvc2PYF+lUUlovhGmXzWc+
0ZTP4WzNuvn3iG0Z5ftgvSaTTKz1+e/RgfjvWRa4b2Lfo11gZcO5G4DYT0LK7Pho
sPEjCJa322MOS28UXQ3v0I5WQwn4k7iSZro+TQbWFORzJn7TL7Ov4Smkm7lpyHtp
xdU83aRjSN5/346xGR10Dx7vxvIAWMIx9IQKfFy48dAHiYSAWvW0KpBa5f0P7Ng0
TjJxMspTfL1UmI4vXP68tYRvThQbNNJxOviNmV0XBiKgQW5bD01j/KwpAD3/8ean
7tRAvfllA+b7dbjZ7ZDBFGJ1ie7sVNzvf/DKkgyxZYzrrJmUKZb2o0saAAw9OsTc
wQ==
-----END CERTIFICATE-----`)
var keyPEM []byte = []byte(`-----BEGIN PRIVATE KEY-----
MIIJQgIBADANBgkqhkiG9w0BAQEFAASCCSwwggkoAgEAAoICAQDR8yndmAonFo0d
KWS3WQ3r60lIwKPOZwsdJy+2+eNrmZixYJ+CdlvH3/AVSBRJfYx14NFrHcRUsbW+
hn63kUwT3XHluLTs+QJWSaWa1zTLTJqiaEiPZI/xliQrTYoAV00jJip7CDWr0xpA
PpBwJmhJLrlwnxxZ6XlYLlGjyp+aImugYVQ+3xs4p18LcAmwf/+CyCPdp0rs6bUI
Emo99DwLvI1avWDkbzT3JAVgk3Kc4Jp3eZ/gGRWBBa0eSM5zr11G3xOouPFpe++e
pMsLdjrYgnt1PZBy8DPi5hL/7ltdfdWGvGkIeq1Y0n987P482nOizYoHhrSPKbz+
dL3e0ifIvxUUVrGmCnSefm4cwxW7GzDAZzUwZGa/qk24oEPeAi4zDrUeSdK6WTjF
ev+g6PTQDjifL1jYZHjxyn0+itDzcHqU9lIZUT5CzdenOhEEu3StUskoHOlq3tz2
bkG1hxnHX/CTbczbx1ave9XNSnZw3lCoAPGiL+Ra9Zaov+VVfhTTMv4uxGrjOV4d
DUTveu6mc+75E5mLBjmkGjtsD3H3e/xHTIdiOZd0emgr4PD8yXQqbDKybcvOOhLZ
uNFQPIE4gqzyGMb9BniOkECASLNBKgZcmtibkdwIghQATh+WbYhhOx+DyY+Dd/tG
LE1Q+Wf5sd2Oc/C59W6zDOKfmXmNXwIDAQABAoICAQDLXDWZJsPuyLE3Jfkgf2o0
slrx1WbVboodWu+k1LesacK1TVo0DGEqYYczlfXQmYOMSo+Oqe6Z+uiH+86SEHMY
as8ALMFTKH9TBVMbgIjqwvClj015V3b2EvBF4X1ihy14dmd/dJxIKtqqj+9oMkuh
V1jX9caIcNXQzEzX0lR2ABEv8BaiL4k2fyhY89Tu2YytKR9Ue87fXCC2COBP0lq3
I5Pn6LgJjI5JNOLggPHrcsMsJurtLl7d8pmVVABlnd9D3qA0Na/g9ONNT2I9X+/v
97OOBGv+aRxZE3Ij5MUq8c/6ClXSmMF/36UNZKF+YDrR3zVrxNbwNQWTk5C2W+mb
kAV15nAsF1RF+BX2KDLTeMnk72iiOho9BPXHbbiSJktHJHlNOd/cqic2R0P+1QAP
PMjKTQLYxci1BgofdBYB2lbdB/V+BtIsJ3TwaUXsQvgLwgU67LqameDZ+k8ROtUl
wZqKpsgnQpZ5eJtnXtpc4U2r9F+Kj718JpIYCZKY22rQgycNf8PKBD5jVY87QXhq
7qP071t2jXnIBE8Jb9/EeCa+pKTV6PlpVdX1DpISb69U640dUGPHjqu1xEOQIpiI
/+fyinbicLpwD8GYjnMjhV9/72Ka47fmvyOSPzo8hZUxII1X5iAvGYpp/Jpb6XBU
RKg8xW+fg43hVNiC65iGgQKCAQEA7Gwza7+bDtsJxsho2qT1cqjXL0UVaCq9ak+8
eHgYf3f7TeyG7OQS7BTWtgDcFfDp9Tdyry9A8ma7Cza3Xvw9u1beuJShUZBFLbZ+
vpbNRlcwP8A/BepQz0Qo3AVfjCIDugTHM0qy4aBqXdX+ynFD63C2bXTF6dJo75Jj
xYkyQOLb1rMVjd/G3P5sklG9RF+fR0UHi/vSYg7mWf0Dq3igzDDMi0BlMBE8+BqT
ciYef7Q3NQjYq1MqVwhRQcH9vA7tKgQzWp8pugEBftD8S1cnGR6psIwpyThEhyFv
GJo/n6Fo5QbalGq7ogJSKOlIJZx/izLkJl1VL8qfns1fg7v5CQKCAQEA41XHm8gw
T+Y/I//6stEf0P+MVKD1tiSfTiTX2LByLp1i43arvWhiid7LD7zEqLEY29XhudOO
szPR1haYuqIhjvhzbllV0NWQeeT9YyiSNQ63t/RvtjhWZ8Ffi2yV8s/iyT814bY0
2wKV+EPhPDsBipfhkNxDjDUxNWq1EvcdeN2FOa2HEgR5RpAd1T3IbDpi/wy3N/3W
rGy6NbbJcHygwsjGPBXPhjAZkqFu3GPys/MZZve+edhD7e68y1r90jIsytS9otsm
meBeFenR70+Tf6YWrJppsVXPb+uwlr2nVNDHZ1zcfYogjLs4tApogGFPzZKfy4Jn
X1kkvmiFE7n1JwKCAQBEXI0JzN+DDibnia93+VbXjqaaDnnAIwueH+w5UVCUGxdZ
Utk4ykIGbYggHGOHHKApvZy1tw4qiTXwaiPfnUQkVVwVNzTmJrc6HpjLd0Nn4XIc
HPScO0Kei/Dcndkg5fz53sPSuvi6cO4Qr/36f4HKJE87mxZXI/Yfv86FocQcKvyy
Ohozac9Qu2idbnExwgyGSRmDio8st243+wcCn+Cu6jVa1oXrvjBI9TZJPWh4OJ32
AdbUwzls7QTB5Nv/crl0+r32qCsik4PhLYCmME8n3kvmtsCmZFS8VhiPnppjCAMS
pkaxv6L9l3o2Ri4MYhInJ9H8neQx6374Jh5GMyYxAoIBACZB6E6iGOdJUzTmvjTb
lqQgbWhMki0t6pVHBAAWaZDIsbyf2vUMHREgqkGiveG5s/pC+zK/lJM51EVYFinK
YSVjUGGwrQ1w81hgHfhS+o/tQyO1AhvDTV82nrKi+nUbYQoHFjU+6ZQ10jEukzgE
ohTFzJMJTmDJDtfzdjeT2KTfeq0jM8jncdVbKXoaZKE6DjDn3emRUVBBF/E0KqBA
iPleumWgMgVeEN+pRTPXqh94eLzoUmjE6WGgPKtoS7DU+s7DkIpYoR1iMdM0Pz0r
wiHIPKadcc4DJ96o5lXn4sIWRIhzizOhTCsC0t8RpVZ9ieWJmFSyRF06bkGQ61xP
fh8CggEAEt7swXZznLParkDyWj0hjMNU5YLSPBikuZ+HtX2baVwlx7h7GnfwdmnD
TSzXQBRmQOgS/1Cntx5ol5ce/FUckP7ynqmTm3hDEgq3VT7Vc5KRUUyQLL93ft0T
K+pJ/hjBApOMnytqJttNz9qPs9jtaFkH0hnIPuwO3VIFi2qVhQM3KTGUl1MliXWL
iUmebg7yevOh8nkHR3B6GuCoiVQORYtVQo6p60i6oqXSz7tx/mlMqbV5o1hcd8iE
WESmigg1ZXkl20NEmfDBVZO2O41ODdM+raNVGgtESV4BStc8LO7K3Z4/OcoplV6I
H/Njg8CqtOWDeTVICuUq60wkbEkxYg==
-----END PRIVATE KEY-----`)
err = ioutil.WriteFile(KeyPEMFile, keyPEM, 0600)
require.NoError(t, err)