Skip to content

Compliant Kubernetes Deployment on Exoscale

This document contains instructions on how to setup a service cluster and a workload cluster in Exoscale. The following are the main tasks addressed in this document:

  1. Infrastructure setup for two clusters: one service and one workload cluster
  2. Deploying Compliant Kubernetes on top of the two clusters.
  3. Creating DNS Records
  4. Deploying Rook Storage Orchestration Service
  5. Deploying Compliant Kubernetes apps

The instructions below are just samples, you need to update them according to your requirements. Besides, the exoscale cli is used to manage DNS. If you are using any other DNS service provider for managing your DNS you can skip it.

Before starting, make sure you have all necessary tools.

Note

This guide is written for compliantkubernetes-apps v0.15.0

Setup

Choose names for your service cluster and workload cluster, as well as a name for your environment:

SERVICE_CLUSTER="testsc"
WORKLOAD_CLUSTERS=( "testwc0" )
CK8S_ENVIRONMENT_NAME=my-environment-name

Infrastructure Setup using Terraform

Before trying any of the steps, clone the Elastisys Compliant Kubernetes Kubespray repo as follows:

git clone --recursive https://github.com/elastisys/compliantkubernetes-kubespray
cd compliantkubernetes-kubespray/kubespray

Note

The Terraform code for Exoscale is pushed to Kubespray upstream, but is not yet released (see this issue)

Please run git checkout b77460ec to use the non-release Kubespray.

After the infrastructure setup step, switch back to Kubespray v2.15.0.

Expose Exoscale credentials to Terraform

For authentication create the file ~/.cloudstack.ini and put your Exoscale credentials in it. The file should look like something like this:

[cloudstack]
key = <API key>
secret = <API secret>

Customize your infrastructure

Create a configuration for the service and the workload clusters:

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
  cp -r inventory/sample inventory/$CLUSTER
  cp contrib/terraform/exoscale/default.tfvars inventory/$CLUSTER/
done

Review and, if needed, adjust the files in inventory/$CLUSTER/default.tfvars, where $CLUSTER is the cluster name:

  • Use different value for the prefix field in /default.tfvars for the two clusters. Failing to do so will result in a name conflict.
  • Set a non-zero value for ceph_partition_size field, e.g., "ceph_partition_size": 50, as it will be used by Rook storage service to provide local disk storage.
  • To security harden your cluster, set ssh_whitelist and api_server_whitelist to the IP addresses from which you expect to operate the cluster.
  • Make sure you configure your SSH keys in ssh_public_keys.

Important

The Linux Ubuntu 20.04 LTS 64-bit image on Exoscale is regularly upgraded, which might cause unexpected changes during terraform apply. Consider uploading your own dated Ubuntu image to reduce the risk of downtime.

Initialize and Apply Terraform

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    pushd inventory/$CLUSTER
    terraform init ../../contrib/terraform/exoscale
    terraform apply \
        -var-file default.tfvars \
        -state=tfstate-$CLUSTER.tfstate  \
        ../../contrib/terraform/exoscale
    popd
done

Important

The Terraform state is stored in inventory/$CLUSTER/tfstate-$CLUSTER.tfstate, where $CLUSTER is the cluster name. It is precious. Consider backing it up or using Terraform Cloud.

You should now have inventory file named inventory/$CLUSTER/inventory.ini for each cluster that you can use with kubespray.

Test access to all nodes

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    pushd inventory/$CLUSTER
    ANSIBLE_HOST_KEY_CHECKING=False ansible all -i inventory.ini -m ping
    popd
done

Deploying vanilla Kubernetes clusters using Kubespray

With the infrastructure provisioned, we can now deploy Kubernetes using kubespray. First, change to the compliantkubernetes-kubespray root directory.

cd ..

Note

Remember to switch back to Kubespray v2.15.0: git checkout v2.15.0.

Init the Kubespray config in your config path

export DOMAIN=<your_domain> # DNS domain to expose the services inside the service cluster i.e. "example.com"
export CK8S_CONFIG_PATH=~/.ck8s/exoscale
export CK8S_PGP_FP=<your GPG key fingerprint>  # retrieve with gpg --list-secret-keys

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    ./bin/ck8s-kubespray init $CLUSTER default $CK8S_PGP_FP
done

Copy the generated inventory files in the right location

Please copy the two inventory files, kubespray/inventory/$CLUSTER/inventory.ini, generated by Terraform to ${CK8S_CONFIG_PATH}/$CLUSTER-config/, where $CLUSTER the name of each cluster (i.e., testsc, testwc0).

Run kubespray to deploy the Kubernetes clusters

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    ./bin/ck8s-kubespray apply $CLUSTER --flush-cache
done
This may take up to 10 minutes for each cluster, 20 minutes in total.

Correct the Kubernetes API IP addresses

Locate the encrypted kubeconfigs in ${CK8S_CONFIG_PATH}/.state/kube_config_*.yaml and edit them using sops. Copy the public IP address of the load balancer from inventory files ${CK8S_CONFIG_PATH}/*-config/inventory.ini and replace the private IP address for the server field in ${CK8S_CONFIG_PATH}/.state/kube_config_*.yaml.

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml
done

Test access to the clusters as follows

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops exec-file ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml \
        'kubectl --kubeconfig {} get nodes'
done

Create the DNS Records

You will need to setup a number of DNS entries for traffic to be routed correctly. Determine the public IP of the load-balancer fronting the Ingress controller of the clusters from the Terraform state file generated during infrastructure setup.

To get the load-balancer IP, run the following command:

SC_INGRESS_LB_IP_ADDRESS=$(terraform output -state kubespray/inventory/$SERVICE_CLUSTER/tfstate-$SERVICE_CLUSTER.tfstate ingress_controller_lb_ip_address)
echo $SC_INGRESS_LB_IP_ADDRESS

Then point these domains to the service cluster using 'exoscale cli' as follows:

exo dns add A $DOMAIN -a $SC_INGRESS_LB_IP_ADDRESS -n *.ops.$CK8S_ENVIRONMENT_NAME
exo dns add A $DOMAIN -a $SC_INGRESS_LB_IP_ADDRESS -n *.$CK8S_ENVIRONMENT_NAME

Deploy Rook

To deploy Rook, please go to the compliantkubernetes-kubespray repo root directory and run the following.

pushd rook
for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops --decrypt ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml > $CLUSTER.yaml
    export KUBECONFIG=$CLUSTER.yaml
    ./deploy-rook.sh
    shred -zu $CLUSTER.yaml
done
popd

Please restart the operator pod, rook-ceph-operator*, if some pods stalls in initialization state as shown below:

rook-ceph     rook-ceph-crashcollector-minion-0-b75b9fc64-tv2vg    0/1     Init:0/2   0          24m
rook-ceph     rook-ceph-crashcollector-minion-1-5cfb88b66f-mggrh   0/1     Init:0/2   0          36m
rook-ceph     rook-ceph-crashcollector-minion-2-5c74ffffb6-jwk55   0/1     Init:0/2   0          14m

Important

Pods in pending state usually indicate resource shortage. In such cases you need to use bigger instances.

Test Rook

To test Rook, proceed as follows:

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops exec-file ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml 'kubectl --kubeconfig {} apply -f https://raw.githubusercontent.com/rook/rook/release-1.5/cluster/examples/kubernetes/ceph/csi/rbd/pvc.yaml';
done

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops exec-file ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml 'kubectl --kubeconfig {} get pvc';
done

You should see PVCs in Bound state. If you want to clean the previously created PVCs:

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops exec-file ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml 'kubectl --kubeconfig {} delete pvc rbd-pvc';
done

Deploying Compliant Kubernetes Apps

Now that the Kubernetes clusters are up and running, we are ready to install the Compliant Kubernetes apps.

Clone compliantkubernetes-apps and Install Pre-requisites

If you haven't done so already, clone the compliantkubernetes-apps repo and install pre-requisites.

git clone https://github.com/elastisys/compliantkubernetes-apps.git
cd compliantkubernetes-apps
ansible-playbook -e 'ansible_python_interpreter=/usr/bin/python3' --ask-become-pass --connection local --inventory 127.0.0.1, get-requirements.yaml

Initialize the apps configuration

export CK8S_ENVIRONMENT_NAME=my-environment-name
#export CK8S_FLAVOR=[dev|prod] # defaults to dev
export CK8S_CONFIG_PATH=~/.ck8s/my-cluster-path
export CK8S_CLOUD_PROVIDER=# [exoscale|safespring|citycloud|aws|baremetal]
export CK8S_PGP_FP=<your GPG key fingerprint>  # retrieve with gpg --list-secret-keys
./bin/ck8s init

Three files, sc-config.yaml and wc-config.yaml, and secrets.yaml, were generated in the ${CK8S_CONFIG_PATH} directory.

ls -l $CK8S_CONFIG_PATH

Configure the apps

Edit the configuration files ${CK8S_CONFIG_PATH}/sc-config.yaml, ${CK8S_CONFIG_PATH}/wc-config.yaml and ${CK8S_CONFIG_PATH}/secrets.yaml and set the appropriate values for some of the configuration fields. Note that, the latter is encrypted.

vim ${CK8S_CONFIG_PATH}/sc-config.yaml
vim ${CK8S_CONFIG_PATH}/wc-config.yaml
sops ${CK8S_CONFIG_PATH}/secrets.yaml

The following are the minimum change you should perform:

# ${CK8S_CONFIG_PATH}/sc-config.yaml and ${CK8S_CONFIG_PATH}/wc-config.yaml
global:
  baseDomain: "set-me"  # set to $CK8S_ENVIRONMENT_NAME.$DOMAIN
  opsDomain: "set-me"  # set to ops.$CK8S_ENVIRONMENT_NAME.$DOMAIN
  issuer: letsencrypt-prod

objectStorage:
  type: "s3"
  s3:
    region: "set-me"  # Region for S3 buckets, e.g, west-1
    regionEndpoint: "set-me"  # e.g., https://s3.us-west-1.amazonaws.com

storageClasses:
  default:  rook-ceph-block
  nfs:
    enabled: false
  cinder:
    enabled: false
  local:
    enabled: false
  ebs:
    enabled: false
# ${CK8S_CONFIG_PATH}/sc-config.yaml (in addition to the changes above)
ingressNginx:
    controller:
      service:
        type: "this-is-not-used"
        annotations: "this-is-not-used"

harbor:
  oidc:
    groupClaimName: "set-me" # set to group claim name used by OIDC provider

issuers:
  letsencrypt:
    prod:
      email: "set-me"  # set this to an email to receive LetsEncrypt notifications
    staging:
      email: "set-me"  # set this to an email to receive LetsEncrypt notifications
# ${CK8S_CONFIG_PATH}/secrets.yaml
objectStorage:
  s3:
    accessKey: "set-me" # set to your s3 accesskey
    secretKey: "set-me" # set to your s3 secretKey

Create S3 buckets

You can use the following script to create required S3 buckets. The script uses s3cmd in the background and gets configuration and credentials for your S3 provider from ${HOME}/.s3cfg file.

# Use your default s3cmd config file: ${HOME}/.s3cfg
scripts/S3/entry.sh create

Important

You should not use your own credentials for S3. Rather create a new set of credentials with write-only access, when supported by the object storage provider (check a feature matrix).

Test S3

To ensure that you have configured S3 correctly, run the following snippet:

(
    access_key=$(sops exec-file ${CK8S_CONFIG_PATH}/secrets.yaml 'yq r {} "objectStorage.s3.accessKey"')
    secret_key=$(sops exec-file ${CK8S_CONFIG_PATH}/secrets.yaml 'yq r {} "objectStorage.s3.secretKey"')
    region=$(yq r ${CK8S_CONFIG_PATH}/sc-config.yaml 'objectStorage.s3.region')
    host=$(yq r ${CK8S_CONFIG_PATH}/sc-config.yaml 'objectStorage.s3.regionEndpoint')

    for bucket in $(yq r ${CK8S_CONFIG_PATH}/sc-config.yaml 'objectStorage.buckets.*'); do
        s3cmd --access_key=${access_key} --secret_key=${secret_key} \
            --region=${region} --host=${host} \
            ls s3://${bucket} > /dev/null
        [ ${?} = 0 ] && echo "Bucket ${bucket} exists!"
    done
)

Install Compliant Kubernetes apps

Start with the service cluster:

ln -sf $CK8S_CONFIG_PATH/.state/kube_config_${SERVICE_CLUSTER}.yaml $CK8S_CONFIG_PATH/.state/kube_config_sc.yaml
./bin/ck8s apply sc  # Respond "n" if you get a WARN

Then the workload clusters:

for CLUSTER in "${WORKLOAD_CLUSTERS[@]}"; do
    ln -sf $CK8S_CONFIG_PATH/.state/kube_config_${CLUSTER}.yaml $CK8S_CONFIG_PATH/.state/kube_config_wc.yaml
    ./bin/ck8s apply wc  # Respond "n" if you get a WARN
done

Settling

Important

Leave sufficient time for the system to settle, e.g., request TLS certificates from LetsEncrypt, perhaps as much as 20 minutes.

You can check if the system settled as follows:

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops exec-file ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml \
        'kubectl --kubeconfig {} get --all-namespaces pods'
done

Check the output of the command above. All Pods needs to be Running or Completed.

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops exec-file ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml \
        'kubectl --kubeconfig {} get --all-namespaces issuers,clusterissuers,certificates'
done

Check the output of the command above. All resources need to have the Ready column True.

Testing

After completing the installation step you can test if the apps are properly installed and ready using the commands below.

Start with the service cluster:

ln -sf $CK8S_CONFIG_PATH/.state/kube_config_${SERVICE_CLUSTER}.yaml $CK8S_CONFIG_PATH/.state/kube_config_sc.yaml
./bin/ck8s test sc  # Respond "n" if you get a WARN

Then the workload clusters:

for CLUSTER in "${WORKLOAD_CLUSTERS[@]}"; do
    ln -sf $CK8S_CONFIG_PATH/.state/kube_config_${CLUSTER}.yaml $CK8S_CONFIG_PATH/.state/kube_config_wc.yaml
    ./bin/ck8s test wc  # Respond "n" if you get a WARN
done

Done. Navigate to the endpoints, for example grafana.$BASE_DOMAIN, kibana.$BASE_DOMAIN, harbor.$BASE_DOMAIN, etc. to discover Compliant Kubernetes's features.

Teardown

Removing Compliant Kubernetes Apps from your cluster

To remove the applications added by compliant kubernetes you can use the two scripts clean-sc.sh and clean-wc.sh, they are located here in the scripts folder.

They perform the following actions:

  1. Delete the added helm charts
  2. Delete the added namespaces
  3. Delete any remaining PersistentVolumes
  4. Delete the added CustomResourceDefinitions

Note: if user namespaces are managed by Compliant Kubernetes apps then they will also be deleted if you clean up the workload cluster.

Remove infrastructure

To teardown the infrastructure, please switch to the root directory of the exoscale branch of the Kubespray repo (see the Terraform section).

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    pushd inventory/$CLUSTER
    terraform init ../../contrib/terraform/exoscale
    terraform destroy \
        -var-file default.tfvars \
        -state=tfstate-$CLUSTER.tfstate  \
        ../../contrib/terraform/exoscale
    popd
done

# Remove DNS records
exo dns remove $DOMAIN *.ops.$CK8S_ENVIRONMENT_NAME
exo dns remove $DOMAIN *.$CK8S_ENVIRONMENT_NAME

Further Reading