Simplifying Secret Distribution Across Kubernetes Clusters

A secret, in the context of software, is any piece of sensitive information that you want to keep confidential. This could include API keys, passwords, certificates, or SSH keys.

Now, imagine managing a fleet of Kubernetes clusters, each requiring access to the same secret. The traditional approach often involves manually creating and distributing the secret to each cluster, a time-consuming and error-prone process.

To streamline this process and enhance security, you need a solution that allows you to:

• Centralize Secret Storage: Store the secret in a single, secure location.

• Automate Secret Distribution: Automatically deploy the secret to all target clusters.

In the following sections, we’ll explore how Sveltos can help you achieve these goals.

What is Sveltos

Sveltos is a set of Kubernetes controllers that run in the management cluster. From the management cluster, Sveltos can manage add-ons and applications on a fleet of managed Kubernetes clusters. It is a declarative tool to ensure that the desired state of an application is always reflected in the actual state of the Kubernetes managed clusters.

In a management cluster, each individual Kubernetes cluster is represented by a dedicated resource. Labels can be attached to those resources.

Sveltos configuration utilises a concept called a cluster selector. This selector essentially acts like a filter based on Kubernetes labels. By defining specific labels or combinations of labels, you can create a subset of clusters that share those characteristics.

Lab Setup

A Kind cluster is used as management cluster. Then two extra Civo clusters and a GKE cluster all with label env=prod.

+------------------------+-------------+-------------------------------------+
|    Cluster Name        |   Version   |             Comments                |
+------------------------+-------------+-------------------------------------+
|    civo/cluster1       | v1.29.8+k3s1| Civo 3 Node - Medium Standard       |
|    civo/cluster2       | v1.30.5+k3s1| Civo 3 Node - Medium Standard       |
+------------------------+-------------+-------------------------------------+

Step 1: Install Sveltos on Managament Cluster

For this tutorial, we will install Sveltos in the management cluster. Sveltos installation details can be found here.

kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.42.0/manifest/manifest.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.42.0/manifest/default-classifier.yaml

Step 2: Register Clusters with Sveltos

Using Civo UI, download the Kubeconfigs, then:

kubectl create ns civo
sveltosctl register cluster --namespace=civo --cluster=cluster1 --kubeconfig=civo-cluster1-kubeconfig --labels=env=production
sveltosctl register cluster --namespace=civo --cluster=cluster2 --kubeconfig=civo-cluster2-kubeconfig --labels=env=production

Verify your Civo clusters were successfully registered:

kubectl get sveltoscluster -A --show-labels
NAMESPACE   NAME       READY   VERSION        LABELS
civo        cluster1   true    v1.29.8+k3s1   env=production,projectsveltos.io/k8s-version=v1.29.8,sveltos-agent=present
civo        cluster2   true    v1.30.5+k3s1   env=production,projectsveltos.io/k8s-version=v1.30.5,sveltos-agent=present
mgmt        mgmt       true    v1.31.2        projectsveltos.io/k8s-version=v1.31.2,sveltos-agent=present

Step 3: Deploy Sveltos configuration

First create a Secret of type kubernetes.io/dockerconfigjson to authenticate with a container registry to pull a private image. This will create a Secret regcred in the default namespace on the management cluster.

kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/refs/heads/main/propagate-secret/secret.yaml

We can now configure Sveltos to automatically copy the regcred Secret to namespaces that require it. This configuration will instruct Sveltos to watch for any new namespaces in your production clusters that have the label imagepullsecret: required.

When Sveltos detects a new namespace with this label, it will automatically copy the regcred Secret from the management cluster to the newly created namespace.

kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/refs/heads/main/propagate-secret/config.yaml

Step 4: Create namespaces within Civo clusters

Create the coke namespace in Civo cluster 1:

KUBECONFIG=<CIVO cluster1 kubeconfig> kubectl create namespace coke
KUBECONFIG=<CIVO cluster1 kubeconfig> kubectl label namespace coke imagepullsecret=required

To confirm that the regcred secret has been successfully propagated to the coke namespace:

sveltosctl show addons  
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+
|    CLUSTER    | RESOURCE TYPE | NAMESPACE |  NAME   | VERSION |             TIME              |                  PROFILES                   |
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+
| civo/cluster1 | :Secret       | coke      | regcred | N/A     | 2024-11-20 14:43:05 +0100 CET | ClusterProfile/sveltos-t4q7bhsrkhul3fkh6v0k |
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+

You can follow the same steps to create additional namespaces in Civo cluster 1 (e.g., pepsi) and Civo cluster 2 (e.g., bar), ensuring they inherit the necessary image pull secret.

KUBECONFIG=<CIVO cluster1 kubeconfig> kubectl create namespace pepsi
KUBECONFIG=<CIVO cluster1 kubeconfig> kubectl label namespace pepsiimagepullsecret=required

KUBECONFIG=<CIVO cluster2 kubeconfig> kubectl create namespace bar
KUBECONFIG=<CIVO cluster2 kubeconfig> kubectl label namespace bar imagepullsecret=required

Verify Sveltos has propagated the Secret to all namespaces:

sveltosctl show addons                                                                           
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+
|    CLUSTER    | RESOURCE TYPE | NAMESPACE |  NAME   | VERSION |             TIME              |                  PROFILES                   |
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+
| civo/cluster1 | :Secret       | coke      | regcred | N/A     | 2024-11-20 14:44:41 +0100 CET | ClusterProfile/sveltos-t4q7bhsrkhul3fkh6v0k |
| civo/cluster1 | :Secret       | pepsi     | regcred | N/A     | 2024-11-20 14:44:42 +0100 CET | ClusterProfile/sveltos-t4q7bhsrkhul3fkh6v0k |
| civo/cluster2 | :Secret       | bar       | regcred | N/A     | 2024-11-20 14:45:03 +0100 CET | ClusterProfile/sveltos-3ef2he7v7fklm1s03coa |
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+

Step 5: Update Secret

While pointing to the management cluster, update the regcred Secret:

kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/refs/heads/main/propagate-secret/update-secret.yaml

Verify Secret has been updated to all production clusters:

sveltosctl show addons                                                                                   
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+
|    CLUSTER    | RESOURCE TYPE | NAMESPACE |  NAME   | VERSION |             TIME              |                  PROFILES                   |
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+
| civo/cluster1 | :Secret       | coke      | regcred | N/A     | 2024-11-20 14:54:14 +0100 CET | ClusterProfile/sveltos-t4q7bhsrkhul3fkh6v0k |
| civo/cluster1 | :Secret       | pepsi     | regcred | N/A     | 2024-11-20 14:54:15 +0100 CET | ClusterProfile/sveltos-t4q7bhsrkhul3fkh6v0k |
| civo/cluster2 | :Secret       | bar       | regcred | N/A     | 2024-11-20 14:54:15 +0100 CET | ClusterProfile/sveltos-3ef2he7v7fklm1s03coa |
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+

Step 5: External Secrets Operator

External Secrets Operator is an open source Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, and many more. The goal of External Secrets Operator is to synchronize secrets from external APIs into Kubernetes. The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret. If the secret from the external API changes, the controller will reconcile the state in the cluster and update the secrets accordingly.

When managing a multitude of Kubernetes clusters, External Secrets Operator can be deployed in the management cluster. Sveltos can be used to distribute the secrets to the managed clusters.

Conclusion

This guide demonstrated how Sveltos simplifies the process of propagating secrets to all your production clusters.

Contact Information

If you have some questions, would like to have a friendly chat or just network to not miss any topics, then don’t use the comment function at medium, just feel free to add me to your LinkedIn network!

Support this project

If you enjoyed this article, please check out the Projectsveltos GitHub repo. You can also star 🌟 the project if you found it helpful.
The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and examples. You can also find the latest news and updates on the project on the GitHub repo.
Thank you for reading!