In today’s dynamic cloud environments, managing Kubernetes resources across multiple clusters can be a complex task. Traditional methods often lack the agility and event-driven architecture needed to respond quickly to changes and automate resource provisioning. This article explores how Sveltos, in conjunction with NATS and JetStream, simplifies multi-cluster Kubernetes management through event-driven automation, streamlining operations and improving responsiveness.
What is Sveltos
Sveltos is a set of Kubernetes controllers operating within a management cluster. From this central point, Sveltos manages add-ons and applications across a fleet of managed Kubernetes clusters. It employs a declarative approach, ensuring that the desired state is consistently reflected across these managed environments.
Within the management cluster, each managed Kubernetes cluster is represented by a dedicated resource, to which labels can be attached. Sveltos leverages cluster selectors, essentially label-based filters, to target specific groups of managed clusters for configuration.
What is NATS
NATS is a lightweight, high-performance messaging system optimized for speed and scalability. It excels at publish/subscribe communication. JetStream enhances NATS with robust streaming and data management features, including message persistence, flow control, and ordered delivery, creating a powerful platform for modern distributed systems.
Sveltos Event Framework: Expanding Beyond Kubernetes with NATS
Sveltos offers an event-driven workflow: define events, select target clusters, and specify the add-ons/applications to deploy when those events occur. (see the documentation and video by Colin for more details.)
Initially, Sveltos’s event framework focused on monitoring Kubernetes resources within the managed clusters, enabling automated responses to changes in those resources. However, with its NATS integration, Sveltos now supports a significantly expanded event horizon. It can now also react to events originating outside the Kubernetes environment. This allows Sveltos to respond to a much wider range of triggers, including events from external systems, applications, and even manual interactions, dramatically increasing its automation and integration capabilities.
Sveltos can connect to a NATS server and listen for CloudEvents published on NATS subjects. When a CloudEvent is received, Sveltos can automatically deploy or manage other Kubernetes resources in response to. Consider user authentication: a successful login (published as a CloudEvent) can trigger Sveltos to automatically create a user-specific namespace. Similarly, a logout event can initiate automated cleanup within the cluster.
Lab Setup
A Kind cluster is used as management cluster. Then two extra Civo clusters all with label env=production.
+------------------------+-------------+-------------------------------------+
| Cluster Name | Version | Comments |
+------------------------+-------------+-------------------------------------+
| civo/cluster1 | v1.29.8+k3s1| Civo 3 Node - Medium Standard |
| civo/cluster2 | v1.30.5+k3s1| Civo 3 Node - Medium Standard |
+------------------------+-------------+-------------------------------------+
Step 1: Install Sveltos on Managament Cluster
For this tutorial, we will install Sveltos in the management cluster. Sveltos installation details can be found here.
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.46.1/manifest/manifest.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.46.1/manifest/default-classifier.yaml
Step 2: Register Clusters with Sveltos
Using Civo UI, download the Kubeconfigs, then:
kubectl create ns civo
sveltosctl register cluster --namespace=civo --cluster=cluster1 --kubeconfig=civo-cluster1-kubeconfig --labels=env=production
sveltosctl register cluster --namespace=civo --cluster=cluster2 --kubeconfig=civo-cluster2-kubeconfig --labels=env=production
Verify your Civo clusters were successfully registered:
kubectl get sveltoscluster -A --show-labels
NAMESPACE NAME READY VERSION LABELS
civo cluster1 true v1.29.8+k3s1 env=production,projectsveltos.io/k8s-version=v1.29.8,sveltos-agent=present
civo cluster2 true v1.30.5+k3s1 env=production,projectsveltos.io/k8s-version=v1.30.5,sveltos-agent=present
mgmt mgmt true v1.31.2 projectsveltos.io/k8s-version=v1.31.2,sveltos-agent=present
Step 3: Deploy NATS in every production cluster
Use a Sveltos ClusterProfile to deploy NATS (version 1.2.9) to all production clusters.
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
name: nats
spec:
clusterSelector:
matchLabels:
env: production
helmCharts:
- chartName: nats/nats
chartVersion: 1.2.9
helmChartAction: Install
releaseName: nats
releaseNamespace: nats
repositoryName: nats
repositoryURL: https://nats-io.github.io/k8s/helm/charts/
values: |-
config:
merge:
authorization:
default_permissions:
publish: [">"]
subscribe: [">"]
users:
- user: "admin"
password: "my-password"
syncMode: Continuous
Step 4: Instruct Sveltos to connect to NATS server
A separate ClusterProfile then configures Sveltos to establish a connection to the deployed NATS server.
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
name: deploy-deploy-sveltos-nats-secret
spec:
dependsOn:
- nats
clusterSelector:
matchLabels:
env: production
policyRefs:
- name: deploy-sveltos-nats-secret
namespace: default
kind: Secret
---
apiVersion: v1
data:
sveltos-nats: YXBpVmVyc2lvbjogdjEKZGF0YToKICBzdmVsdG9zLW5hdHM6IGV3b2dJQ0p1WVhSeklqb0tJQ0FnZXdvZ0lDQWdJQ0pqYjI1bWFXZDFjbUYwYVc5dUlqb0tDWHNLQ1NBZ0lDQWlkWEpzSWpvZ0ltNWhkSE02THk5dVlYUnpMbTVoZEhNdWMzWmpMbU5zZFhOMFpYSXViRzlqWVd3Nk5ESXlNaUlzQ2drZ0lDQWdJbk4xWW1wbFkzUnpJam9nV3dvSkNTSjFjMlZ5TFc5d1pYSmhkR2x2YmlJS0NTQWdJQ0JkTEFvSklDQWdJQ0poZFhSb2IzSnBlbUYwYVc5dUlqb2dld29KQ1NKMWMyVnlJam9nZXdvSkNTQWdJQ0FpZFhObGNpSTZJQ0poWkcxcGJpSXNDZ2tKSUNBZ0lDSndZWE56ZDI5eVpDSTZJQ0p0ZVMxd1lYTnpkMjl5WkNJS0NRbDlDZ2tnSUNBZ2ZRb0pmUW9nSUNCOUNuMEsKa2luZDogU2VjcmV0Cm1ldGFkYXRhOgogIG5hbWU6IHN2ZWx0b3MtbmF0cwogIG5hbWVzcGFjZTogcHJvamVjdHN2ZWx0b3MKdHlwZTogT3BhcXVlCg==
kind: Secret
metadata:
name: deploy-sveltos-nats-secret
namespace: default
type: addons.projectsveltos.io/cluster-profile
Step 5: Automate namespace creation on user login
Sveltos is configured to listen on the user-operation NATS subject. When a user login CloudEvent is received, Sveltos automatically creates a Kubernetes namespace in the originating cluster. A user logout CloudEvent triggers the deletion of that namespace.
apiVersion: lib.projectsveltos.io/v1beta1
kind: EventSource
metadata:
name: user-operation
spec:
messagingMatchCriteria:
- subject: "user-operation"
cloudEventSource: "auth.example.com"
---
apiVersion: lib.projectsveltos.io/v1beta1
kind: EventTrigger
metadata:
name: manage-namespace
spec:
sourceClusterSelector:
matchLabels:
env: production
eventSourceName: user-operation
oneForEvent: true
syncMode: ContinuousWithDriftDetection
cloudEventAction: '{{if eq .CloudEvent.type "auth.example.com.logout"}}Delete{{else}}Create{{end}}'
policyRefs:
- name: namespace
namespace: default
kind: ConfigMap
---
apiVersion: v1
kind: ConfigMap
metadata:
name: namespace
namespace: default
annotations:
projectsveltos.io/instantiate: ok
data:
namespace.yaml: |
kind: Namespace
apiVersion: v1
metadata:
name: {{ .CloudEvent.subject }}
Step 6: Submit a CloudEvent for user login
Trigger the user login event by publishing the following CloudEvent to the user-operation NATS subject:
CLOUDEVENT_JSON=$(cat << EOF
{
"specversion": "1.0",
"type": "auth.example.com.login",
"source": "auth.example.com",
"id": "10001",
"subject": "mgianluc",
"datacontenttype": "application/json",
"data": {
"message": "User mgianluc login"
}
}
EOF
)
KUBECONFIG=<production cluster kubeconfig> kubectl exec -it deployment/nats-box -n nats -- nats pub user-operation $CLOUDEVENT_JSON --user=admin --password=my-password
Confirm the namespace mgianluc has been created using:
sveltosctl show addons
+-----------------------------+---------------+----------------+--------------+---------+-------------------------------+--------------------------------------------------+
| CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | VERSION | TIME | PROFILES |
+-----------------------------+---------------+----------------+--------------+---------+-------------------------------+--------------------------------------------------+
| default/clusterapi-workload | helm chart | nats | nats | 1.2.9 | 2025-02-04 14:06:14 +0100 CET | ClusterProfile/nats |
| default/clusterapi-workload | :Secret | projectsveltos | sveltos-nats | N/A | 2025-02-04 14:06:36 +0100 CET | ClusterProfile/deploy-deploy-sveltos-nats-secret |
| default/clusterapi-workload | :Namespace | | mgianluc | N/A | 2025-02-04 14:12:03 +0100 CET | ClusterProfile/sveltos-gbv99bcdsk1aa04jkdzv |
+-----------------------------+---------------+----------------+--------------+---------+-------------------------------+--------------------------------------------------+
Step 7: Submit a CloudEvent representing user logout
Publish the user logout CloudEvent:
CLOUDEVENT_JSON=$(cat << EOF
{
"specversion": "1.0",
"type": "auth.example.com.logout",
"source": "auth.example.com",
"id": "10001",
"subject": "mgianluc",
"datacontenttype": "application/json",
"data": {
"message": "User mgianluc logout"
}
}
EOF
)
KUBECONFIG=<production cluster kubeconfig> kubectl exec -it deployment/nats-box -n nats -- nats pub user-operation $CLOUDEVENT_JSON --user=admin --password=my-password
Verify namespace deletion:
sveltosctl show addons
+-----------------------------+---------------+----------------+--------------+---------+-------------------------------+--------------------------------------------------+
| CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | VERSION | TIME | PROFILES |
+-----------------------------+---------------+----------------+--------------+---------+-------------------------------+--------------------------------------------------+
| default/clusterapi-workload | helm chart | nats | nats | 1.2.9 | 2025-02-04 14:06:14 +0100 CET | ClusterProfile/nats |
| default/clusterapi-workload | :Secret | projectsveltos | sveltos-nats | N/A | 2025-02-04 14:06:36 +0100 CET | ClusterProfile/deploy-deploy-sveltos-nats-secret |
+-----------------------------+---------------+----------------+--------------+---------+-------------------------------+--------------------------------------------------+
Conclusion
This article demonstrates how Sveltos can use NATS to react to external events and automate Kubernetes resource provisioning (in this case, namespaces) across a fleet of managed clusters. This provides a powerful, event-driven approach to multi-cluster Kubernetes management.
Contact Information
If you have some questions, would like to have a friendly chat or just network to not miss any topics, then don’t use the comment function at medium, just feel free to add me to your LinkedIn network!
Support this project
If you enjoyed this article, please check out the Projectsveltos GitHub repo. You can also star 🌟 the project if you found it helpful.
The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and examples. You can also find the latest news and updates on the project on the GitHub repo.
Thank you for reading!
Top comments (0)