DEV Community

Cover image for Automated Kubernetes Cluster Cleanup at Scale
Gianluca
Gianluca

Posted on

Automated Kubernetes Cluster Cleanup at Scale

Deploying add-ons to a fleet of Kubernetes cluster

Sveltos is a set of Kubernetes controllers that run in the management cluster. From the management cluster, Sveltos can manage add-ons and applications on a fleet of managed Kubernetes clusters. It is a declarative tool to ensure that the desired state of an application is always reflected in the actual state of the Kubernetes managed clusters.

In a management cluster, each individual Kubernetes cluster is represented by a dedicated resource. Labels can be attached to those resources.

Sveltos configuration utilises a concept called a cluster selector. This selector essentially acts like a filter based on Kubernetes labels. By defining specific labels or combinations of labels, you can create a subset of clusters that share those characteristics.

Deploying Kubernetes add-ons and applications with Sveltos

Automated Kubernetes Resource Cleanup

Cleaner is a Kubernetes controller that automates resource management, keeping your cluster clean and efficient. It proactively identifies and removes (or updates) unused resources, optimizing resource utilization. Powerful features include:

  • Flexible Scheduling: Schedule cleanup tasks at regular intervals.
  • Precise Targeting: Select resources based on labels or use Lua scripting for complex criteria.
  • Dry Run Mode: Test cleanup actions before making permanent changes.
  • Integrated Notifications: Receive alerts about resource cleanup via Slack, Webex, MS Teams, or Discord.

Deploying k8s-cleaner across Kubernetes Clusters with Sveltos

By combining k8s-cleaner with Sveltos, you gain a powerful solution for maintaining clean, efficient, and secure Kubernetes clusters across your entire fleet. This approach simplifies management, reduces operational overhead, and optimizes resource utilization.

Deploying K8s-cleaner with Sveltos

Lab Setup

Management cluster is created using Kind cluster. Then two Civo clusters, all with label env=fv :



+------------------------+-------------+-------------------------------------+
|    Cluster Name        |   Version   |             Comments                |
+------------------------+-------------+-------------------------------------+
|    cluster1            | v1.29.2+k3s1| Civo 3 Node - Medium Standard       |
|    cluster2            | v1.28.7+k3s1| Civo 3 Node - Medium Standard       |
+------------------------+-------------+-------------------------------------+


Enter fullscreen mode Exit fullscreen mode

Step 1: Install Sveltos on Managament Cluster

For this demonstration, we will install Sveltos in the management cluster. Sveltos installation details can be found here.



kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.34.1/manifest/manifest.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.34.1/manifest/default-classifier.yaml


Enter fullscreen mode Exit fullscreen mode

Step 2: Register Civo Cluster with Sveltos

Create two Kubernetes clusters using Civo UI. Download the Kubeconfigs, then:



kubectl create ns civo
sveltosctl register cluster --namespace=civo --cluster=cluster1 --kubeconfig=civo-cluster1-kubeconfig --labels=env=fv
sveltosctl register cluster --namespace=civo --cluster=cluster2 --kubeconfig=civo-cluster2-kubeconfig --labels=env=fv


Enter fullscreen mode Exit fullscreen mode

Verify your Civo were successfully registered:



kubectl get sveltoscluster -n civo
NAME       READY   VERSION
cluster1   true    v1.29.2+k3s1
cluster2   true    v1.28.7+k3s1 


Enter fullscreen mode Exit fullscreen mode

Step 3: Create Deployment Configuration

This step showcases deploying k8s-cleaner with Sveltos across all clusters (labeled env=fv) to optimize resource utilization. We’ll leverage pre-configured ConfigMaps provided by the projectsveltos/demos repository:

  • install-cleaner.yaml: contains K8s-cleaner resources.

  • unused-configmaps.yaml: contains a Cleaner instance configured to identify ConfigMaps that are not currently used by any Pods in the cluster. This helps optimize resource utilization by removing unused configuration data.

  • pod-with-outdated-secret-data.yaml: contains a Cleaner instance configured to detect Pods that are still using old data from mounted Secrets. This scenario might occur if Pods haven’t been restarted after a Secret update. Cleaner can trigger a restart or other actions to ensure Pods use the latest secret data.



kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/k8s-cleaner/install-cleaner.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/k8s-cleaner/unused-configmaps.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/k8s-cleaner/pod-with-outdated-secret-data.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/k8s-cleaner/clusterprofile-deploy-cleaner.yaml


Enter fullscreen mode Exit fullscreen mode

We’ll use a Sveltos ClusterProfile named deploy-cleaner to target these deployments to all clusters with the label env=fv. This ensures Cleaner runs in all relevant clusters.



apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-cleaner
spec:
  clusterSelector:
    matchLabels:
      env: fv
  policyRefs:
  - name: install-cleaner
    namespace: default
    kind: ConfigMap
  - name: pod-with-outdated-secret
    namespace: default
    kind: ConfigMap
  - name: unused-configmaps
    namespace: default
    kind: ConfigMap


Enter fullscreen mode Exit fullscreen mode

Using sveltosctl we can verify all resources have been deployed:



sveltosctl show addons  
+---------------+-----------------------------------------------+----------------+----------------------------------------+---------+--------------------------------+-------------------------------+
|    CLUSTER    |                 RESOURCE TYPE                 |   NAMESPACE    |                  NAME                  | VERSION |              TIME              |           PROFILES            |
+---------------+-----------------------------------------------+----------------+----------------------------------------+---------+--------------------------------+-------------------------------+
| civo/cluster1 | rbac.authorization.k8s.io:ClusterRole         |                | k8s-cleaner-proxy-role                 | N/A     | 2024-07-15 15:40:37 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | rbac.authorization.k8s.io:ClusterRoleBinding  |                | k8s-cleaner-proxy-rolebinding          | N/A     | 2024-07-15 15:40:39 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | :Service                                      | projectsveltos | k8s-cleaner-controller-metrics-service | N/A     | 2024-07-15 15:40:40 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | apps:Deployment                               | projectsveltos | k8s-cleaner-controller                 | N/A     | 2024-07-15 15:40:41 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | apps.projectsveltos.io:Cleaner                |                | unused-configmaps                      | N/A     | 2024-07-15 15:40:43 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | apiextensions.k8s.io:CustomResourceDefinition |                | cleaners.apps.projectsveltos.io        | N/A     | 2024-07-15 15:40:31 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | apiextensions.k8s.io:CustomResourceDefinition |                | reports.apps.projectsveltos.io         | N/A     | 2024-07-15 15:40:32 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | rbac.authorization.k8s.io:ClusterRole         |                | k8s-cleaner-metrics-reader             | N/A     | 2024-07-15 15:40:36 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | rbac.authorization.k8s.io:ClusterRoleBinding  |                | k8s-cleaner-controller-rolebinding     | N/A     | 2024-07-15 15:40:38 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | apps.projectsveltos.io:Cleaner                |                | list-pods-with-outdated-secret-data    | N/A     | 2024-07-15 15:40:42 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | :Namespace                                    |                | projectsveltos                         | N/A     | 2024-07-15 15:40:30 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | :ServiceAccount                               | projectsveltos | k8s-cleaner-controller                 | N/A     | 2024-07-15 15:40:33 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | rbac.authorization.k8s.io:ClusterRole         |                | k8s-cleaner-controller-role            | N/A     | 2024-07-15 15:40:35 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | :Service                                      | projectsveltos | k8s-cleaner-controller-metrics-service | N/A     | 2024-07-15 15:40:27 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | apps:Deployment                               | projectsveltos | k8s-cleaner-controller                 | N/A     | 2024-07-15 15:40:28 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | apiextensions.k8s.io:CustomResourceDefinition |                | cleaners.apps.projectsveltos.io        | N/A     | 2024-07-15 15:40:19 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | apiextensions.k8s.io:CustomResourceDefinition |                | reports.apps.projectsveltos.io         | N/A     | 2024-07-15 15:40:20 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | :ServiceAccount                               | projectsveltos | k8s-cleaner-controller                 | N/A     | 2024-07-15 15:40:22 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | rbac.authorization.k8s.io:ClusterRole         |                | k8s-cleaner-metrics-reader             | N/A     | 2024-07-15 15:40:23 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | rbac.authorization.k8s.io:ClusterRole         |                | k8s-cleaner-proxy-role                 | N/A     | 2024-07-15 15:40:24 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | rbac.authorization.k8s.io:ClusterRoleBinding  |                | k8s-cleaner-controller-rolebinding     | N/A     | 2024-07-15 15:40:25 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | apps.projectsveltos.io:Cleaner                |                | unused-configmaps                      | N/A     | 2024-07-15 15:40:30 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | :Namespace                                    |                | projectsveltos                         | N/A     | 2024-07-15 15:40:19 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | rbac.authorization.k8s.io:ClusterRole         |                | k8s-cleaner-controller-role            | N/A     | 2024-07-15 15:40:23 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | rbac.authorization.k8s.io:ClusterRoleBinding  |                | k8s-cleaner-proxy-rolebinding          | N/A     | 2024-07-15 15:40:26 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | apps.projectsveltos.io:Cleaner                |                | list-pods-with-outdated-secret-data    | N/A     | 2024-07-15 15:40:29 +0200 CEST | ClusterProfile/deploy-cleaner |
+---------------+-----------------------------------------------+----------------+----------------------------------------+---------+--------------------------------+-------------------------------+


Enter fullscreen mode Exit fullscreen mode

Step 4: Create Deployment Configuration

We will use Kustomize with configMapGenerator to deploy an Nginx application.



git clone git@github.com:projectsveltos/demos.git
cd demos/k8s-cleaner/nginx 


Enter fullscreen mode Exit fullscreen mode

And deploy to both our Civo clusters:



kustomize build |KUBECONFIG=<CIVO CLUSTER1 KUBECONFIG> kubectl apply -f - 
kustomize build |KUBECONFIG=<CIVO CLUSTER2 KUBECONFIG> kubectl apply -f -


Enter fullscreen mode Exit fullscreen mode

Check if the deployments are running successfully on both clusters using kubectl get deployments -n k3s-home with the appropriate kubeconfig file for each cluster.

When we update any configuration files (index.html), the configMap created by configMapGenerator will have a new hash, and applying new configuration will ask the deployment to use the new configmap, but the old configmap (with the old hash string) will not removed. You can read more about this here.

Let’s we change the content of index.html file and redeploy kustomize output only to cluster2:



vim index.html
kustomize build |KUBECONFIG=<CIVO CLUSTER2 KUBECONFIG> kubectl apply -f -


Enter fullscreen mode Exit fullscreen mode

Step 5: Visualize the unused ConfigMaps

In this step, we’ll leverage Sveltos, your cluster management platform, to identify unused ConfigMaps across your managed clusters. While Sveltos takes center stage for visualization, it’s important to understand the supporting role played by k8s-cleaner.This tool, deployed in each cluster, actively scans for unused ConfigMaps and generates reports listing these potential cleanup candidates.

Let’s configure Sveltos to automatically collect these reports from each managed cluster.



kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/k8s-cleaner/clusterhealthcheck.yaml


Enter fullscreen mode Exit fullscreen mode

Then using sveltosctl we can see which ConfigMap is unused:



sveltosctl show resources

+---------------+----------------------------------+-----------+-------------------+------------------------------------------+
| CLUSTER | GVK | NAMESPACE | NAME | MESSAGE |
+---------------+----------------------------------+-----------+-------------------+------------------------------------------+
| civo/cluster2 | apps.projectsveltos.io/v1alpha1, | | unused-configmaps | Affected |
| | Kind=Report | | | Resources:k3s-home/nginx-html-kkkff7h9h5 |
| | | | | |
+---------------+----------------------------------+-----------+-------------------+------------------------------------------+

Enter fullscreen mode Exit fullscreen mode




Conclusion: Automated Cluster Cleanup at Scale with Sveltos and k8s-cleaner

This guide demonstrated how to leverage Sveltos and k8s-cleaner to achieve automated Kubernetes cluster cleanup across a fleet of clusters. By combining these tools, you gain a centralized and efficient solution for:

  • Optimizing resource utilization: k8s-cleaner proactively identifies and removes unused resources like ConfigMaps, freeing up valuable cluster resources.

  • Simplified management: Sveltos provides a central dashboard for visualizing unused resources across all clusters, eliminating the need to manage each cluster individually.

👏 Support those projects

Every contribution counts! If you enjoyed this article, check out:

You can star 🌟 the project if you find it helpful.

The GitHub repos are a great resource for getting started with the project. It contains the code, documentation, and many more examples.

Thanks for reading!

Contact Information

If you have questions or would like to have a friendly chat, just feel free to add me to your LinkedIn network!

Top comments (0)