Kubernetes DaemonSets: Managing System-Level Components Across Every Node

Kubernetes DaemonSets for System Components

In Kubernetes, DaemonSets are a specialized type of controller used to manage the deployment of pods on all or specific nodes within a cluster. They are particularly useful for running system-level services or agents that need to run on every node in a cluster or on a subset of nodes.

A DaemonSet ensures that a copy of a specific pod runs on all nodes (or on a subset of nodes, depending on the configuration). It is typically used for system components or background services that must run on every node, such as logging agents, monitoring agents, network proxies, or any service that needs to interact with the node itself.

In this article, we’ll explore the concept of DaemonSets in Kubernetes, their use cases, and how to implement and manage them effectively.

---

What is a DaemonSet?

A DaemonSet in Kubernetes is a controller that ensures a specific pod is running on every node in a cluster or on a subset of nodes. Unlike other controllers like Deployments or StatefulSets, which are generally used for managing stateless or stateful applications, DaemonSets are typically used for workloads that need to be deployed on every node in a cluster.

Key features of DaemonSets include:

• One pod per node: A DaemonSet ensures that a pod is running on each node in the cluster.
• Dynamic scaling: If new nodes are added to the cluster, the DaemonSet automatically creates a pod on those new nodes.
• Node selection: DaemonSets can be configured to run only on specific nodes using node selectors, affinity, or taints and tolerations.

DaemonSets are particularly important for deploying system components like logging, monitoring, or networking services that must be available on all nodes in a cluster.

---

Why Use DaemonSets?

DaemonSets are primarily used for workloads that need to run on every node in a Kubernetes cluster. Common use cases include:

1. Cluster-level logging: DaemonSets can be used to deploy logging agents (e.g., fluentd or logstash) on every node in the cluster, allowing the collection and forwarding of logs to a centralized logging system.

2. Cluster-level monitoring: Monitoring agents like Prometheus node exporters or datadog agents can be deployed using DaemonSets to collect node-level metrics across the entire cluster.

3. Networking components: DaemonSets can be used to deploy network proxies or VPN agents (e.g., Calico or Cilium) to every node for controlling network traffic and security.

4. System daemons: System-level daemons, such as firewall agents, load balancers, or network routers, often need to be deployed on each node in a cluster.

5. Service Mesh: In service mesh scenarios, DaemonSets can be used to deploy proxies (e.g., Envoy or Istio sidecars) on each node to manage communication between microservices.

---

How DaemonSets Work

When you create a DaemonSet in Kubernetes, the controller ensures that a pod is scheduled on each node according to the defined criteria (like node affinity, selectors, or tolerations). The key workflow includes:

1. Pod Scheduling: Kubernetes schedules one pod on each node that matches the DaemonSet’s node selectors and constraints.
2. Automatic Scaling: If new nodes are added to the cluster, the DaemonSet automatically creates a pod on those nodes. Conversely, if nodes are removed, the DaemonSet ensures the pods on those nodes are deleted.
3. Pod Deletion: When you delete a DaemonSet, Kubernetes will terminate the pods running on all nodes that are associated with the DaemonSet.

DaemonSets are highly dynamic and ensure that as the cluster scales, the relevant services are deployed on the newly added nodes without requiring any manual intervention.

---

How to Define a DaemonSet in Kubernetes

A DaemonSet is defined using a YAML file that specifies the desired state of the pods, including the containers to run, resource requests, and any node-specific constraints.

Example YAML for a DaemonSet (e.g., for running a logging agent):

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  labels:
    app: logging
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
        - name: fluentd
          image: fluent/fluentd:v1.12-1
          volumeMounts:
            - mountPath: /fluentd/etc
              name: config-volume
            - mountPath: /var/log
              name: log-volume
      volumes:
        - name: config-volume
          configMap:
            name: fluentd-config
        - name: log-volume
          hostPath:
            path: /var/log
            type: DirectoryOrCreate

In this example:

• A fluentd container is deployed as a DaemonSet on each node.
• The volumes section is used to mount host directories (in this case, /var/log) to collect logs.
• The ConfigMap fluentd-config provides configuration data to the fluentd container.

---

DaemonSet Node Selection

DaemonSets provide several ways to control where pods are scheduled to run using node selectors, affinity, taints, and tolerations. These mechanisms ensure that pods only run on the appropriate nodes, based on specific conditions.

1. Node Selectors: You can use node selectors to specify that DaemonSet pods should only run on nodes with specific labels.
   

   spec:
     template:
       spec:
         nodeSelector:
           disktype: ssd
   

   This ensures that DaemonSet pods only run on nodes labeled with disktype=ssd.

2. Node Affinity: Node affinity allows for more complex scheduling rules based on node labels.
   

   spec:
     template:
       spec:
         affinity:
           nodeAffinity:
             requiredDuringSchedulingIgnoredDuringExecution:
               nodeSelectorTerms:
                 - matchExpressions:
                     - key: disktype
                       operator: In
                       values:
                         - ssd
   

3. Taints and Tolerations: Taints and tolerations are useful for ensuring that DaemonSet pods run only on nodes that are "tainted" with certain conditions (e.g., only run on dedicated nodes).
   

   spec:
     template:
       spec:
         tolerations:
           - key: "dedicated"
             operator: "Equal"
             value: "logging"
             effect: "NoSchedule"
   

   This ensures that the DaemonSet pods only run on nodes with a dedicated=logging taint.

---

DaemonSet Rolling Updates

DaemonSets support rolling updates, which allow you to update the pods managed by the DaemonSet one at a time without disrupting the entire system. This is particularly useful when updating system components or services.

Rolling updates can be controlled by the following parameters:

• maxUnavailable: Limits the number of pods that can be unavailable during an update.
• maxSurge: Allows new pods to be created during the update process before old pods are terminated.

Example rolling update configuration:

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1

In this case, at most 1 pod can be unavailable during the update, and 1 extra pod can be created before terminating the old pods.

---

Common Use Cases for DaemonSets

• Log Collection: Running log collection agents (e.g., Fluentd, Logstash) on each node to collect and forward logs to a central location.
• Monitoring Agents: Deploying monitoring agents like Prometheus node exporters or Datadog agents on every node to gather metrics.
• Network Proxies: Running network proxies, such as Envoy or Istio sidecar proxies, on each node to manage communication between services.
• Security Agents: Deploying security agents like anti-virus software, intrusion detection systems, or firewall agents on every node.
• System Daemons: Running background services such as backup agents, configuration management (e.g., Ansible), or service mesh components on all nodes.

---

Best Practices for Using DaemonSets

1. Use for System-Level Daemons: DaemonSets are best suited for system-level components that need to run on every node in the cluster (e.g., logging, monitoring, network proxies).

2. Resource Requests and Limits: Ensure that each pod in the DaemonSet has defined resource requests and limits to prevent resource contention across nodes.

3. Consider Node Affinity: If your cluster has different types of nodes (e.g., GPU nodes, dedicated storage nodes), use node selectors or affinity rules to ensure that DaemonSet pods only run on the appropriate nodes.

4. Rolling Updates: Use rolling updates for DaemonSets to minimize disruptions when updating system components.

5. Taints and Tolerations: Use taints and tolerations to ensure that DaemonSets only run on nodes that are specifically intended for these workloads.

---

Conclusion

DaemonSets in Kubernetes are essential for managing system-level services that need to run on every node in a cluster or on a specific subset of nodes. They ensure that each node gets a copy of a pod, providing benefits like scaling, dynamic node addition, and automatic management of system daemons such as logging, monitoring, and networking agents.

By using DaemonSets effectively, you can ensure your Kubernetes cluster is equipped with critical components for operational monitoring, security, and logging, while also ensuring that your infrastructure remains scalable and highly available.

---