How to Fix Kubernetes Node Disk Pressure

Imagine you deploy an application, but after a few days, it starts throwing warnings like this:

Warning  NodePressure  [timestamp]  kubelet  Node [node-name] status is now: NodeHasDiskPressure

Your application slows, pods get evicted, and new ones fail to schedule. This common error in Kubernetes is known as Node Disk Pressure, and if left unchecked, it can severely impact application performance.

What is Kubernetes Node Disk Pressure?
Node Disk Pressure occurs when a node’s filesystem is under strain due to low available disk space or inodes.

Kubernetes automatically detects these low resource conditions and sets a NodeHasDiskPressure status.

This status signals that the node has insufficient disk resources for further scheduling, evicting non-critical pods to prevent critical system disruptions.

How to Check Kubernetes Node Disk Pressure
How to Check Kubernetes Node Disk Pressure:

kubectl describe node [node-name]

Look for any nodes with the condition type DiskPressure and status True. In the output, focus on the Conditions section.
Here’s an example where ops-node1 is experiencing disk pressure:

ops-node1 Ready worker 14d v1.28.1 DiskPressure=True,MemoryPressure=False,PIDPressure=False,Ready=True

Additionally, use this command to monitor disk usage:

kubectl top nodes

This shows overall CPU, memory, and disk usage for each node, helping you pinpoint where Disk Pressure is affecting your nodes.

Why Should You Care About Node Disk Pressure?
Ignoring Disk Pressure can lead to various issues:

• Pod Eviction: Kubernetes evicts lower-priority pods to free up disk space, which can cause disruptions in non-critical workloads.

• Scheduling Failures: New workloads may not deploy if nodes are in a Disk Pressure state.

• Performance Degradation: Insufficient disk space impacts node performance and can lead to application latency.

How to Fix Kubernetes Node Disk Pressure
Here are some strategies to address Disk Pressure:

Clean Up Disk Space: Clear out unused images and containers, which can take up significant space.

Increase Node Disk Size: If your nodes are in a cloud environment, consider resizing disks. In AWS, for instance - Increase the EBS volume size.

Move Logs and Data to Separate Disks: If your node frequently generates large logs, consider mounting a separate disk for log storage to keep system space free.

Implement Resource Quotas and Limits:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: storage-quota
  namespace: [namespace]
spec:
  hard:
    requests.storage: 20Gi

Monitor with Alerts:
Use Prometheus or another monitoring tool to set up alerts when disk usage exceeds a threshold. This proactive approach helps you intervene before Disk Pressure arises.

Hope this helps in tackling one of the common Kubernetes challenges.