Why don't EC2 CloudWatch Metrics Show Memory and Disk Utilization metrics by default?

#aws #devops #cloudwatch #monitoring

To understand that, you need to understand the difference between hypervisor and OS-level metrics.

🖥️ Virtualization and the Hypervisor:

To understand the distinction, you first need to grasp the basics of virtualization. Virtualization allows multiple virtual machines (VMs), including EC2 instances in the case of AWS, to run on a single physical machine. The component responsible for this is called the hypervisor.

The hypervisor sits between the hardware and the operating systems of the VMs. It allocates resources like CPU, memory, and I/O to the VMs, ensuring they don't interfere with each other.

📊 Hypervisor-Level Metrics vs. OS-Level Metrics:

✅ Hypervisor-Level Metrics: These are metrics that the hypervisor can observe directly without needing to interact with the guest operating systems (the OSes running inside the VMs). Example:

1️⃣ CPU utilization: The hypervisor manages CPU allocation and can measure how much CPU time is allocated to each VM.

2️⃣ Disk I/O: Similarly, disk reads/the hypervisor manages writes to measure these metrics directly.

3️⃣ Network: Network packets go through the hypervisor, allowing it to measure network activity.

✅ OS-Level Metrics: These metrics relate to what's happening inside the virtual machine's operating system. The hypervisor doesn't have direct access to these metrics because they are encapsulated within the VM. Example:

1️⃣ 🧠 Memory Usage: While the hypervisor allocates a certain amount of memory to a VM, it doesn't see how that memory is used inside the VM. For instance, the hypervisor doesn't know which processes within the VM are consuming the most memory or if there's free memory available.

2️⃣ 💾 Disk Space: The hypervisor can observe how much data is read/written, but it doesn't know the available disk space or how it's partitioned within the VM's file system.

🛠️ How to get memory and disk space metrics into CloudWatch:

While AWS doesn't provide these metrics by default, they've made it possible for users to push custom metrics to CloudWatch. The CloudWatch agent can collect additional system-level metrics, including memory and disk space usage.

When you deploy the CloudWatch agent on an EC2 instance, the agent runs as a background service. It gathers the specified system-level metrics at defined intervals and sends them to CloudWatch using the PutMetricData API call. The agent is configurable, allowing you to specify which metrics to collect, the granularity of the data, and how frequently to send it to CloudWatch.