Integrating Elastic APM with Kubernetes Clusters for Comprehensive Monitoring

In today's cloud-native ecosystem, Kubernetes has become the standard for container orchestration, enabling teams to deploy and manage applications at scale. As microservices architectures grow, monitoring application performance across distributed systems becomes crucial. Elastic APM (Application Performance Monitoring) offers a powerful solution for tracking application performance, diagnosing errors, and gaining insight into system behaviors.

This article provides a step-by-step guide on how to link all your Kubernetes (k8s) applications with Elastic APM, ensuring comprehensive monitoring without relying on Helm charts.

---

Prerequisites

• Elastic Stack Deployment: An instance of the Elastic Stack (Elasticsearch and Kibana) is up and running. This can be self-hosted or managed via Elastic Cloud.
• APM Server: APM Server is deployed to receive data from agents and forward it to Elasticsearch.
• Kubernetes Cluster Access: You have administrative access to your Kubernetes cluster.
• Applications to Monitor: Your applications are deployed on Kubernetes, and you have access to modify their configurations.
• Elastic APM Agents: Familiarity with installing and configuring Elastic APM agents for your application languages (e.g., Java, Node.js, Python).

---

Step 1: Deploy Elastic APM Server on Kubernetes

The APM Server acts as the intake for performance data collected by APM agents running with your applications.

Option 1: Deploy Using Elastic's Helm Charts

Since you mentioned not using Helm, you might skip this option. However, for completeness:

1. Add Elastic Helm Repository:

   helm repo add elastic https://helm.elastic.co
   helm repo update

1. Deploy APM Server:

   helm install apm-server elastic/apm-server \
     --set output.elasticsearch.hosts="{<ELASTICSEARCH_HOST>}" \
     --set output.elasticsearch.username="elastic" \
     --set output.elasticsearch.password="<ELASTIC_PASSWORD>"

Replace <ELASTICSEARCH_HOST> and <ELASTIC_PASSWORD> with your Elasticsearch hostname and password.

Option 2: Deploy Manually with Kubernetes YAML Files

1. Create APM Server Deployment:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: apm-server
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: apm-server
     template:
       metadata:
         labels:
           app: apm-server
       spec:
         containers:
         - name: apm-server
           image: docker.elastic.co/apm/apm-server:8.10.0
           ports:
           - containerPort: 8200
           env:
           - name: ELASTICSEARCH_HOSTS
             value: "http://elasticsearch:9200"
           - name: ELASTICSEARCH_USERNAME
             value: "elastic"
           - name: ELASTICSEARCH_PASSWORD
             value: "<ELASTIC_PASSWORD>"

Replace the Elasticsearch connection details accordingly.

1. Create APM Server Service:

   apiVersion: v1
   kind: Service
   metadata:
     name: apm-server
   spec:
     selector:
       app: apm-server
     ports:
     - protocol: TCP
       port: 8200
       targetPort: 8200

1. Apply the Configuration:

   kubectl apply -f apm-server-deployment.yaml
   kubectl apply -f apm-server-service.yaml

---

Step 2: Configure Network Policies (Optional)

If your cluster has network policies enabled, ensure that your applications can communicate with the APM Server service.

---

Step 3: Instrument Applications with Elastic APM Agents

Each application needs to be instrumented with the Elastic APM agent for its language.

General Steps:

1. Install the Agent: Add the Elastic APM agent to your application dependencies.

2. Configure the Agent: Set the configuration to point to the APM Server.

3. Set Environment Variables: Use Kubernetes ConfigMaps and Secrets to manage configurations securely.

Example: Instrumenting a Java Application

1. Add Agent as a Dependency: Include the agent in your application build.

2. Update Deployment YAML:
   

   spec:
     containers:
     - name: java-app
       image: your-java-app-image
       env:
       - name: ELASTIC_APM_SERVER_URL
         value: "http://apm-server:8200"
       - name: ELASTIC_APM_SERVICE_NAME
         value: "java-app"
       - name: ELASTIC_APM_ENVIRONMENT
         value: "production"
     args:
       - "-javaagent:/path/to/elastic-apm-agent.jar"

Ensure the agent JAR is accessible by the application.

Example: Instrumenting a Node.js Application

1. Install the Agent:

   npm install elastic-apm-node --save

1. Modify Entry Point:

At the very start of your application:

   var apm = require('elastic-apm-node').start({
     serviceName: 'node-app',
     serverUrl: 'http://apm-server:8200',
     environment: 'production'
   });

Example: Instrumenting a Python Application

1. Install the Agent:

   pip install elastic-apm

1. Configure the Agent in Code:

   from elasticapm import Client
   client = Client({
     'SERVICE_NAME': 'python-app',
     'SERVER_URL': 'http://apm-server:8200',
     'ENVIRONMENT': 'production'
   })

---

Step 4: Automate APM Injection Across All Applications

To avoid modifying each application manually, you can automate the injection of the APM agent.

Option 1: Using Kubernetes Mutating Admission Webhook

Create a webhook that automatically injects the APM agent configurations into pods.

• Benefits:
  • Centralized configuration.
  • No code changes required.
• Steps:
  • Develop or use an existing webhook that adds environment variables and volume mounts to pods.
  • Ensure the webhook is configured to target the appropriate namespaces or labels.

Option 2: Sidecar Containers

Deploy the APM agent as a sidecar container, though this method is less common due to integration complexity.

---

Step 5: Configure Agents via Environment Variables

Use Kubernetes ConfigMaps and Secrets to manage agent configurations.

1. Create a ConfigMap for APM Configurations:

   kubectl create configmap apm-config --from-literal=ELASTIC_APM_SERVER_URL=http://apm-server:8200

1. Update Deployments to Use ConfigMap:

   envFrom:
   - configMapRef:
       name: apm-config

---

Step 6: Validate the Setup

1. Deploy the Instrumented Applications:

Apply the Kubernetes manifests with the updated configurations.

1. Check Application Logs:

Ensure that applications are starting without errors related to the APM agent.

1. Verify APM Server Receives Data:

Monitor the APM Server logs to confirm it receives data from agents.

---

Step 7: Access APM Data in Kibana

1. Open Kibana:

Navigate to your Kibana instance.

1. Go to the APM Section:

Select Observability > APM.

1. Explore Services:

You should see a list of services corresponding to your applications.

1. Analyze Metrics and Traces:

• Transactions: View request performance.
• Errors: Identify and troubleshoot issues.
• Service Maps: Visualize service dependencies.

---

Additional Considerations

Security

• Secure Communication:

  • Use TLS to encrypt traffic between APM agents and APM Server.
  • Enable authentication if not already configured.

• Kubernetes RBAC:

  • Implement Role-Based Access Control to restrict access to sensitive components.

Performance

• APM Server Scaling:

  • Monitor the resource usage of your APM Server.
  • Scale the deployment replicas as needed.

Monitoring APM Server

• Resource Limits:

  • Set appropriate resource requests and limits for the APM Server deployment.

• Health Checks:

  • Add readiness and liveness probes to the APM Server deployment.

   readinessProbe:
     httpGet:
       path: /
       port: 8200
   livenessProbe:
     httpGet:
       path: /
       port: 8200

---

Conclusion

By integrating Elastic APM with your Kubernetes cluster, you gain deep insights into your applications' performance and can proactively address issues. Automating the instrumentation process simplifies deployment and ensures consistency across your services.

Remember, while Elastic APM provides powerful features, it's essential to plan and monitor resource usage to maintain optimal performance in your cluster.

---

References

• Elastic APM Documentation: Getting Started with APM

• Elastic APM Agents:

  • Java Agent
  • Node.js Agent
  • Python Agent

• Kubernetes Documentation: Configuration Best Practices

---

Appendix: Sample Deployment with APM Injection

Below is a sample Deployment manifest demonstrating how to inject APM configurations using environment variables.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  labels:
    app: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - name: sample-app
        image: your-app-image
        env:
        - name: ELASTIC_APM_SERVER_URL
          valueFrom:
            configMapKeyRef:
              name: apm-config
              key: ELASTIC_APM_SERVER_URL
        - name: ELASTIC_APM_SERVICE_NAME
          value: "sample-app"
        - name: ELASTIC_APM_ENVIRONMENT
          value: "production"