DEV Community

Subham Nandi
Subham Nandi

Posted on

Day 5 - Logging with EFK Stack

Logging: An Introduction

Logs are messages generated by applications or systems to provide insights into their operations, helping end-users or developers understand what is happening at any given point. In cases where an application isn’t functioning as expected, logs serve as a key troubleshooting tool. Logs allow for error tracking and issue identification, making it easier to pinpoint where and why an application might be failing.

Consider a simple example of a program for adding two numbers. If this program doesn’t have any logging, a user may struggle to understand what type of input is required or the purpose of each input prompt. With logging, however, the program could clearly communicate its steps and requirements, making it more user-friendly.

In complex systems, logging becomes crucial because it enables developers to trace issues without manually navigating through extensive codebases. With properly implemented logging, developers can quickly understand the context and the possible causes of errors, which significantly reduces troubleshooting time.

Centralized Logging with the EFK Stack

In a large-scale deployment, such as a Kubernetes cluster with multiple applications, tracking logs across individual services can be cumbersome. This is where a centralized logging system like the EFK stack is invaluable. The EFK stack simplifies log management by collecting logs from various services into a single, accessible repository.

Components of EFK Stack

  1. Elasticsearch (E) – Acts as the database to store logs, enabling fast retrieval and querying of log data.
  2. Fluent Bit (F) – A lightweight log collector and forwarder, installed on each node as a DaemonSet to gather logs from all pods and forward them to Elasticsearch.
  3. Kibana (K) – A visualization tool for Elasticsearch, offering a graphical interface to search, view, and analyze logs.

Together, these components create an efficient logging pipeline:

  • Fluent Bit collects logs from the Kubernetes pods.
  • Logs are sent to Elasticsearch, where they are stored and indexed.
  • Kibana visualizes and queries the logs, enabling users to search for specific issues or analyze patterns.

Why Use Fluent Bit over Logstash?

While some organizations use the ELK stack (Elasticsearch, Logstash, and Kibana), Fluent Bit is often preferred over Logstash because of its lightweight nature. Here’s how they differ:

  • Fluent Bit is a log forwarder that collects and sends logs to Elasticsearch with minimal processing.
  • Logstash is a log aggregator with advanced filtering, parsing, and enrichment capabilities, which can be resource-intensive.

For most use cases, especially in Kubernetes, Fluent Bit is more than sufficient and is favored for its simplicity and lower resource consumption. However, if you need to perform complex transformations on your logs before storage, Logstash may be the better option.

Detailed Architecture of the EFK Stack

  1. Fluent Bit as DaemonSet

    Fluent Bit is deployed as a DaemonSet on Kubernetes, meaning an instance runs on every node in the cluster. Each instance collects logs from the node’s pods and sends them to Elasticsearch. Fluent Bit is configured to monitor specific directories (such as /var/log/containers), where logs are stored.

  2. Elasticsearch for Log Storage

    Once Fluent Bit forwards logs, they’re stored in Elasticsearch. Elasticsearch indexes the logs, making it easy to search and query for specific events or patterns. Because logs can accumulate quickly, especially in systems with numerous microservices, storage management becomes critical. Elasticsearch can be configured to take snapshots of logs, which can be stored in persistent volumes like AWS EBS to prevent data loss and aid in long-term log retention.

  3. Kibana for Visualization

    Kibana serves as the front-end for Elasticsearch, providing a user-friendly interface where users can visualize log data. Through Kibana, users can create custom dashboards, run queries, and analyze log patterns to identify potential issues or monitor the health of the system.

Querying Logs in EFK

With centralized log storage in Elasticsearch, you can execute queries to filter logs based on specific messages, errors, or fields. This is especially valuable in incident response scenarios where logs from specific timeframes or services are needed.

For instance, if a database connection issue intermittently appears in your services, you could search across all logs for "DB connection timeout" or "connection refused." Kibana will display instances of this error, showing which services are affected and when, allowing you to respond swiftly.

Here is a detailed, step-by-step demo for setting up an ElasticSearch (ES), Fluent Bit, and Kibana (EFK) logging stack on a Kubernetes cluster, specifically using AWS Elastic Kubernetes Service (EKS). This stack facilitates log forwarding, log storage, and visualization for Kubernetes applications.

1. Prepare IAM Role and Kubernetes Service Account

Goal: Grant ElasticSearch access to create EBS volumes using an IAM role mapped to a Kubernetes service account.

  • Step 1: Create an IAM role with permissions to manage EBS volumes. Attach policies like AmazonEBSCSIDriverPolicy.
  • Step 2: Map the IAM role to a Kubernetes service account. This account allows the Kubernetes ElasticSearch service to use the IAM role’s permissions. 
   eksctl create iamserviceaccount \
       --name <service_account_name> \
       --namespace <namespace_name> \
       --cluster <cluster_name> \
       --attach-policy-arn arn:aws:iam::<account_id>:policy/<policy_name> \
       --approve
Enter fullscreen mode Exit fullscreen mode
  • Step 3: Confirm the IAM role name by running: 
   aws iam get-role --role-name <IAM_role_name>
Enter fullscreen mode Exit fullscreen mode

2. Install the EBS CSI Driver

Goal: Use the Container Storage Interface (CSI) driver to dynamically create EBS volumes for ElasticSearch.

  • Step 1: Deploy the AWS EBS CSI driver on EKS: 
   eksctl create addon --name aws-ebs-csi-driver --cluster <cluster_name> --service-account-role-arn <IAM_role_arn>
Enter fullscreen mode Exit fullscreen mode
  • Step 2: Verify the driver installation with: 
   kubectl get pods -n kube-system | grep ebs
Enter fullscreen mode Exit fullscreen mode

3. Create the Logging Namespace

Goal: Organize EFK resources by creating a dedicated namespace.

   kubectl create namespace logging
Enter fullscreen mode Exit fullscreen mode

4. Deploy ElasticSearch

Goal: Set up ElasticSearch using Helm with EBS as the storage backend.

  • Step 1: Add the ElasticSearch Helm repository if not already added: 
   helm repo add elastic https://helm.elastic.co
   helm repo update
Enter fullscreen mode Exit fullscreen mode
  • Step 2: Install ElasticSearch with a gp2 EBS-backed storage class: 
   helm install elasticsearch elastic/elasticsearch \
       --namespace logging \
       --set volumeClaimTemplate.storageClassName=gp2
Enter fullscreen mode Exit fullscreen mode
  • Step 3: Retrieve the ElasticSearch username and password to be used later: 
   ELASTIC_USERNAME=$(kubectl get secret elasticsearch-es-elastic-user -o go-template='{{.data.elastic | base64decode}}' -n logging)
Enter fullscreen mode Exit fullscreen mode

5. Deploy Kibana

Goal: Install Kibana to visualize the logs stored in ElasticSearch.

  • Step 1: Install Kibana using Helm, exposing it through a LoadBalancer for UI access. 
   helm install kibana elastic/kibana \
       --namespace logging \
       --set service.type=LoadBalancer
Enter fullscreen mode Exit fullscreen mode
  • Step 2: Retrieve the Kibana URL: 
   kubectl get svc kibana-kibana -n logging
Enter fullscreen mode Exit fullscreen mode

6. Configure Fluent Bit

Goal: Use Fluent Bit to collect and forward logs to ElasticSearch.

  • Step 1: Prepare a values.yaml file for Fluent Bit configuration with ElasticSearch credentials:

     output:
   host: "<ElasticSearch_host>"
   port: 9200
   http_user: "<ElasticSearch_username>"
   http_passwd: "<ElasticSearch_password>"

  • Step 2: Add Fluent Bit Helm repository and install it:

   helm repo add fluent https://fluent.github.io/helm-charts
   helm repo update
   helm install fluent-bit fluent/fluent-bit --namespace logging -f values.yaml
Enter fullscreen mode Exit fullscreen mode

7. Verify Fluent Bit and EFK Setup

Goal: Ensure Fluent Bit is forwarding logs to ElasticSearch and they’re viewable in Kibana.

  • Step 1: Confirm Fluent Bit logs are processed by checking Fluent Bit’s pod logs: 
   kubectl logs <fluent_bit_pod_name> -n logging
Enter fullscreen mode Exit fullscreen mode

  • Step 2: Access Kibana and search logs in the Discover section:

    • Go to the Discover tab in Kibana.
    • Create a data view using ElasticSearch’s index pattern (log-*).
    • You should see logs from your Kubernetes applications and Fluent Bit.

8. Deploy a Test Application

Goal: Deploy an application to verify end-to-end log forwarding.

  • Step 1: Create a namespace and apply manifests for a sample app: 
   kubectl create namespace dev
   kubectl apply -f <application_manifest.yaml> -n dev
Enter fullscreen mode Exit fullscreen mode
  • Step 2: Check Fluent Bit logs to verify that logs from the test application are being processed.

9. Use Kibana to Query and Visualize Logs

Goal: Visualize specific logs and explore Kibana’s querying capabilities.

  • Step 1: Use Kibana Query Language (KQL) to filter logs by namespaces or specific messages.

    • Example: namespace : "dev"

  • Step 2: Save custom views for frequently queried log patterns.

Top comments (0)

Read next

aiosoftware_dev profile image

Enhancing Customer Support with a Live Chat Widget on Your Website

Andrei AIO Software -

justaguyfrombr profile image

Menu de Game Retrô com Godot4

Misael Braga de Bitencourt -

matthiaskrt profile image

Integrating what3words in Your Angular 18 Application

Matthias -

bradyguyc profile image

Azure OpenAI C# some Learnings

Brady Guy Chambers -