Logging: An Introduction
Logs are messages generated by applications or systems to provide insights into their operations, helping end-users or developers understand what is happening at any given point. In cases where an application isn’t functioning as expected, logs serve as a key troubleshooting tool. Logs allow for error tracking and issue identification, making it easier to pinpoint where and why an application might be failing.
Consider a simple example of a program for adding two numbers. If this program doesn’t have any logging, a user may struggle to understand what type of input is required or the purpose of each input prompt. With logging, however, the program could clearly communicate its steps and requirements, making it more user-friendly.
In complex systems, logging becomes crucial because it enables developers to trace issues without manually navigating through extensive codebases. With properly implemented logging, developers can quickly understand the context and the possible causes of errors, which significantly reduces troubleshooting time.
Centralized Logging with the EFK Stack
In a large-scale deployment, such as a Kubernetes cluster with multiple applications, tracking logs across individual services can be cumbersome. This is where a centralized logging system like the EFK stack is invaluable. The EFK stack simplifies log management by collecting logs from various services into a single, accessible repository.
Components of EFK Stack
- Elasticsearch (E) – Acts as the database to store logs, enabling fast retrieval and querying of log data.
- Fluent Bit (F) – A lightweight log collector and forwarder, installed on each node as a DaemonSet to gather logs from all pods and forward them to Elasticsearch.
- Kibana (K) – A visualization tool for Elasticsearch, offering a graphical interface to search, view, and analyze logs.
Together, these components create an efficient logging pipeline:
- Fluent Bit collects logs from the Kubernetes pods.
- Logs are sent to Elasticsearch, where they are stored and indexed.
- Kibana visualizes and queries the logs, enabling users to search for specific issues or analyze patterns.
Why Use Fluent Bit over Logstash?
While some organizations use the ELK stack (Elasticsearch, Logstash, and Kibana), Fluent Bit is often preferred over Logstash because of its lightweight nature. Here’s how they differ:
- Fluent Bit is a log forwarder that collects and sends logs to Elasticsearch with minimal processing.
- Logstash is a log aggregator with advanced filtering, parsing, and enrichment capabilities, which can be resource-intensive.
For most use cases, especially in Kubernetes, Fluent Bit is more than sufficient and is favored for its simplicity and lower resource consumption. However, if you need to perform complex transformations on your logs before storage, Logstash may be the better option.
Detailed Architecture of the EFK Stack
Fluent Bit as DaemonSet
Fluent Bit is deployed as a DaemonSet on Kubernetes, meaning an instance runs on every node in the cluster. Each instance collects logs from the node’s pods and sends them to Elasticsearch. Fluent Bit is configured to monitor specific directories (such as/var/log/containers
), where logs are stored.Elasticsearch for Log Storage
Once Fluent Bit forwards logs, they’re stored in Elasticsearch. Elasticsearch indexes the logs, making it easy to search and query for specific events or patterns. Because logs can accumulate quickly, especially in systems with numerous microservices, storage management becomes critical. Elasticsearch can be configured to take snapshots of logs, which can be stored in persistent volumes like AWS EBS to prevent data loss and aid in long-term log retention.Kibana for Visualization
Kibana serves as the front-end for Elasticsearch, providing a user-friendly interface where users can visualize log data. Through Kibana, users can create custom dashboards, run queries, and analyze log patterns to identify potential issues or monitor the health of the system.
Querying Logs in EFK
With centralized log storage in Elasticsearch, you can execute queries to filter logs based on specific messages, errors, or fields. This is especially valuable in incident response scenarios where logs from specific timeframes or services are needed.
For instance, if a database connection issue intermittently appears in your services, you could search across all logs for "DB connection timeout" or "connection refused." Kibana will display instances of this error, showing which services are affected and when, allowing you to respond swiftly.
Here is a detailed, step-by-step demo for setting up an ElasticSearch (ES), Fluent Bit, and Kibana (EFK) logging stack on a Kubernetes cluster, specifically using AWS Elastic Kubernetes Service (EKS). This stack facilitates log forwarding, log storage, and visualization for Kubernetes applications.
1. Prepare IAM Role and Kubernetes Service Account
Goal: Grant ElasticSearch access to create EBS volumes using an IAM role mapped to a Kubernetes service account.
-
Step 1: Create an IAM role with permissions to manage EBS volumes. Attach policies like
AmazonEBSCSIDriverPolicy
. - Step 2: Map the IAM role to a Kubernetes service account. This account allows the Kubernetes ElasticSearch service to use the IAM role’s permissions.
eksctl create iamserviceaccount \
--name <service_account_name> \
--namespace <namespace_name> \
--cluster <cluster_name> \
--attach-policy-arn arn:aws:iam::<account_id>:policy/<policy_name> \
--approve
- Step 3: Confirm the IAM role name by running:
aws iam get-role --role-name <IAM_role_name>
2. Install the EBS CSI Driver
Goal: Use the Container Storage Interface (CSI) driver to dynamically create EBS volumes for ElasticSearch.
- Step 1: Deploy the AWS EBS CSI driver on EKS:
eksctl create addon --name aws-ebs-csi-driver --cluster <cluster_name> --service-account-role-arn <IAM_role_arn>
- Step 2: Verify the driver installation with:
kubectl get pods -n kube-system | grep ebs
3. Create the Logging Namespace
Goal: Organize EFK resources by creating a dedicated namespace.
kubectl create namespace logging
4. Deploy ElasticSearch
Goal: Set up ElasticSearch using Helm with EBS as the storage backend.
- Step 1: Add the ElasticSearch Helm repository if not already added:
helm repo add elastic https://helm.elastic.co
helm repo update
-
Step 2: Install ElasticSearch with a
gp2
EBS-backed storage class:
helm install elasticsearch elastic/elasticsearch \
--namespace logging \
--set volumeClaimTemplate.storageClassName=gp2
- Step 3: Retrieve the ElasticSearch username and password to be used later:
ELASTIC_USERNAME=$(kubectl get secret elasticsearch-es-elastic-user -o go-template='{{.data.elastic | base64decode}}' -n logging)
5. Deploy Kibana
Goal: Install Kibana to visualize the logs stored in ElasticSearch.
- Step 1: Install Kibana using Helm, exposing it through a LoadBalancer for UI access.
helm install kibana elastic/kibana \
--namespace logging \
--set service.type=LoadBalancer
- Step 2: Retrieve the Kibana URL:
kubectl get svc kibana-kibana -n logging
6. Configure Fluent Bit
Goal: Use Fluent Bit to collect and forward logs to ElasticSearch.
-
Step 1: Prepare a
values.yaml
file for Fluent Bit configuration with ElasticSearch credentials:
output: host: "<ElasticSearch_host>" port: 9200 http_user: "<ElasticSearch_username>" http_passwd: "<ElasticSearch_password>"
Step 2: Add Fluent Bit Helm repository and install it:
helm repo add fluent https://fluent.github.io/helm-charts
helm repo update
helm install fluent-bit fluent/fluent-bit --namespace logging -f values.yaml
7. Verify Fluent Bit and EFK Setup
Goal: Ensure Fluent Bit is forwarding logs to ElasticSearch and they’re viewable in Kibana.
- Step 1: Confirm Fluent Bit logs are processed by checking Fluent Bit’s pod logs:
kubectl logs <fluent_bit_pod_name> -n logging
-
Step 2: Access Kibana and search logs in the Discover section:
- Go to the
Discover
tab in Kibana. - Create a data view using ElasticSearch’s index pattern (
log-*
). - You should see logs from your Kubernetes applications and Fluent Bit.
- Go to the
8. Deploy a Test Application
Goal: Deploy an application to verify end-to-end log forwarding.
- Step 1: Create a namespace and apply manifests for a sample app:
kubectl create namespace dev
kubectl apply -f <application_manifest.yaml> -n dev
- Step 2: Check Fluent Bit logs to verify that logs from the test application are being processed.
9. Use Kibana to Query and Visualize Logs
Goal: Visualize specific logs and explore Kibana’s querying capabilities.
-
Step 1: Use Kibana Query Language (KQL) to filter logs by namespaces or specific messages.
- Example:
namespace : "dev"
- Example:
Step 2: Save custom views for frequently queried log patterns.
Top comments (0)