How I use Cloudwatch and fluentbit

Fluent Bit is deployed as a DaemonSet (a POD that runs on every node of the cluster). When Fluent Bit runs, it will read, parse and filter the logs of every POD and will enrich log with some more information.

This will enable container (open policy agent) logs available in aws cloudwatch. The log group name where the logs will be /aws/containerinsights/${CLUSTER_NAME}/application ; here CLUSTER_NAME will be "tooling" for prod.

fluentbit.yaml will have

Here’s how the ClusterRole, ClusterRoleBinding, and ConfigMap are linked and their roles in this configuration:

ClusterRole

The ClusterRole named fluent-bit-role defines the permissions that Fluent Bit requires to interact with Kubernetes resources. It specifies:

• Non-resource URL access: Allows access to /metrics with the get verb.
• Resource access: Grants permissions to namespaces, pods, pods/logs, nodes, and nodes/proxy with the get, list, and watch verbs.

ClusterRoleBinding

The ClusterRoleBinding named fluent-bit-role-binding links the ClusterRole to a subject, enabling Fluent Bit to use the permissions.

• Subject: The ServiceAccount named fluent-bit in the logging namespace.
• RoleRef: Specifies that the binding refers to the fluent-bit-role ClusterRole.

This linkage ensures that the fluent-bit ServiceAccount has the necessary permissions to collect logs and interact with Kubernetes objects.

ConfigMap

The ConfigMap named fluent-bit-config provides configuration data for Fluent Bit. It contains:

• Fluent Bit configurations: Specifies input sources (e.g., application logs), filtering (e.g., Kubernetes metadata), and output destinations (e.g., CloudWatch Logs).
• Parser definitions: Defines parsers for structured log formats, such as docker and syslog.

How They Are Linked

1. Permissions for Log Access:

   • The fluent-bit DaemonSet runs pods using the fluent-bit ServiceAccount.
   • The fluent-bit-role-binding binds the fluent-bit-role ClusterRole to the fluent-bit ServiceAccount.
   • This setup allows Fluent Bit to access logs, Kubernetes metadata, and node information.

2. Configuration Data:

   • The DaemonSet mounts the fluent-bit-config ConfigMap to /fluent-bit/etc/ within its pods.
   • Fluent Bit reads configurations from this directory to process logs according to the defined rules.

This structure ensures Fluent Bit operates with the correct permissions and configurations in a Kubernetes environment. Let me know if you need further clarification or adjustments!