Get up and running Apache Kafka using Strimzi on Kubernetes

It's first post on series of blog posts on Apache Kafka using Strimzi in Kubernetes. In this series we'll see how we can get up and running Apache Kafka using Strimzi on Kubernetes, export metrics, use exported metrics in Prometheus and finally build beautiful graphs in Grafana.

Perquisites:

kubectl

Welcome to Strimzi:

Strimzi is an open source project licensed under Apache License 2.0 that is part of the Cloud Native Computing Foundation (CNCF) as a sandbox project. Its main focus is running Apache Kafka on Kubernetes while providing container images for Apache Kafka itself, Zookeeper, and other components that are part of the Strimzi ecosystem.

Leveraging the Kubernetes Operator pattern, it addresses the whole lifecycle from creating, managing, and monitoring Kafka clusters to managing all the related entities like topics and users. You get a real Kubernetes-native experience for handling all of the components in the Apache Kafka ecosystem.

Get up and running Apache Kafka using Strimzi on Kubernetes:

Follow the simple steps to install Strimzi Kafka on Kubernetes:

1. Install Strimzi Operator, before running this command please make sure to create kafka namespace.

kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka'

1. Install basic Kafka cluster and Zookeeper with metrics enabled. We'll see in future post how we can use exported metrics in Prometheus and also build dashboard on top of those metrics in Grafana.

kubectl apply -f https://github.com/waqasdilawar/strimzi-kafka-k8s/blob/main/kafka/01-strimzi-kafka-with-metrics.yaml -n kafka

1. Try producing messages in a topic. Before producing message to a topic make sure you have Strimzi operator and it's different components are up and running as shown in below image.
   
   kubectl -n kafka run kafka-producer -ti --image=quay.io/strimzi/kafka:0.25.0-kafka-2.8.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --broker-list my-cluster-kafka-bootstrap:9092 --topic my-topic

2. In another terminal we can use a consumer to receive messages.

kubectl -n kafka run kafka-consumer -ti --image=quay.io/strimzi/kafka:0.25.0-kafka-2.8.0 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic --from-beginning

Enjoy your Apache Kafka cluster, running on Kubernetes!

Reference: Source code, Strimzi Quick start