Kafka is a powerful, open-source stream processing platform that has become increasingly popular in recent years for its ability to handle large amounts of data and real-time streaming scenarios. Node.js, on the other hand, is a JavaScript runtime built on Chrome's V8 JavaScript engine, known for its ability to create scalable, high-performance network applications. In this blog post, we'll explore the ways in which Kafka and Node.js can be used together to build efficient, real-time data processing systems that can handle high volumes of data and traffic.
👋 You can also check my weekly posts in Linkedin
Requirements:
- Basic knowledge of nodejs
- Basic Knowledge of docker
- Basic knowledge of javascript
What we will do:
- Set up kafka.
- Set up two Node.js apps: one for the producer and one for the consumer.
- sent message from producer and consume it
- scenario: A user will be automatically created in the consumer when a producer is created.
Discuss some crucial terms:
- producer: a producer is a program that writes data to a Kafka cluster. The producer sends messages to one or more topics.
- consumer: a consumer is a program that reads data from a Kafka cluster.
- borker: A broker is a Kafka server that stores and serves data.
- topic: topics are the channels through which producers and consumers communicate
- partition: a partition is a unit of data storage in a topic. Each topic is divided into one or more partitions
Let's play the game:
To begin, we will install Kafka
Install Docker: Kafka is typically run in a Docker container, so you will need to install Docker on your system if it is not already installed. Follow the instructions for your operating system to install Docker.
-
we will need two Docker images:
- wurstmeister/zookeeper
- wurstmeister/kafka
1️⃣ Create a docker-compose.yml file and include the following in it.
version: "3"
services:
zookeeper:
image: 'wurstmeister/zookeeper:latest'
ports:
- '2181:2181'
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
image: 'wurstmeister/kafka:latest'
ports:
- '9092:9092'
environment:
- KAFKA_BROKER_ID=1
- KAFKA_LISTENERS=PLAINTEXT://:9092
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
depends_on:
- zookeeper
2️⃣ Run the following command to start the containers (-d to start the containers in the foreground)
docker-compose up -d
3️⃣ Start kafka shell:(change kafka_kafka_1 by your container name)
docker exec -it <u>kafka_kafka_1</u> /bin/sh
4️⃣ Create Topic:
kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic your-topic-name-here
Additional CLIs:
▪️ kafka-topics.sh --list --zookeeper zookeeper:2181: list all topics.
▪️ docker ps: to show running containers.
and that's all.
Next step: Set up two Node.js apps
1️⃣ create a package.json:
npm init
2️⃣ install packages:
npm i express nodemon dotenv mongoose kafkajs
3️⃣ Now we will configure our producer:
4️⃣ Add the following two lines: (to your index)
the result : 👇
5️⃣ configure our consumer by adding this following in it:
6️⃣ now we will start our scenario:
First of all, create your model, controller, and route. Inside the controller, you must add the following two lines:
import {run} from "../utils/producer.js";
run(response);
like this example below: 👇
7️⃣ update our producer:
8️⃣ update our consumer:
THE RESULT 👏
from producer:
from consumer:
In conclusion, Apache Kafka is a powerful distributed streaming platform that enables real-time data processing and stream processing applications. It can be integrated with Node.js, a popular JavaScript runtime, through the use of Kafka client libraries, allowing developers to build event-driven and data-driven applications using the two technologies together. Kafka is known for its scalability, high performance, and ability to handle high volumes of data, making it a valuable tool for building robust and scalable applications. However, it can also be complex to set up and maintain, and requires a good understanding of its architecture and configuration options in order to use it effectively. Overall, Kafka and Node.js can be a powerful combination for building efficient and scalable applications, and are worth considering for any project that requires real-time data processing or stream processing capabilities.
Top comments (8)
great post!
curious, why have you added two brokers of the same address? (assuming theres a reason, if not just say :P)
The connection between producers and consumers in a Kafka cluster can be either to the same broker or to different brokers, depending on the configuration of the Kafka cluster and the requirements of the application. Factors that can influence this decision include the structure of the cluster, the goals of the application, and the desired level of scalability and performance.
Hi first congrats for your post.
I need get the last 10 messages ( desc ) for find a particular message. I tried with seek but the connection never ends.
Could you sent me a example to get "N" messages and disconnect if found or not this message
Good overview 👍
What use case are you using kafka for ?
Generally, I use it for communication between microservices, and thisi is a small example
Good luck mate!🔥
Hi Tarek thank you for sharing 🔥
Thanks for sharing Tarek. That's so helpful