DEV Community

Rajat Kumar Nayak
Rajat Kumar Nayak

Posted on

Unleashing Real-Time Data: Setting Up Kafka on EC2 and Connecting from Your Local Machine

This blog post will show you how to install Apache Kafka on an Amazon EC2 instance and connect to it locally to take use of the potential of real-time data. We will guide you through every step of the process, including setting up Kafka on the cloud and managing data streams using a producer and consumer. By the end, you will be able to use a specialized GitHub repository to conduct an analysis of the stock market. For those who want to learn more about real-time analytics and data streaming, this extensive guide is ideal.

So let's begin

Steps to Setup Kafka

Create an EC2 Instance

Create EC2 Instance

Edit Security Group

Edit Security Group

Connect to EC2 Instance
1.Go to the Downloads folder where the key is located
2.Fetch the SSH Connect Command

  • Sign in to the AWS Management Console.
  • Navigate to the EC2 service.
  • In the EC2 Dashboard, click on "Instances" in the left sidebar.
  • Select the instance you want to connect to by clicking on its checkbox.
  • Click the "Connect" button at the top of the page.
  • In the "Connect to instance" page that opens, select the "SSH client" tab.
  • Scroll down to the section titled "Example". Here, you'll see the exact SSH command to use for connecting to your instance.
  • You can copy this command directly from the dashboard. It will look something like this: 
ssh -i "your-key-name.pem" ec2-user@ec2-12-34-56-78.compute-1.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

Now you are connected to remote EC2 Instance

Run this command to Download Kafka

wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
Enter fullscreen mode Exit fullscreen mode

Install Amazon Corretto (OpenJDK):

Amazon Corretto is a no-cost, multiplatform, production-ready distribution of the Open Java Development Kit (OpenJDK).

sudo yum install -y java-1.8.0-amazon-corretto-devel
Enter fullscreen mode Exit fullscreen mode

Navigate to the kafka folder

 cd kafka_2.13-3.7.0/
Enter fullscreen mode Exit fullscreen mode

Now Similarly open two terminal connecting to the EC2 Instance

Image description

In the first terminal run the following command to start zookeeper:

bin/zookeeper-server-start.sh config/zookeeper.properties
Enter fullscreen mode Exit fullscreen mode

In the second terminal run the following command to start server:

export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M"
Enter fullscreen mode Exit fullscreen mode 
cd kafka_2.13-3.7.0/
Enter fullscreen mode Exit fullscreen mode 
bin/kafka-server-start.sh config/server.properties
Enter fullscreen mode Exit fullscreen mode

Hurray you can see your server starting !!

Now stop both the services and now focus on an important step.

There is an important step here:
To make our local system connect to the Kafka that is on the remote EC2 instance we need to make a little bit of configurational changes.

Run the following command in your terminal

sudo nano config/server.properties
Enter fullscreen mode Exit fullscreen mode

Image description

Find this line:
ADVERTISED_LISTENERS

  • In your case, it will be localhost.
  • Change it to the public IPV4 of your instance.

Image description

  • Press CTRL+X and SHIFT+Y and then Enter

And we are done !!

We can check the running of kafka

cd kafka_2.13-3.7.0
Enter fullscreen mode Exit fullscreen mode 
bin/kafka-topics.sh --create --topic sample_topic --bootstrap-server {Public IP of your EC2 Instance:9092} --replication-factor 1 --partitions 1
Enter fullscreen mode Exit fullscreen mode
  • Start Producer: 
bin/kafka-console-producer.sh --topic sample_topic --bootstrap-server {Public IP of your EC2 Instance:9092}
Enter fullscreen mode Exit fullscreen mode

  • Start Consumer:

  • Duplicate the session & enter in a new console --

cd kafka_2.13-3.7.0/
Enter fullscreen mode Exit fullscreen mode 
bin/kafka-console-consumer.sh --topic sample_topic --bootstrap-server {Public IP of your EC2 Instance:9092}
Enter fullscreen mode Exit fullscreen mode

Write some messages in producer side and you can receive the same in consumer side.

Now we can build our project locally and by properly mentioning the details we can connect to the Kakfa.

I have worked on a project that leads to stock market analysis and plots the graph in real time. You can find the code below.

https://github.com/rajat-gith/stock-market-analysis.git

Top comments (1)

Collapse
 
diedataou profile image
Ajit Kumar Jena •

Interesting one. Keep going!!

Read next

gulshan_sharma_7d2daa9d67 profile image

How to integrate GitHub CopilotKit with Prisma Integration into your nextJs project Using OpenAI

Gulshan Sharma -

s3cloudhub profile image

How to Force Delete a Directory/Folder in Linux

S3CloudHub -

learninggs profile image

Body Fat Percentage Calculator - Calculate Your Body Fat Easily

gaganjot -

dms profile image

🌟 Inspiring Words from the President of India at NIT Raipur Convocation

D Meenakshi Sundaram -