How would you perform a Kafka operations from your local machine?

Let's say you have a cloud Kafka in your application (e.g., AWS MSK). To form the connection from your local machine without code, follow these instructions:

1. Download Apache Kafka from here.

2. Extract the downloaded zip file and navigate to the "bin" folder.

3. In the "bin" folder, you will find all the .sh files, e.g., kafka-topics.sh, kafka-console-consumer.sh.

Once you've completed the setup, there's one more important step:

1. You need to create a client.properties file for AWS MSK. The configuration would mostly be like this:

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

Now we are done with the complete setup and ready to execute the first command:
### To create a Topic:
./kafka-topics.sh --create --bootstrap-server <bootstrap-server/s> --topic <topic-name> --command-config client.properties

Now you have created the topic, simply lets check the way to describe our created topic:

Describe a Topic:

./kafka-topics.sh --describe --bootstrap-server <bootstrap-server/s> --topic <topic> --command-config client.properties
see, its that easy to describe by specifying the --describe flag with the kafka-topics.sh command.

Let's move forward, by producing some message in existing topic:

Produce messages in Kafka topic:

./kafka-console-producer.sh --bootstrap-server <bootstrap-server/s> --producer.config client.properties --topic <topic-name>

Similarly lets check for consumer,

Consume messages from Kafka topic:

./kafka-console-consumer.sh --bootstrap-server <bootstrap-server/s> --consumer.config client.properties --topic <topic-name>

What if you wanted to list down all the existing topics then,

List All Topics in Kafka:

./kafka-topics.sh --bootstrap-server <bootstrap-server/s> --list --command-config client.properties

If you wanted to list down all the consumer groups then,

List All Consumer Groups in Kafka:

./kafka-consumer-groups.sh --bootstrap-server <bootstrap-server/s> --list --command-config client.properties

If you don't know what is consumer group and how it will help you then read this blog it will give you detail idea, and how you can use it in your application:
https://dev.to/codewithved/the-kafka-conundrum-choosing-between-consumer-groups-and-partitions-for-efficient-message-consumption-b8n