Kafka - Overview

Agenda

• Kafka Architecture
  • Basic concepts(topic/partition/consumer group/commit log/offset)
• Delver into the process of sending a message to the broker
• Technical Highlights

Architecture - Overview

Architecture - Detail

Architecture - Basic Concepts

Topics

A Kafka topic (https://www.instaclustr.com/support/documentation/kafka/using-kafka/topic-management/) defines a channel through which data is streamed.Producers publish messages to topics,and consumers read messages from the topic they subscribe to.

Topics organize and structure messages,with particular types of messages published to particular topics.Topics are identified by unique names within a Kafka cluster, and there is no limit on the number of topics that can be created.

Partitions

Within the Kafka cluster, topics are divided into partitions (https://www.instaclustr.com/blog/the-power-of-kafka-partitions-how-to-get-the-most-out-of-your-kafka-cluster/), and the partitions are replicated across
brokers. From each partition, multiple consumers can read from a topic in parallel. It's also
possible to have producers add a key to a message-all messages with the same key will go to
the same partition.enter image description here
While messages are added and stored within
partitions in sequence,messages without keys are
written to partitions in a round robin fashion.By
leveraging keys,you can guarantee the order of
processing for messages in Kafka that share the
same key.This is a particularly useful feature for
applications that require total control over records.
There is no limit on the number of Kafka
partitions that can be created(subject to the
processing capacity of a cluster).