Felpfe Inc.
Close this search box.
call 24/7

+484 237-1364‬

Close this search box.

Overview of Kafka architecture: brokers, topics, partitions, and replication

In this section, we will explore the architecture of Apache Kafka, understanding its key components such as brokers, topics, partitions, and replication. Having a solid understanding of Kafka’s architecture is crucial for building scalable and fault-tolerant real-time data streaming applications.

  1. Brokers: Kafka runs as a cluster of servers called brokers. Brokers are responsible for handling incoming data streams from producers, storing the data in partitions, and serving it to consumers. Each broker in the cluster can handle a subset of the data and client requests.
  2. Topics: In Kafka, data streams are organized into topics. A topic is a category or feed name to which records are published by producers. Topics can be thought of as logical containers for organizing related data streams. For example, a topic could be “sensor_data” or “user_logs.”
  3. Partitions: Each topic in Kafka is divided into one or more partitions. Partitions allow data to be distributed and processed in parallel across multiple brokers. Each partition is an ordered and immutable sequence of records. The number of partitions in a topic determines the parallelism and throughput of data processing.
  4. Replication: Kafka provides a replication mechanism for fault tolerance and data durability. Each partition in Kafka has a configurable number of replicas, which are copies of the partition. Replicas are distributed across different brokers to ensure that data is replicated and available even in the event of broker failures.

Code Sample:

To illustrate the concepts of brokers, topics, partitions, and replication, consider the following code examples:

Creating a Kafka Topic (Command Line):

bin/kafka-topics.sh --create --topic my_topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 2

Producing Data to a Kafka Topic (Java):

import org.apache.kafka.clients.producer.*;

import java.util.Properties;

public class KafkaProducerExample {

    public static void main(String[] args) {
        // Configure Kafka producer
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        // Create Kafka producer
        Producer<String, String> producer = new KafkaProducer<>(props);

        // Produce a sample record
        ProducerRecord<String, String> record = new ProducerRecord<>("my_topic", "my_key", "Hello, Kafka!");
        producer.send(record, (metadata, exception) -> {
            if (exception == null) {
                System.out.println("Message sent successfully. Offset: " + metadata.offset());
            } else {
                System.out.println("Failed to send message: " + exception.getMessage());

        // Close the producer

Consuming Data from a Kafka Topic (Java):

import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.TopicPartition;

import java.util.Collections;
import java.util.Properties;

public class KafkaConsumerExample {

    public static void main(String[] args) {
        // Configure Kafka consumer
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("group.id", "my_consumer_group");

        // Create Kafka consumer
        Consumer<String, String> consumer = new KafkaConsumer<>(props);

        // Subscribe to a topic


        // Consume records
        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(1000);
            for (ConsumerRecord<String, String> record : records) {
                System.out.println("Received message: " + record.value() + ", Offset: " + record.offset());

Reference Links:

  • Apache Kafka documentation on architecture: link
  • Apache Kafka replication and fault tolerance: link

Helpful Video:

  • “Apache Kafka Architecture Explained” by Confluent: link

In this module, we have explored the core components of Apache Kafka’s architecture, including brokers, topics, partitions, and replication. We have seen that brokers handle the incoming data streams, topics organize related data streams, partitions allow for parallel processing, and replication provides fault tolerance and data durability. Through the provided code examples, we have learned how to create Kafka topics, produce data to topics, and consume data from topics using the Kafka Java API.

Understanding Kafka’s architecture is essential for designing and building scalable and fault-tolerant real-time data streaming applications. With this knowledge, you are now equipped to harness the power of Kafka’s distributed streaming platform and leverage its capabilities to handle high-volume, high-velocity data streams in real-time.

About Author
Ozzie Feliciano CTO @ Felpfe Inc.

Ozzie Feliciano is a highly experienced technologist with a remarkable twenty-three years of expertise in the technology industry.

Stream Dream: Diving into Kafka Streams
In “Stream Dream: Diving into Kafka Streams,”...
Talking in Streams: KSQL for the SQL Lovers
“Talking in Streams: KSQL for the SQL Lovers”...
Stream Symphony: Real-time Wizardry with Spring Cloud Stream Orchestration
Description: The blog post, “Stream Symphony:...
Kafka Chronicles: Saga of Resilient Microservices Communication with Spring Cloud Stream
“Kafka Chronicles: Saga of Resilient Microservices...
Tackling Security in Kafka: A Comprehensive Guide on Authentication and Authorization
As the usage of Apache Kafka continues to grow in organizations...
1 2 3 58
90's, 2000's and Today's Hits
Decades of Hits, One Station

Listen to the greatest hits of the 90s, 2000s and Today. Now on TuneIn. Listen while you code.