Home/Learn/Apache Kafka/Apache Kafka Introduction

Apache Kafka Introduction

Beginner

Fundamentals

Kafka is a distributed, fault-tolerant, high-throughput event streaming platform used for real-time data pipelines, messaging, and event-driven architectures.

Overview

Apache Kafka is an open-source distributed event streaming platform originally developed at LinkedIn and donated to the Apache Software Foundation. At its core, Kafka is a distributed commit log — producers append records to topics, and consumers read them at their own pace. Unlike traditional message queues, Kafka persists records to disk for a configurable retention period, allowing consumers to replay events and multiple independent consumer groups to read the same topic. Kafka handles millions of messages per second with low latency, making it the backbone of real-time data pipelines, event-driven microservices, stream processing, and activity tracking.

Core Concepts

A Kafka topic is a named, ordered, immutable log split into partitions. Producers write to topics; consumers read from them. Each record has an offset — its position within a partition. Consumer groups enable parallel consumption: each partition is assigned to exactly one consumer in the group.

Java — minimal producer and consumer

// Producer: send a message
ProducerRecord<String, String> record = new ProducerRecord<>(
    "orders",        // topic
    "order-123",     // key  → determines partition
    "{"amount":99}" // value
);
producer.send(record, (metadata, exception) -> {
    if (exception == null) {
        System.out.printf("Sent to %s[%d] at offset %d%n",
            metadata.topic(), metadata.partition(), metadata.offset());
    }
});

// Consumer: poll loop
consumer.subscribe(List.of("orders"));
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> r : records) {
        System.out.printf("[%d] %s = %s%n", r.offset(), r.key(), r.value());
    }
    consumer.commitSync();
}

Kafka in Spring Boot

spring-kafka provides KafkaTemplate for sending and @KafkaListener for consuming. Auto-configuration reads spring.kafka.* properties and creates the necessary factories.

Java — Spring Boot Kafka setup

// application.properties
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=order-service
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer

// Producer
@Service
public class OrderEventPublisher {
    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;

    public void publish(OrderEvent event) {
        kafkaTemplate.send("orders", event.getOrderId(), event);
    }
}

// Consumer
@Component
public class OrderEventConsumer {
    @KafkaListener(topics = "orders", groupId = "inventory-service")
    public void onOrder(OrderEvent event) {
        // process event
    }
}

Key Use Cases

Kafka excels at scenarios requiring durable, high-throughput, and replayable event streams. Common patterns include event sourcing, CDC (change data capture), log aggregation, metrics pipelines, and inter-service communication in microservices.

Conceptual — Kafka use cases

// Use cases at a glance:

// 1. Event-driven microservices — decouple services via events
// OrderService → Kafka topic "orders" → InventoryService, ShippingService

// 2. CDC with Debezium — stream DB changes to Kafka
// MySQL binlog → Debezium connector → Kafka topic "db.orders"

// 3. Log aggregation — collect logs from many services
// App logs → Filebeat → Kafka → Elasticsearch

// 4. Metrics pipeline
// App metrics → Kafka → Kafka Streams (aggregation) → InfluxDB

// 5. Event sourcing — topic is the source of truth
// All state changes persisted as events; rebuild state by replaying

Key Points to Remember

1Kafka is a distributed, durable, ordered commit log — not a traditional queue.
2Topics are split into partitions; each partition is an ordered, immutable sequence.
3Consumer groups enable parallel processing; one partition → one consumer per group.
4Offsets track each consumer group's position — consumers control when they commit.
5Records are retained for a configurable period (default 7 days) regardless of consumption.
6Common uses: event-driven microservices, CDC, log aggregation, stream processing, event sourcing.

Interview Questions

What is the difference between a Kafka topic and a partition?

EasyInfosys

How does Kafka differ from RabbitMQ?

EasyTCS

What is a consumer group and how does partition assignment work?

MediumFlipkart

How does Kafka guarantee message ordering?

MediumAmazon

Explain the role of offsets and the difference between at-least-once, at-most-once, and exactly-once delivery.

HardNetflix

Ask Aria about Apache Kafka Introduction

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

All Kafka Concepts

Kafka vs Traditional Messaging