Home/Learn/Apache Kafka/Message Compression

Message Compression

Intermediate

Advanced

Producer-side compression (snappy, lz4, zstd, gzip) compresses record batches; the broker stores them compressed and consumers decompress, reducing I/O and storage significantly.

Overview

Kafka compression is configured on the producer and applied per batch. The broker stores records compressed and passes the compressed batch to consumers unchanged (end-to-end compression). Decompression happens only at the consumer — the broker never decompresses for reading (except when producing with acks=all, where followers must decompress to validate). Compression dramatically reduces network I/O, broker disk usage, and consumer fetch time for text-heavy payloads (JSON, Avro, XML) — typically 40–80% size reduction. Four codecs are available: gzip (best compression, highest CPU), snappy (balanced), lz4 (fastest, Kafka default recommendation), and zstd (best ratio at moderate CPU, recommended for new deployments).

Compression codec comparison and producer configuration

Choose a codec based on your throughput/CPU trade-off. lz4 is the best default — very fast compression with good ratio. zstd (available since Kafka 2.1) achieves significantly better ratios than lz4 at similar speed. gzip has the best ratio but is CPU-intensive. snappy is a middle ground. Compression is configured on the producer; consumers auto-detect and decompress.

YAML + CLI — producer compression configuration and codec comparison

# application.yml — producer compression
spring:
  kafka:
    producer:
      compression-type: zstd   # none | gzip | snappy | lz4 | zstd
      # Compression is applied per batch — larger batches compress better
      batch-size: 65536        # 64 KB (default 16 KB)
      properties:
        linger.ms: 20          # wait 20ms to accumulate batch before sending

# Compression ratio benchmarks (JSON payloads, approximate)
# none:    1.0x ratio, 0 CPU
# snappy:  2.5x ratio, low CPU    — good for CPU-constrained producers
# lz4:     3.0x ratio, very low CPU — best throughput
# zstd:    4.0x ratio, moderate CPU — best ratio/CPU balance
# gzip:    4.5x ratio, high CPU   — best compression, highest cost

# Per-topic compression override (broker-level)
# compression.type=producer means: use whatever the producer sent (default)
# compression.type=gzip means: broker recompresses with gzip (additional CPU)

kafka-configs.sh --bootstrap-server broker:9092 \
  --entity-type topics --entity-name my-topic \
  --alter --add-config compression.type=producer

How end-to-end compression works

The producer compresses a batch of records as a single unit — compression across many records achieves much better ratios than compressing each record individually. The compressed batch is sent to the broker, stored as-is, and delivered to consumers as-is. The consumer's deserialiser handles decompression transparently. Verify compression savings by comparing network bytes with and without compression.

Java — end-to-end compression: producer config only, consumer auto-decompresses

// Compression is fully transparent to producer/consumer code
// No changes needed in produce/consume logic

// Producer — compression configured in properties
@Bean
public ProducerFactory<String, OrderEvent> producerFactory() {
    Map<String, Object> config = new HashMap<>();
    config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker:9092");
    config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
    config.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");
    config.put(ProducerConfig.BATCH_SIZE_CONFIG, 65536);
    config.put(ProducerConfig.LINGER_MS_CONFIG, 20);
    return new DefaultKafkaProducerFactory<>(config);
}

// Consumer — no compression config needed; auto-detects codec from batch header
@Bean
public ConsumerFactory<String, OrderEvent> consumerFactory() {
    Map<String, Object> config = new HashMap<>();
    config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker:9092");
    config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
    // No compression config — consumer auto-decompresses
    return new DefaultKafkaConsumerFactory<>(config);
}

Monitoring compression effectiveness

Monitor compression ratio via the producer metric compression-rate-avg (values < 1.0 indicate compression savings). Also monitor record-size-avg (average compressed size) vs topic's uncompressed equivalent. High compression savings on a topic with high throughput can significantly reduce broker disk usage and network bandwidth costs.

CLI — monitoring compression ratio and verifying effectiveness

# Kafka producer JMX metrics for compression
# kafka.producer:type=producer-metrics,client-id=<id>
#   compression-rate-avg    — average compression rate (0.0-1.0, lower = more compressed)
#   record-size-avg         — average compressed record size in bytes
#   batch-size-avg          — average batch size in bytes

# Prometheus metric via JMX Exporter
kafka_producer_metrics_compression_rate_avg{client_id="order-service"}

# Alert if compression rate > 0.9 (< 10% savings — possible codec mismatch)

# Verify broker stores compressed records
kafka-log-dirs.sh --bootstrap-server broker:9092 \
  --topic-list my-topic --describe
# Shows: logSize (compressed on-disk size)

# Consumer metric — decompression overhead
# fetch-throttle-time-avg, records-consumed-rate

# Check topic's compression type
kafka-configs.sh --bootstrap-server broker:9092 \
  --entity-type topics --entity-name my-topic --describe
# compression.type = producer (use whatever producer sent)

Key Points to Remember

1Compression is applied per batch on the producer — larger batches produce better compression ratios
2End-to-end compression: producer compresses → broker stores compressed → consumer auto-decompresses (no broker CPU cost)
3zstd is recommended for new deployments: best ratio/CPU balance; lz4 for maximum throughput on CPU-constrained producers
4compression.type=producer on the broker means "store as received" — the default and most efficient setting
5compression-rate-avg JMX metric shows effectiveness (0.3 = 70% size reduction); monitor to verify savings
6JSON and Avro payloads compress well (40–80% reduction); binary or already-compressed payloads compress poorly

Interview Questions

Where is Kafka compression configured — producer, broker, or consumer?

EasyConfluent

What is end-to-end compression in Kafka and why does the consumer need no compression configuration?

MediumLinkedIn

Compare lz4, zstd, and gzip for Kafka — when would you choose each?

MediumUber

Why does compressing larger batches give better compression ratios than compressing individual records?

MediumAmazon

What happens if the producer uses lz4 but the topic config has compression.type=gzip?

HardNetflix

Ask Aria about Message Compression

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Dead Letter Topic (DLT)

Kafka Cluster Scaling