Home/Learn/Apache Kafka/Serializers & Schema Registry

Serializers & Schema Registry

Intermediate

Producers

Serializers convert Java objects to bytes; the Confluent Schema Registry enforces Avro/JSON/Protobuf schemas and enables schema evolution with backward/forward compatibility.

Overview

Every Kafka producer serialises Java objects to bytes before publishing; every consumer deserialises bytes back to objects. For simple types, Kafka ships StringSerializer and ByteArraySerializer. For structured domain objects, Apache Avro is the most common choice in production because it provides compact binary encoding, rich type support, and schema evolution guarantees enforced by the Confluent Schema Registry. The Schema Registry is a central store of versioned schemas; producers register schemas on first publish and consumers fetch them by schema ID embedded in the message. The registry enforces compatibility rules (BACKWARD, FORWARD, FULL) preventing breaking changes from reaching consumers — the critical governance layer in a large event-driven architecture.

Avro schema and Schema Registry producer

Define an Avro schema (.avsc file), use Maven/Gradle Avro plugin to generate Java classes, then publish with KafkaAvroSerializer which registers the schema automatically.

Avro + Java — Avro schema + producer setup

// order-event.avsc
{
  "type": "record",
  "name": "OrderEvent",
  "namespace": "com.example.events",
  "fields": [
    {"name": "orderId",    "type": "string"},
    {"name": "customerId", "type": "string"},
    {"name": "amount",     "type": "double"},
    {"name": "status",     "type": {"type": "enum",
                                    "name": "OrderStatus",
                                    "symbols": ["PENDING","CONFIRMED","SHIPPED"]}}
  ]
}

# pom.xml — Avro codegen plugin
<plugin>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-maven-plugin</artifactId>
  <version>1.11.3</version>
  <executions>
    <execution>
      <goals><goal>schema</goal></goals>
      <configuration>
        <sourceDirectory>src/main/avro</sourceDirectory>
      </configuration>
    </execution>
  </executions>
</plugin>

// Producer config with Schema Registry
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
          KafkaAvroSerializer.class);
props.put("schema.registry.url", "http://schema-registry:8081");

// KafkaAvroSerializer auto-registers schema and prepends schema ID (4 bytes) to message
producer.send(new ProducerRecord<>("order-events", event.getOrderId().toString(), avroEvent));

Schema evolution compatibility rules

The Schema Registry enforces compatibility rules per subject. BACKWARD is the default: new schema can read data written by old schema. This allows adding optional fields (with defaults) but not removing required fields.

Shell — Schema Registry compatibility config

# Check / set compatibility for a subject
# GET current compatibility
curl http://schema-registry:8081/config/order-events-value

# SET to FULL_TRANSITIVE (strictest — both backward and forward for all versions)
curl -X PUT http://schema-registry:8081/config/order-events-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "FULL_TRANSITIVE"}'

# Compatibility rules:
# BACKWARD: new schema reads old data (add fields WITH defaults, remove optional)
# FORWARD:  old schema reads new data (add fields WITHOUT defaults, remove with defaults)
# FULL:     both BACKWARD and FORWARD
# *_TRANSITIVE: against all previous versions, not just the latest

# BACKWARD example — adding an optional field with default is safe:
# Old:  {"name": "orderId", "type": "string"}
# New:  {"name": "orderId", "type": "string"},
#       {"name": "channel",  "type": ["null","string"], "default": null}

# BREAKING — removing a field without default is NOT backward compatible:
# Old consumers reading "channel" will fail — rejected by registry

JSON Schema and Spring Boot integration

For teams not using Avro, Spring Kafka's JsonSerializer/JsonDeserializer with Schema Registry provides schema governance with JSON. Spring Boot auto-configures this via properties.

Properties — JSON Schema Registry vs Spring JsonDeserializer

# application.properties — JSON with Schema Registry
spring.kafka.producer.value-serializer=  io.confluent.kafka.serializers.KafkaJsonSchemaSerializer
spring.kafka.consumer.value-deserializer=  io.confluent.kafka.serializers.KafkaJsonSchemaDeserializer
spring.kafka.properties.schema.registry.url=http://schema-registry:8081
spring.kafka.properties.json.fail.unknown.properties=false

# Without Schema Registry — Spring's built-in JSON serializer
spring.kafka.producer.value-serializer=  org.springframework.kafka.support.serializer.JsonSerializer
spring.kafka.consumer.value-deserializer=  org.springframework.kafka.support.serializer.JsonDeserializer
spring.kafka.consumer.properties.spring.json.trusted.packages=com.example.events
spring.kafka.consumer.properties.spring.json.value.default.type=  com.example.events.OrderEvent

// Type header approach (avoids hardcoding target class in consumer)
// Producer adds __TypeId__ header automatically
// Consumer uses it to determine target class for deserialisation

Key Points to Remember

1Schema Registry stores versioned schemas; the schema ID (4 bytes) is prepended to every Avro/JSON Schema message.
2BACKWARD compatibility (default): add fields with defaults, remove optional fields — consumers on old schema still work.
3FULL_TRANSITIVE is the safest setting: enforces both backward and forward compatibility against all historical versions.
4Avro is more efficient (binary, no field names in each message) than JSON but requires the avsc file and codegen.
5For Spring Kafka without Schema Registry, use JsonSerializer + trusted.packages to prevent deserialization attacks.
6Schema evolution in Avro must add new fields with "default" values — fields without defaults are breaking changes.

Interview Questions

What problem does the Schema Registry solve that plain JSON serialisation does not?

EasyConfluent

Explain BACKWARD vs FORWARD schema compatibility with a concrete example.

MediumLinkedIn

How does Avro KafkaAvroSerializer embed the schema in a Kafka message?

MediumUber

What happens if a consumer tries to read a message with a schema ID not present in the registry?

HardNetflix

What is FULL_TRANSITIVE compatibility and when is it the right choice?

HardAmazon

Ask Aria about Serializers & Schema Registry

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Producer Batching & Compression

Consumer Groups