Kafka Idempotency: Delivery Guarantees in Practice

# Kafka Delivery Guarantees: Producer and Consumer Idempotency in Practice

The inventory-service is switching from in-memory storage to PostgreSQL to support the Outbox Pattern and delivery guarantees. Add a postgres-inventory container to docker-compose.yaml with image postgres:17, ports 5432:5432, and environment variables POSTGRES_USER=postgres, POSTGRES_PASSWORD=postgres, POSTGRES_DB=inventory_db.

In the inventory-service's pom.xml, add the following dependencies:

<dependency>
  <groupId>org.postgresql</groupId>
  <artifactId>postgresql</artifactId>
  <scope>runtime</scope>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>

In application.properties, specify the connection:

spring.datasource.url=jdbc:postgresql://localhost:5432/inventory_db
spring.datasource.username=postgres
spring.datasource.password=postgres

Create the inventory table:

CREATE TABLE inventory (
    id BIGSERIAL PRIMARY KEY,
    product_name VARCHAR(256) NOT NULL UNIQUE,
    quantity INT NOT NULL CHECK (quantity >= 0)
);

INSERT INTO inventory (product_name, quantity) VALUES
('Smartphone', 5),
('Tablet', 10),
('Desktop', 6);

Inventory entity:

@Entity
@Table(name = "inventory")
public class Inventory {
    @Id
    @Column(name = "id")
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(name = "product_name", nullable = false, unique = true, length = 256)
    private String productName;

    @Column(name = "quantity", nullable = false)
    private Integer quantity;

    public Inventory(String productName, Integer quantity) {
        this.productName = productName;
        this.quantity = quantity;
    }
    // getters, setters
}

Repository with deductStock:

public interface InventoryRepository extends JpaRepository<Inventory, Long> {
    @Modifying
    @Query("UPDATE Inventory i SET i.quantity = i.quantity - :quantity WHERE i.productName = :productName AND i.quantity >= :quantity")
    int deductStock(@Param("productName") String productName, @Param("quantity") Integer quantity);
}

The KafkaListener handler in the service checks stock availability and sends InventoryReservedEvent or InventoryRejectedEvent.

Setting up acks and understanding exactly-once semantics

The spring.kafka.producer.acks parameter controls acknowledgments: 0 for at-most-once, 1 for partial, all for at-least-once with replication on all followers.

Exactly-once delivery in distributed systems is impossible due to the Two Generals' Problem. It is implemented as exactly-once semantics: at-least-once delivery + producer and consumer idempotency.

Producer Idempotency: ProducerId, Epoch, and sequence number

Idempotency prevents duplicates on retry due to ack loss. It is enabled by:

spring.kafka.producer.properties.enable.idempotence=true

The broker assigns a (ProducerId, ProducerEpoch) pair to the connection:

• ProducerId: unique producer identifier
• ProducerEpoch: increments on restart, protection against zombie producers

Each message is numbered with a sequence number. The broker rejects duplicates within ProducerId/Epoch/sequence.

Zombie producer scenario:

• Producer crashes, but "revives" after a new one restarts
• Old one continues with high sequence number
• New one starts from 1 — ProducerEpoch distinguishes them

Transactional.id ensures ProducerId preservation on restart (Spring calls initTransactions()). Without it — new random ProducerId.

Critical idempotency parameters

spring.kafka.producer.acks=all
spring.kafka.producer.retries=2147483647  # >0
spring.kafka.producer.properties.max.in.flight.requests.per.connection=5  # ≤5
spring.kafka.producer.properties.enable.idempotence=true

Violation disables idempotency. Apply to order-service and inventory-service.

Consumer Idempotency Against Double Processing

The consumer may process a message twice if it crashes after committing the offset but before completing business logic. For inventory-service:

• Reads OrderPlacedEvent
• deductStock() in PostgreSQL (atomically)
• Sends InventoryReservedEvent
• Crash
• Re-reads offset, repeats deductStock() — but quantity already reduced, UPDATE won't work

This ensures idempotency at the business logic level via conditional UPDATE.

Advantages of the approach:

• No additional offset storage
• Uses DB ACID transactions
• Simple implementation via WHERE quantity >= :quantity

Key Points

• Producer Idempotency: enable.idempotence=true + acks=all + retries>0 + max.in.flight≤5
• ProducerEpoch: protection against zombie producers on restart
• Transactional.id: preserves ProducerId, requires initTransactions()
• Consumer: business logic with conditional UPDATE in DB ensures idempotence
• Exactly-once semantics: at-least-once + idempotence, not literal exactly-once

— Editorial Team