Lags retry Spring Kafka: how to fix

Non-Blocking Retry in Spring Kafka: Diagnosing and Fixing Lag Issues

When implementing asynchronous retry for Kafka using @RetryableTopic in Spring Boot 3.2 and Spring Kafka 3.1, lag builds up on retry topics under load. The main topic processes messages evenly, but retry-0 accumulates messages for 10–12 minutes, then partially clears and stalls again. This causes uneven processing.

Metrics show sporadic fetch request spikes instead of consistent polling. The consumer periodically polls the broker for heartbeats, but actual reading happens sporadically with high latency.

Key Metrics for Analysis

For diagnosis, gather these Spring Kafka metrics:

• kafka_consumer_time_between_poll_avg / max — intervals between polls.
• kafka_consumer_last_poll_seconds_ago — time since last poll.
• kafka_consumer_poll_idle_ratio_avg — proportion of idle polls.
• kafka_consumer_fetch_manager_fetch_rate / fetch_latency_avg — fetch speed and latency.
• spring_kafka_listener_seconds_count / sum — listener processing time.

Graphs of fetch-rate and listener invocations confirm: processing activates in bursts, poll idle is high.

Diagnosis via Thread Dump

Thread dump revealed ~180 threads in TIMED_WAITING on ScheduledThreadPoolExecutor$DelayedWorkQueue.take. This indicates scheduler overload, used for delayed partition resume.

Pause Mechanism in RetryableTopic

@RetryableTopic uses ContainerPartitionPausingBackOffManager: on first retry message processing, it checks backoff, pauses the partition for now() + delay, and schedules resume via ListenerContainerPauseService.

Pause code:

private final TaskScheduler scheduler;

public void pausePartition(MessageListenerContainer messageListenerContainer, 
                           TopicPartition partition,
                           Duration pauseDuration) {
    Instant resumeAt = Instant.now().plusMillis(pauseDuration.toMillis());
    messageListenerContainer.pausePartition(partition);
    this.scheduler.schedule(() -> {
        messageListenerContainer.resumePartition(partition);
    }, resumeAt);
}

Under high load, the scheduler task queue overflows. By default, spring.task.scheduling.pool.size=1, creating a bottleneck: resume tasks accumulate, fetch doesn't start timely.

Solution: Increase Scheduler Pool

Set spring.task.scheduling.pool.size=10 in application.properties.

Results:

• Retry-0 lag stabilized at an acceptable level (doesn't drop to 0 due to backoff).
• Poll intervals shortened, fetch-rate increased.
• Listener invocations became consistent.

Lag graph after fix: remains flat without growth. Fetch latency decreased, polls more frequent.

Key Takeaways

• Limited scheduler: default pool of 1 thread can't handle mass pause/resume under retry load.
• Metrics first: monitor fetch-rate, poll-idle, listener-time before thread dump.
• Transactions secondary: ack-mode=record, max.poll.records=1 don't affect lag.
• Concurrency OK: Spring Kafka scales correctly by partitions, even with multiplied topics.
• Alternative: for highload, consider custom retry with timestamp checks in a loop instead of pauses.

Monitoring Recommendations

Add dashboards for:

• Commit/rollback rate for exactly-once.
• Scheduler queue size (custom metric).
• Partition pause/resume events.
• Backoff timestamp vs current time delta.

Test under load before production. Spring Kafka source code provides full mechanics insight.

— Editorial Team