Kafka in Node.js: reliable consumers and scaling

# Effective Kafka Message Processing in Node.js: Consumers, Rebalancing, and Scaling

Properly implementing Kafka consumers in Node.js requires understanding not just the basic API, but also the architectural features of this distributed queue. Mistakes in partition setup, reading strategy, or offset management lead to message loss, excessive rebalances, and reduced throughput. In this article, we break down the key aspects of building reliable and high-performance consumers using KafkaJS.

Partitions as the Foundation of Scaling

Scaling message consumption from a Kafka topic is limited by the number of partitions. No matter how many instances you run, the number of active consumers in a group can't exceed the number of partitions in the topic. The formula is simple: N_consumers ≤ N_partitions. Exceeding this ratio means some instances sit idle without assignments, creating unnecessary load on the group coordinator.

Choosing the initial number of partitions is a strategic decision. While Kafka lets you increase partitions on the fly, it breaks the message ordering guarantee by key: new messages with the same key might land in a different partition than previous ones. For flexibility, pick a number with many divisors (like 24) to ensure even distribution across any consumer count from 1 to 24.

Keep in mind: the total number of partitions across the entire cluster shouldn't exceed 200,000. For low-traffic topics, over-partitioning doesn't make sense.

Parallel Processing and Replica Selection

Even with multiple partitions assigned to a single consumer, processing is sequential by default. To enable parallelism within an instance, use the partitionsConsumedConcurrently option:

consumer.run({
    partitionsConsumedConcurrently: 3,
    eachMessage: async ({ topic, partition, message }) => {
        // processing
    },
});

This preserves ordering guarantees within a single partition, but not across them. It safely boosts throughput without breaking processing logic.

For geographically distributed clusters, network latency is critical. By default, consumers read from the leader replica, which also handles writes—creating a bottleneck. The fix is rack-aware replication. Set the broker.rack parameter on brokers and specify rackId in the client:

const consumer = kafka.consumer({
  groupId: 'user-service',
  rackId: 'sibir_tomsk'
});

Kafka then routes requests to a replica in the same "rack domain," minimizing latency. A slight replication lag is acceptable in most scenarios, as Kafka isn't built for strict real-time.

Offset Management and Initial Reading

New consumer groups often run into a common issue: they don't see historical messages. The culprit is the auto.offset.reset parameter, which defaults to latest. Without a committed offset in __consumer_offsets, the consumer starts from the end of the partition.

The alternative is earliest, but using it without safeguards is risky: processing years of history can overload your service and downstream systems. A safer approach:

• Use fromBeginning: true on subscribe if you need the full history:

await consumer.subscribe({ topic: 'user_events', fromBeginning: true });

This option only works on first connection. After the first offset commit, Kafka ignores fromBeginning.

• For precise recovery, manually set offsets via CLI or UI tools (KafkaUI, Kadeck) after stopping the consumer.

This control prevents accidental offset rewinds and ensures processing integrity.

Rebalance Strategies and Their Pitfalls

Rebalancing redistributes partitions when the group membership changes. KafkaJS only supports the Eager strategy, where all consumers pause processing ("stop-the-world"), finish current tasks, commit offsets, and wait for new assignments.

Frequent rebalances hurt availability and increase latency. Main causes:

• Mixed functionality in one process—HTTP server, background jobs, and consumer in the same instance. Any fatal error outside message handling crashes the whole process and triggers a rebalance.
• Incorrect timeouts—sessionTimeout and heartbeatInterval must align. Rule of thumb: heartbeatInterval ≤ sessionTimeout / 3.
• Slow message processing—if handling one message exceeds rebalanceTimeout, the consumer gets kicked from the group.

For diagnostics, subscribe to the REBALANCING event:

consumer.on(consumer.events.REBALANCING, ({ payload }) => {
  console.log('Rebalancing started for group:', payload.groupId);
});

This lets you collect metrics and spot abnormal rebalance rates.

While Kafka 2.3+ offers Static Membership and CooperativeStickyAssignor to ease these issues, KafkaJS doesn't support them yet. Architectural fixes are your only path to stability.

Key Takeaways

• Partition count sets the upper limit for horizontal consumer scaling.
• Parallel processing (partitionsConsumedConcurrently) speeds things up without losing per-partition order.
• Reading from local replicas (rackId) cuts network latency in geo-distributed setups.
• New consumer groups skip history by default—use fromBeginning or manual offset setting.
• Isolate consumers from other app components to avoid unnecessary rebalances.

— Editorial Team