Kafka producer errors in Node.js: solutions and optimization

Challenges and Solutions When Working with Kafka Producer in Node.js

Working with Apache Kafka via the KafkaJS client library in Node.js requires understanding not only core concepts but also specific errors that can degrade system performance and reliability. In this first part focused on the producer, we'll examine key issues when publishing messages and methods to resolve them.

Partition Selection Errors

Choosing a partition for a message is a critical step affecting load distribution among consumers. Incorrect partitioning logic can lead to imbalance and loss of scalability.

A common mistake is using a partitioning function based on integer division of the message key (key). If the key isn't a number or can't be properly converted to an integer (e.g., MongoDB ObjectID), all messages will end up in a single partition.

const producer = await kafka.producer(topic, {
  createPartitioner: function () {
    return ({ partitionMetadata, message }: PartitionerArgs) => {
      return partitionMetadata.length
        ? partitionMetadata[Number(message.key) % partitionMetadata.length]?.partitionId || 0
        : 0;
    };
  },
});

In this case, consumers in the group won't be able to efficiently distribute the load since all data is concentrated in one partition.

Solution:

• Validate the type and format of the partition key at the producer level.
• Set up metrics to monitor partition size imbalances.
• Use more reliable partitioning algorithms, such as hashing string keys.

Publishing to Non-Existent Partitions

Partition numbering in a topic starts from 0. If the producer attempts to publish a message to a partition index exceeding the available count (e.g., partition 1 in a topic with only one partition), KafkaJS won't throw an error. The send() method returns an empty array, and the message is effectively not written.

const result = await producer.send({
  topic: 'user_events',
  messages: [
    { key: '2', value: 'user_created', partition: 1 }
  ]
});
console.log(result); // []

Solution:

• Create topics and partitions through configuration files during deployment to avoid manual errors.
• Validate the send() response at the application level. If the result is empty or contains an error (e.g., errorCode is not 0), take action: throw an exception, log it, publish to a backup topic, or send a metric.
• Regularly check topic metadata (list of available partitions) before sending.

Inefficient Producer Usage

Creating a new producer instance (kafka.producer()) for every request or operation is a common mistake leading to significant overhead. Each new producer establishes heavy network connections to Kafka brokers.

app.post('/send-message', async (request) => {
  const { topic, message } = request.body;
  const producer = kafka.producer(); // New instance per request
  // ...
});

Solution:

• Use the Singleton pattern or a producer pool. Create one producer instance and reuse it for all publishing operations (producer.send()). This improves throughput and reduces broker load.
• For multi-threaded or microservices architectures, consider using a shared Kafka client with a shared producer.

Network Issues and Retry Strategy

The network between the producer and the Kafka cluster may be unstable. To improve message delivery reliability:

• Configure handling of the request_timeout event to track network problems.

producer.on('producer.network.request_timeout', (error) => {
  console.error({
    code: 'producer_network_timeout',
    message: error.message,
    data: { stack: error.stack },
  });
})

• Configure retry mechanisms (retry) when creating the producer. By default, retries are set to 5.

kafka.producer({
  retry: { retries: 10 }
});

Important: Increasing retry attempts without a proper strategy (e.g., without increasing intervals between attempts) can cause DDoS-like load on brokers. Use configurations with increasing intervals (exponential backoff).

Handling Large Messages

Kafka is not designed for messages larger than 1 MB. Publishing large data (5–10 MB) leads to serious issues:

• Frequent segment rotation: With a small maximum segment size (e.g., 512 MB), large messages trigger rapid rotation, increasing system load.
• High network traffic: Replicating large messages across all In-Sync Replicas (ISR) increases network burden.
• Reduced throughput: The producer's message buffer grows, increasing risks of data loss and longer acknowledgment wait times when acks=-1.
• Resource consumption: More RAM is required for os page cache on brokers and for the Kafka JVM, increasing CPU load due to garbage collection (GC).

Message Size Recommendations:

• Use the formula: message.max.bytes = average message size * 2, but less than 1MB.
• For large files (video, audio, documents), use external storage (e.g., S3). Send only a reference to the object in the Kafka message.
• If large JSON/XML must be sent without external storage, the message likely contains redundant data for different consumers. Consider splitting the topic into multiple specialized ones, each with the minimal necessary fields.
• An alternative, but more complex approach: split large messages into chunks and reassemble them on the consumer side.

Key Takeaways

• Partitioning: Incorrect partition calculation causes load imbalance and loss of scalability. Key validation and metric monitoring are mandatory.
• Partition Existence: Publishing to a non-existent partition doesn’t trigger an error in KafkaJS, but the message isn’t written. Validating the send() response is critical.
• Producer Efficiency: Creating a new producer per request is costly. Use Singleton or a pool.
• Network and Retries: Configuring network timeout handling and retry strategies improves delivery reliability.
• Message Size: Sending data over 1 MB negatively impacts cluster performance. Use external storage for large files and optimize message structure.

— Editorial Team