Redis Lua Queue: partition order and worker pool

Redis Queue with Lua: Guaranteed Order in Partitions with a Shared Worker Pool

In high-load systems, message processing requires strict order within logical partitions across hundreds of thousands of keys. A limited worker pool must evenly distribute the load without being blocked by heavy partitions. Support for retries and delayed tasks is implemented via Redis with Lua scripts in the smart-redis-queue library.

The solution uses Lua's atomicity for operations on ZSET, HASH, and STRING without race conditions. Keys are formed with the prefix queue:{queueName}:. Ordered partitions (code with prefix !) are processed exclusively with locking, while regular ones are handled in parallel.

Comparison with Kafka and RabbitMQ

Kafka ensures order within topic partitions and scales with consumer groups, but it fixes partition assignments to consumers. A dynamic worker pool that picks tasks from any ready partitions requires different semantics. TTL in Kafka does not replace precise scheduling until time T.

RabbitMQ supports TTL and DLQ, but strict order with multiple consumers is broken by round-robin. Separate queues for millions of keys do not scale.

Redis + Lua offers a lightweight alternative: one round-trip per atomic operation, no cluster or complex monitoring needed.

Key Structure

| Key | Structure | Purpose |

|------|-----------|------------|

| queue:{name}:partitions | SET | Active partitions |

| queue:{name}:partition:{code}:{priority} | ZSET | Task queue by priority, score = availability time (ms) |

| queue:{name}:partition:{code}:priorities | ZSET | Priority levels |

| queue:{name}:payload:{taskId} | STRING | Payload |

| queue:{name}:partition:{taskId} | STRING | Partition code |

| queue:{name}:consumer:{id}:tasks | HASH | Tasks in progress |

| queue:{name}:partition:{code}:lock | STRING | Lock for ordered partitions |

| queue:{name}:partition:{code}:block | STRING (TTL) | Temporary block (rate limit) |

Task Lifecycle

• Publish: Idempotent addition with SET NX, writing to ZSET with score = scheduled (ms). Partition is added to SET.

local ok = redis.call('SET', payloadKey, payload, 'NX')
if ok then
    redis.call('ZADD', partitionQueueKey, scheduled, taskId)
    redis.call('SADD', partitionsKey, partitionCode)
end

• Get: Heartbeat, iteration over partitions, checking lock/block for ordered. Take by descending priorities (ZREVRANGE), head of ZSET if scheduled <= now. Move to consumer's HASH.

• Ack: Remove from HASH, payload, keys. Release lock if ordered counter = 0.

• Reject: Return to ZSET with new score. For ordered — priority +1, reject_seq for tie-break. Consumption rejects the entire batch of an ordered partition if the first task is rejected.

Performance Benchmarks

Tests on Go 1.22+, Redis 5, Intel i7-13700H.

Publishing (tasks/s):

• 1 task: 17–23k
• Batch 10: 80–92k
• Batch 100: 100–117k

Batching increases throughput via a single Lua call.

Consumption by one worker (tasks/s):

| Prefetch | ns/op | Tasks/s |

|----------|--------|---------|

| 1 | 80–103 µs | 9.7–12.5k |

| 5 | 57–67 µs | 15–17.5k |

In parallel:

• 2 workers: 28–30k/s
• 8 workers: 23–26k/s

Scaling is non-linear due to Redis contention.

Failure Handling

• Heartbeat with TTL ~120s.
• Live consumer scans dead ones via distributed lock.
• Tasks from dead consumer's HASH are returned to the queue (ordered with +priority).

Rate limit: RejectWithDelay sets a block with TTL.

Recommendation: priority steps of 100+ to avoid collisions with reject increments.

Key Takeaways

• Lua atomicity ensures no race conditions when moving tasks between ZSET and HASH.
• Ordered partitions maintain order via exclusive lock and reject logic.
• Batching in Publish/Get boosts throughput to 100k+ tasks/s.
• Automatic recovery after worker failures without task loss.
• Suitable for scenarios with millions of partitions without cluster overhead.

— Editorial Team