TurboQuant: 6x Smaller LLM KV Cache Without Accuracy Loss
In autoregressive language model generation, the attention mechanism relies on Key and Value vectors from prior tokens. Recalculating the full context at each step is inefficient, so intermediate representations are cached. The KV cache grows linearly with context length, limiting throughput and maximum context size.
On long sequences, inference hits memory bandwidth limits: GPUs spend more time reading data than computing. This increases latency and cost. TurboQuant tackles this by compressing the cache while preserving semantic meaning for attention.
How PolarQuant Works
The first stage is PolarQuant. Vectors are transformed into polar coordinates after a random rotation. The radius encodes vector magnitude, the angle captures direction.
# Pseudocode: PolarQuant
rotated = rotate_random(vector)
rho = norm(rotated) # radius
phi = angle(rotated) # angle
quantized = quantize(rho, phi)Random rotation simplifies data distribution, making it predictable for quantization. No additional parameters or codebooks needed—compression reaches 3–4 bits per value without fine-tuning the model.
Error Correction via QJL
Quantization introduces noise. The second stage is QJL (Quantized Johnson-Lindenstrauss), which encodes residual error in just 1 bit per value. The transformation preserves Euclidean distances between vectors—critical for dot-product attention.
QJL approximates the Johnson-Lindenstrauss lemma in a quantized form:
- Input: residual after PolarQuant
- Output: 1-bit correction
- Property: ||Qx - Qy|| ≈ ||x - y|| for x, y ∈ KV-cache
Together, they achieve 1.25–2 bits per value with minimal perplexity degradation.
Benchmarks & Performance
Tested on Llama-3.1-8B-Instruct (LongBench):
- Compression: 6–8x vs baseline
- Logit computation speedup: up to 8x on H100 (JAX baseline)
- Quality: <1% drop in needle-in-haystack and long-context tasks
On GloVe (d=200), TurboQuant leads in the trade-off between recall and compression ratio (1:k optimal).
| Method | Bits/Value | Compression | Perplexity Drop |
|--------|------------|-------------|------------------|
| Baseline | 16 | 1x | 0% |
| INT8 | 8 | 2x | 0.5% |
| TurboQuant (3-bit) | 3 | 6x | 0.2% |
Achieves up to 3 bits without retraining—a breakthrough for zero-shot quantization.
Key Takeaways
- Extreme compression: KV cache reduced by 6x at 3 bits per value.
- Faster inference: Up to 8x speedup on memory-bound workloads.
- Distance preservation: QJL ensures attention compatibility.
- Universal applicability: Works for vector search and RAG.
- No fine-tuning required: Plug-and-play replacement for existing LLMs.
Beyond LLMs
TurboQuant extends beyond transformer architectures. In vector databases, embedding compression maintains ANN search performance (HNSW, FAISS). On edge devices, it reduces DRAM demands, enabling on-device inference.
For senior engineers: integrate into custom inference engines. JAX/Flax support simplifies prototyping. Test on long contexts (>128k tokens) to unlock real-world gains.
— Editorial Team
No comments yet.