TurboQuant: KV Cache Compression Algorithm for Optimizing AI Models
Google's TurboQuant addresses bottlenecks in transformer KV-caches using vector quantization to reduce memory consumption without sacrificing accuracy. The algorithm combines PolarQuant and QJL, delivering compression down to 3 bits per element while preserving performance on long-context tasks. This makes it possible to run large LLMs on resource-constrained hardware.
The method targets key-value pairs in attention mechanisms, where traditional quantization incurs overhead from storing constants for data blocks. TurboQuant eliminates these costs by using polar coordinates and residual error quantization.
PolarQuant: Compression Without Overhead
PolarQuant converts Cartesian coordinates of memory vectors into polar coordinates, eliminating redundant constants. Instead of storing distances along the X, Y, Z axes, the method uses radius and angles to minimize codebook memory.
Advantages of the approach:
- Zero overhead for storing metadata.
- Preservation of vector space geometry.
- Compatibility with existing vector search pipelines.
This is especially useful for speeding up indexing in semantic search systems, where TurboQuant achieves 1@k recall on the GloVe dataset (d=200).
QJL: Error Correction with Johnson-Lindenstrauss
After PolarQuant, QJL is applied to the 1-bit residual. The algorithm leverages the Johnson-Lindenstrauss Transform (JLT) to project high-dimensional data into a lower-dimensional space while preserving distances.
QJL includes:
- Projection of errors onto a random JLT matrix.
- A specialized estimator to balance query accuracy and compressed data.
- No extra memory overhead for decoding.
The result is high-precision similarity recovery without needing to fine-tune the model.
Benchmarks and Performance
TurboQuant was tested on open LLMs (Gemma, Mistral) across benchmarks like LongBench, Needle In A Haystack, ZeroSCROLLS, RULER, and L-Eval. On Llama-3.1-8B-Instruct, it outperforms baselines in scalar product distortion and recall while minimizing KV-cache size.
In the Needle In A Haystack task (finding a fragment in a long context), KV compression to 3 bits maintains the original model's accuracy and accelerates inference. TurboQuant delivers speedup in attention logits compared to an optimized JAX baseline across 3- to 8-bit levels.
Key metrics:
- At least 6x reduction in KV-cache memory.
- Faster vector search thanks to smaller indexes.
- Stable performance without retraining.
What's Important
- TurboQuant combines PolarQuant and QJL for extreme KV-cache compression without quality loss.
- Supports long contexts in LLMs, boosting inference speed by 20-30%.
- Ideal for vector search and semantic indexes with minimal memory use.
- Works on Gemma, Mistral, and Llama models without retraining.
- Limited to inference; doesn't solve training challenges.
Deployment Prospects
The algorithm fits production scenarios with large vector databases, cutting index build latency. For LLMs, it optimizes KV-cache for long sequences, enabling models to run on consumer hardware. A presentation at ICLR 2026 will validate its scalability.
— Editorial Team
No comments yet.