# Compressing LLM KV Cache to 3 Bits: TurboQuant from Google Research
Google Research has developed TurboQuant—a method for compressing the key-value cache (KV cache) of large language models (LLMs) to 3 bits per parameter. The algorithm preserves generation accuracy without retraining or additional memory overhead. TurboQuant combines PolarQuant for primary quantization and QJL for error correction, enabling 6–8x faster processing of long contexts.
KV cache accumulates intermediate representations from the attention mechanism during sequence processing. For contexts of 128k+ tokens, the cache size exceeds gigabytes, limiting GPU inference. TurboQuant solves this by converting data to polar coordinates and applying one-bit correction.
How the Algorithm Works
TurboQuant consists of two stages:
- PolarQuant: converts KV vectors to polar coordinates. Amplitude is quantized to high precision (8–16 bits), while phases are compressed to 1–2 bits. This minimizes error on principal components.
- QJL (Quantized Joint Learning): one-bit post-correction for residual artifacts. Requires no gradient training and runs during inference.
The combination delivers a 3-bit representation without perplexity degradation. The approach is proven optimal against theoretical compression lower bounds for attention matrices.
Benchmarks and Performance
Testing on open models Gemma-2B/7B and Mistral-7B showed:
- 6–10x KV cache memory compression.
- Up to 8x attention speedup on H100 GPU (3-bit vs FP32 baseline).
- Perfect accuracy on Needle In A Haystack (NIAH) for contexts up to 128k tokens—0% fact retrieval errors.
| Benchmark | FP32 (baseline) | TurboQuant 3-bit | Relative speedup |
|-----------|-----------------|------------------|------------------|
| LongBench | 100% | 99.8% | 6.2x memory, 7.1x speed |
| RULER | 100% | 100% | 8.3x speed |
| NIAH | 100% | 100% | 6.8x memory |
In 4-bit mode, speedup peaks on H100 with linear scaling on A100/H200.
Applications Beyond LLMs
TurboQuant applies to vector search in RAG systems and semantic ranking. Tests on ANN benchmarks (SIFT, GIST) outperform FP16 quantization:
- Recall@10: +2–5% over Int8 baselines.
- Latency: 4–6x lower without index retraining.
The algorithm is domain-independent, easing production deployment. For LLMs, it integrates into inference frameworks like vLLM or Hugging Face TGI without model changes.
Key advantages for mid/senior devs:
- Zero-shot quantization—plug-and-play.
- Full-precision accuracy on long-context tasks.
- Scalability on multi-GPU setups.
Key Takeaways
- TurboQuant compresses LLM KV cache to 3 bits, accelerating inference 8x without quality loss.
- Two-stage algorithm (PolarQuant + QJL) is mathematically optimal, no fine-tuning required.
- Full compatibility with Gemma/Mistral; ideal for 100k+ token contexts.
- Extends to vector search: higher recall with 6x smaller footprint.
- Presentation at ICLR 2026—open-source implementation expected.
The method is critical for scaling LLMs in production, where KV cache memory is the primary bottleneck. (Total length: ~3200 characters)
— Editorial Team
No comments yet.