Articles by tag: quantization

TurboQuant: 3-bit KV-cache for LLM without losses

Learn how Google TurboQuant compresses key-value cache of LLM to 3 bits, speeding up inference 8x. Benchmarks on Gemma/Mistral, application in RAG. For developers.

Lila-Leech: 24D lattice in 10 lines of code

Deploy 4B models on smartphones with 44.9x compression. Frozen Leech Core for Edge AI without servers. Explore code and SR/CN metrics. For middle/senior dev.