Articles by tag: kv-cache

TurboQuant: lossless KV-cache compression for AI

Learn how Google's TurboQuant compresses transformer memory down to 3 bits with PolarQuant and QJL. Benchmarks on Gemma, Mistral. Optimization for AI developers.

TurboQuant: 3-bit KV-cache for LLM without losses

Learn how Google TurboQuant compresses key-value cache of LLM to 3 bits, speeding up inference 8x. Benchmarks on Gemma/Mistral, application in RAG. For developers.

Prompt Caching LLM: KV cache 10 times cheaper

Prompt Caching Breakdown: how OpenAI and Anthropic cache KV attention to reduce costs and delays. Technical details for developers, inference examples. Speed up your LLM queries.

File access in local AI agent: tools

Implementation of file access for local AI agent: permission gate, auto-verify, fixing KV-cache bugs. Technical details for developers. Study the architecture.

Local 200B models: Bonsai and TurboQuant

Breakdown of 1-bit weights Bonsai 8B and KV-cache compression TurboQuant. Memory reduction 14x for weights, 6x for cache. Calculations for 235B models. Learn how to run giants locally.

TurboQuant: 6x compression of LLM KV-cache

Break down Google's TurboQuant: PolarQuant + QJL for 6x KV-cache compression without accuracy loss. Inference acceleration up to 8x. Benchmarks, principle of operation for LLM developers.