Articles by tag: turboquant

Local 200B models: Bonsai and TurboQuant

Breakdown of 1-bit weights Bonsai 8B and KV-cache compression TurboQuant. Memory reduction 14x for weights, 6x for cache. Calculations for 235B models. Learn how to run giants locally.

TurboQuant: 6x compression of LLM KV-cache

Break down Google's TurboQuant: PolarQuant + QJL for 6x KV-cache compression without accuracy loss. Inference acceleration up to 8x. Benchmarks, principle of operation for LLM developers.